Data reconstruction method, medium, device, program product, and distributed storage system

Abstract

A data reconstruction method, a medium, a device, a program product, and a distributed storage system, which are applied to a distributed storage system comprising a plurality of data nodes, where each data node stores a data block for reconstructing original data, and each piece of original data is obtained by reconstructing the plurality of data blocks respectively stored on the plurality of data nodes. The method comprises: acquiring a plurality of data blocks currently being respectively and asynchronously transmitted by a plurality of data nodes, wherein a data node, in response to completion of transmission of a previous data block on the node, transmits a current data block on the node; on the basis of each data block and an intermediate reconstruction result of original data reconstructed by the data block, performing partial data reconstruction, so as to obtain a current reconstruction result of the original data reconstructed by the data block; and for any piece of original data, if the number of obtained data blocks for reconstructing the original data is less than the minimum number of data blocks required for reconstructing the original data, returning to the step of acquiring a plurality of data blocks currently being respectively transmitted by a plurality of data nodes.

Description

[0001]This disclosure claims priority to Chinese patent application number 202410349751.8, filed with the China Patent Office on March 25, 2024, entitled "Data Reconstruction Method, Medium, Device, Program Product, and Distributed Storage System," the entire contents of which are incorporated herein by reference. TECHNICAL FIELD This disclosure relates to the field of communications technology, and more particularly to a data reconstruction method, medium, device, program product, and distributed storage system. Background: In a distributed storage system, to improve data fault tolerance, multiple data blocks used to reconstruct original data can be stored on multiple data nodes of the distributed storage system. When original data is lost or damaged, data blocks can be read from multiple data nodes and reconstructed based on the read data blocks. In related technologies, each data node synchronously transmits the data blocks required to reconstruct the same piece of original data. Only after all data blocks required to restore the same piece of original data have been transmitted does the data node continue to transmit the data blocks required to reconstruct the next piece of original data. This approach results in low data throughput and low data reconstruction efficiency. Invention content: In a first aspect, an embodiment of the present disclosure provides a data reconstruction method, which is applied to a distributed storage system including multiple data nodes, each data node stores data blocks for reconstructing original data, and each copy of original data is reconstructed from multiple data blocks respectively stored on the multiple data nodes; the method includes: obtaining multiple data blocks currently sent asynchronously by the multiple data nodes; performing data reconstruction based on each data block and an intermediate reconstruction result of the original data reconstructed by the data block, to obtain a current reconstruction result of the original data reconstructed by the data block; the intermediate reconstruction result of the original data is obtained by partially reconstructing data blocks historically sent by at least one data node for reconstructing the original data; for any copy of the original data, if the number of data blocks obtained for reconstructing the original data is less than the minimum number of data blocks required to reconstruct the original data, returning to the step of obtaining the multiple data blocks currently sent by the multiple data nodes.In a second aspect, embodiments of the present disclosure provide a data reconstruction method, applied to a distributed storage system comprising multiple data nodes, each of which stores data blocks for reconstructing original data, wherein each piece of original data is reconstructed from multiple data blocks stored on the multiple data nodes. The method comprises: if a data block for reconstructing the original data is stored on the local node, sending the data block to a reconstruction node; wherein the local node and other data nodes in the distributed storage system asynchronously send data blocks for reconstructing the same piece of original data, such that upon receiving each data block, the reconstruction node partially reconstructs the original data based on the currently received data block until the original data is reconstructed; and if the local node is configured to receive the reconstructed original data, obtaining the reconstructed original data returned by the reconstruction node. In a third aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any embodiment of the present disclosure. In a fourth aspect, embodiments of the present disclosure provide a computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the program, implements the method described in any embodiment of the present disclosure. In a fifth aspect, embodiments of the present disclosure provide a computer program product, comprising a computer program. When executed by a processor, the computer program implements the method described in any embodiment of the present disclosure. In a sixth aspect, embodiments of the present disclosure provide a distributed storage system, comprising a metadata node and multiple data nodes. The multiple data nodes store data blocks used to reconstruct original data, and each copy of original data is reconstructed from multiple data blocks stored on the multiple data nodes. The metadata node is configured to distribute a data reconstruction task to a reconstruction node among the multiple data nodes. The data node is configured to, when the node is the reconstruction node, respond to the data reconstruction task and execute the method described in the first aspect, and, when the node is a data node other than the reconstruction node, execute the method described in the second aspect. In embodiments of the present disclosure, each data node asynchronously transmits data blocks used to reconstruct the same copy of original data. As soon as the previous data block on the node is transmitted, the current data block can be transmitted immediately, without having to wait for all data blocks required to reconstruct the same copy of original data to be transmitted before continuing to transmit data blocks. This improves data block transmission efficiency and, in turn, data reconstruction efficiency.Because the complete original data cannot be reconstructed from the data blocks that are sent first, the data block receiver can partially reconstruct the corresponding original data based on the currently acquired data block for each acquired data block, obtaining the current reconstruction result corresponding to the original data. After all the data blocks required to reconstruct the original data are acquired, the complete original data can be reconstructed. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and do not limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings herein are incorporated into and constitute a part of the present disclosure. These drawings illustrate embodiments consistent with the present disclosure and, together with the description, serve to illustrate the technical solutions of the present disclosure. Figure 1 is a schematic diagram of a distributed storage system according to an embodiment of the present disclosure; Figure 2 is a schematic diagram of a data reconstruction process according to an embodiment of the present disclosure; Figure 3 is a schematic diagram of a synchronous transmission mode according to an embodiment of the present disclosure; Figure 4 is a flow chart of a data reconstruction method according to an embodiment of the present disclosure; Figure 5 is a schematic diagram of an asynchronous transmission mode according to an embodiment of the present disclosure; Figure 6 is a schematic diagram of the reconstruction process and memory allocation according to an embodiment of the present disclosure; Figure 7 is a schematic diagram of differences between data block transmission progresses of data nodes according to an embodiment of the present disclosure; Figure 8 is a flow chart of a data reconstruction method according to another embodiment of the present disclosure; and Figure 9 is a schematic diagram of a computer device according to an embodiment of the present disclosure. Specific Embodiments: Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numerals in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims. The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to be limiting of the present disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items. Additionally, the term "at least one" as used herein refers to any combination of at least two of any one or more of a plurality. It should be understood that although the terms "first," "second," "third," etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another.For example, without departing from the scope of this disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the term "if" as used herein may be interpreted as "at the time of," "when," or "in response to a determination." To help those skilled in the art better understand the technical solutions in the embodiments of this disclosure and to make the aforementioned objectives, features, and advantages of the embodiments of this disclosure more readily apparent, the technical solutions in the embodiments of this disclosure are further described below in conjunction with the accompanying drawings. A distributed storage system refers to a system that stores data on multiple independent data nodes to achieve distributed data storage and management. A distributed storage system must ensure that the state presented to the outside world is consistent and that state rollbacks do not occur. As shown in Figure 1, a distributed storage system 10 includes a metadata node (MetaNode) 102 and multiple data nodes (ChunkServers) 104. Metanode 102 is a centralized metadata storage node, typically used to store data status information, storage location information, data length information, and so on. Data nodes 104 are nodes in the distributed storage system 10 used to store data. They are typically responsible for operations such as writing, storing, reading, and deleting data replicas. To improve data fault tolerance, multiple data blocks used to reconstruct the original data can be stored on multiple data nodes 104 within the distributed storage system 10. To improve data reliability within the distributed storage system 10, when some data is lost or damaged, the distributed storage system 10 can retrieve data blocks from multiple data nodes 104 to reconstruct the lost or damaged data and write the reconstructed data to other data nodes 104, thereby ensuring data security. One data reconstruction method is based on erasure codes. In this method, a segment of data is divided into multiple data blocks of equal length. Each of the multiple data blocks includes the original data block and at least one check block. The original data block contains the original data content to be stored or transmitted and can be obtained by dividing the original data into blocks according to a certain partitioning method. The check block is redundant check information calculated from the data block and is used to detect and recover from errors or loss in the data block. Parity blocks do not contain original user data, but rather redundant information calculated based on the contents of data blocks. If any one or more data blocks are lost, the lost data blocks can be reconstructed and restored based on the remaining data blocks. Different data blocks can be stored on different data nodes 104.As shown in FIG2 , the example of data nodes 104 being four is used for illustration. For ease of distinction, the different data nodes 104 are denoted as data node CS1, data node CS2, data node CS3, and data node CS4. Data node CS1 stores data block A1, data node CS2 stores data block A2, data node CS3 stores data block A3, and data node CS4 stores data block A4. Data blocks A1, A2, and A3 may be original data blocks, and data block A4 may be a parity block. If some of the data blocks A1, A2, and A3 are lost or damaged, the data blocks can be reconstructed using other data blocks. For example, if data block A1 is damaged, data block A1 can be reconstructed using data blocks A2, A3, and A4. Specifically, data nodes CS2, CS3, and CS4 can send data blocks A2, A3, and A4, respectively, to reconstruction node Y. Reconstruction node Y can reconstruct data based on the received data blocks A2, A3, and A4 to obtain reconstructed data block A1, and then send the reconstructed data block A1 to data node CS1 for storage. In some embodiments, when the number of original data blocks is k and the number of check blocks is m, the total number of data blocks and check blocks required to reconstruct any original data block is k. Assuming k is 8 and m is 3, the total number of data blocks and check blocks required to reconstruct any original data block is 8. For example, data reconstruction can be performed using 7 original data blocks and 1 check block, or using 6 original data blocks and 2 check blocks. Related technologies generally synchronously read data blocks from multiple data nodes to perform data reconstruction. After all data blocks used to reconstruct the same original data are read, the data blocks used to reconstruct the next original data block are read. This method is called synchronous mode. In synchronous mode, throughput is limited by the slowest data node among multiple data nodes. In situations such as heterogeneous devices, partial device failures, high device front-end pressure, and large amounts of data to be copied from devices, the transmission speed of some data nodes in the distributed storage system may remain low for extended periods. This prevents efficient utilization of the distributed storage system's cluster bandwidth, resulting in low data reconstruction throughput and poor throughput stability.As shown in FIG3 , assume that there are four data nodes 104, each of which is designated CSL CS2, CS3, and CS4. Each rounded rectangle represents a data block required to reconstruct a set of original data. The original data reconstructed by the data block is indicated by the label within the solid rounded rectangle. For example, the rounded rectangle labeled "Original Data 1" represents the data block used to reconstruct original data 1, the rounded rectangle labeled "Original Data 2" represents the data block used to reconstruct original data 2, and so on. A dashed rounded rectangle indicates that the corresponding data node 104 is currently in an idle waiting state. The height of the rounded rectangle represents the time required to transmit the data block. t1 to t8 represent different moments. For ease of illustration, assume that the time interval between adjacent moments is 1s, i.e., t1 represents the 1s, t2 represents the 2s, and so on. As can be seen, when reconstructing original data 1, data nodes CS1 and CS3 complete transmission of the data blocks required for reconstructing original data 1 on their own nodes in the 1st second, while data nodes CS2 and CS4 complete transmission of the data blocks required for reconstructing original data 1 on their own nodes in the 2nd second. Therefore, after completing transmission of the data blocks required for reconstructing original data 1 on their own nodes, data nodes CS1 and CS3 need to wait 1 second before continuing to transmit the data blocks required for reconstructing original data 2 on their own nodes. When reconstructing original data 2, data node CS1 requires 3 seconds to complete transmission of the data blocks required for reconstructing original data 2 on their own nodes, while data nodes CS2, CS3, and CS4 only require 1 second to complete transmission of the data blocks required for reconstructing original data 2 on their own nodes. Therefore, after completing transmission of the data blocks required for reconstructing original data 2 on their own nodes, data nodes CS2, CS3, and CS4 need to wait 2 seconds before continuing to transmit the data blocks required for reconstructing original data 3 on their own nodes. The transmission method for other data blocks is similar to the above process and will not be described in detail here. It can be seen that since each data node 104 is in an idle waiting state for a long time, only the data blocks required for reconstructing three copies of the original data can be transmitted within the first 8 seconds, and the cluster throughput is low.Based on this, an embodiment of the present disclosure provides a data reconstruction method, which is applied to a distributed storage system 10 including multiple data nodes 104. Each data node 104 stores data blocks for reconstructing original data, and each copy of original data is reconstructed from multiple data blocks stored on the multiple data nodes 104. Referring to FIG4 , the method includes: Step S12: Acquiring multiple data blocks currently and asynchronously sent by the multiple data nodes 104; Step S14: Performing data reconstruction based on each data block and an intermediate reconstruction result of the original data reconstructed by the data block to obtain a current reconstruction result of the original data reconstructed by the data block; The intermediate reconstruction result of the original data is obtained by partially reconstructing data blocks historically sent by at least one data node 104 for reconstructing the original data; Step S16: For any copy of original data, if the number of acquired data blocks for reconstructing the original data is less than the minimum number of data blocks required to reconstruct the original data, returning to Step S12. In the embodiment of the present disclosure, each data node 104 sends the data blocks for reconstructing the same copy of original data on the node in an asynchronous manner. Each data node 104 transmits its current data block in response to the completion of the previous data block transmission. This eliminates the need to wait for all data blocks required to reconstruct the same set of original data to be transmitted before continuing to transmit data blocks. This improves data block transmission efficiency and, in turn, data reconstruction efficiency. At the data block receiving end, each acquired data block is partially reconstructed based on the currently acquired data block to obtain the current reconstruction result corresponding to the original data. Once all data blocks required to reconstruct the original data are acquired, the complete original data can be reconstructed. The following provides examples of specific implementations of the disclosed embodiments. The method of the disclosed embodiments can be executed by a reconstruction node. The reconstruction node can be selected from multiple data nodes 104. For example, a source data node (the data node where the data block used to reconstruct the original data is located) or a destination data node (the data node used to store the reconstructed original data) can be selected as the reconstruction node. Alternatively, any other data node can be selected as the reconstruction node. Each set of original data can be reconstructed based on multiple data blocks, which can include both original data blocks and check blocks. Each data node 104 can store one or more data blocks. During data reconstruction, each data node 104 can send at least some of the data blocks on the node to a reconstruction node for data reconstruction. For ease of description, the following example uses the storage of one data block on each data node 104 as an example.In step S12, when data is missing, the metadata node 102 may send a data reconstruction task to the reconstruction node. The reconstruction node may receive the data reconstruction task from the metadata node 102. The data reconstruction task may include information about the original data to be reconstructed and information about each data block used to reconstruct the original data. The original data information includes the storage location and length of the original data, and the data block information includes the storage location and length of the data block. The storage location may include the identification information of the data node 104 storing the original data or data block, as well as the starting storage location of the original data or data block on the corresponding data node 104. For each piece of original data requiring data reconstruction, the data reconstruction task may be used to determine the multiple target data blocks required to reconstruct the original data, and to determine the candidate data nodes where these multiple target data blocks are located. The reconstruction node may send a data request to a target data node among the candidate data nodes, so that the target data node returns the data block in response to the data request. The reconstruction node can then reconstruct the original data based on the data blocks returned by each target data node and return the reconstructed original data to the data node 104 used to store the original data. The data node 104 used to store the reconstructed original data can be the same as or different from the original data node where the original data resides. Any data node 104 can transmit its current data block in response to the completion of transmission of the previous data block on its own node. As shown in Figure 5, taking data node CS1 as an example, data node CS1 can immediately transmit its data block for reconstructing original data 2 in response to the completion of transmission of the data block for reconstructing original data 1 on its own node, without having to wait for data nodes CS2, CS3, and CS4 to complete transmission of the data blocks for reconstructing original data 1 before transmitting its data block for reconstructing original data 2. Similarly, in response to the completion of sending the data block for reconstructing the original data 2 on the data node, the data node CS1 may also immediately send the data block for reconstructing the original data 3 on the data node, without having to wait for the completion of sending the data block for reconstructing the original data 2 on the data nodes CS2, CS3, and CS4 before sending the data block for reconstructing the original data 3 on the data node. The process of sending data blocks by the data nodes CS2, CS3, and CS4 is similar and will not be described in detail here.As can be seen, in the asynchronous mode embodiment shown in FIG5 , upon completing transmission of a data block on its own node, each data node can immediately transmit the next data block. Data nodes CS1, CS2, CS3, and CS4 complete transmission of all data blocks used to reconstruct original data 1, original data 2, and original data 3 in just 5 seconds, compared to the 8 seconds required to complete transmission of all data blocks used to reconstruct original data 1, original data 2, and original data 3 in the synchronous mode shown in FIG3 . The asynchronous mode shown in FIG5 effectively improves data block transmission efficiency. In situations such as foreground request fluctuations and HDD random reads, background data transmission is prone to random read latency fluctuations. When random read latency fluctuations occur, the latency for reading different data blocks by the same data node 104 is often inconsistent, and the latency for different data nodes 104 to read data blocks used to reconstruct the same original data is also often inconsistent. Employing the asynchronous mode of the disclosed embodiment can effectively improve data block transmission efficiency. In some embodiments, a cache queue can be set up for each data node 104. Multiple data nodes 104 correspond to multiple cache queues. The cache queue corresponding to each data node 104 is used to cache data requests received by that data node 104. Data requests received by data node 104 are used to request data blocks stored by that data node for reconstructing the original data. Taking the system shown in Figure 5 as an example, a cache queue corresponding to data node CS1, denoted as cache queue Q1, can be set up to store data requests for data blocks on data node CS1. A cache queue corresponding to data node CS2, denoted as cache queue Q2, can also be set up to store data requests for data blocks on data node CS2. A cache queue corresponding to data node CS3, denoted as cache queue Q3, can also be set up to store data requests for data blocks on data node CS3. A cache queue corresponding to data node CS4, denoted as cache queue Q4, can also be set up to store data requests for data blocks on data node CS4. The reconstruction node can send data requests to each of the multiple data nodes 104. After receiving a data request, each data node 104 may add the data request to its corresponding cache queue. The data node 104 may sequentially read each data request cached in its corresponding cache queue. For each data request read, the data node 104 transmits the data block stored on the node to the reconstruction node in response to the data request. After the data block is successfully transmitted, the corresponding data request is deleted from the cache queue. The cache queue may be an asynchronous queue.In some embodiments, each piece of original data is a data block in the data to be reconstructed (referred to as a to-be-reconstructed data block). The to-be-reconstructed data block may be a data block of a preset size (e.g., 1 MB). Each data block stored on any data node 104 is sequentially sent by the corresponding data node 104 according to the position of the original data corresponding to each data block in the data to be reconstructed. For example, assuming the size of the data to be reconstructed is 10 MB and includes 10 1 MB data blocks to be reconstructed, each data node 104 first sends the data block used to reconstruct the first 1 MB data block to be reconstructed in the data to be reconstructed, then sends the data block used to reconstruct the second 1 MB data block to be reconstructed in the data to be reconstructed, and so on. In step S14, because the data blocks used to reconstruct the same piece of original data are not sent synchronously, a reconstruction node may only receive a portion of the data blocks used to reconstruct the original data, resulting in an inability to fully reconstruct the original data. Based on this, partial data reconstruction (hereinafter referred to as partial reconstruction) can be performed on the original data based on the acquired data blocks, and the intermediate reconstruction results obtained from the partial reconstruction can be stored. Subsequently, when additional data blocks used to reconstruct the original data are acquired, the original data can be further reconstructed until the complete original data is reconstructed. Partial data reconstruction refers to reconstructing a portion of the original data. Specifically, when one or more data blocks used to reconstruct a particular piece of original data are first acquired, a memory space can be allocated for the original data in memory to store the intermediate reconstruction results corresponding to the original data. Data reconstruction can be performed based on the data blocks first acquired to reconstruct the original data to obtain a current reconstruction result corresponding to the original data. This current reconstruction result corresponding to the original data can be stored as the intermediate reconstruction result corresponding to the metadata data in the memory space corresponding to the original data. When data blocks used to reconstruct the original data are acquired again, the intermediate reconstruction result corresponding to the original data can be retrieved from the memory, and data reconstruction can be performed again based on the newly acquired data blocks and the intermediate reconstruction result corresponding to the original data to obtain a new intermediate reconstruction result. The intermediate reconstruction result in the memory space corresponding to the original data can then be updated with the new intermediate reconstruction result. And so on, until the complete original data is reconstructed.As shown in Figure 6, when reconstructing original data 1, assuming that data nodes CS3 and CS4 have already obtained the data blocks used to reconstruct original data 1, while data nodes CS1 and CS2 have not, a memory space can be allocated for original data 1. Assume that this is a memory space of 0MB to 1MB in memory, denoted as memory space 1. Based on the data blocks used to reconstruct original data 1 on data nodes CS3 and CS4, data reconstruction can be performed on original data 1 to obtain the current reconstruction result corresponding to original data 1. This current reconstruction result is then written into memory space 1 as the intermediate reconstruction result corresponding to original data 1. Since data nodes CS3 and CS4 have already sent the data blocks used to reconstruct original data 1, data nodes CS3 and CS4 can continue to send data blocks used to reconstruct original data 2. After obtaining the data blocks used to reconstruct original data 1 on data nodes CS1 and CS2, data reconstruction can be further performed based on the intermediate results in memory space 1 and the data blocks sent by data nodes CS1 and CS2 to obtain the complete original data 1. Similarly, at some point later, assuming that data nodes CS1, CS3, and CS4 have already obtained the data blocks used to reconstruct original data 2, while data node CS2 has not, a memory space of 1 MB to 2 MB can be allocated for original data 2, denoted as memory space 2. Based on the data blocks used to reconstruct original data 2 on data nodes CS1, CS3, and CS4, data reconstruction of original data 2 can be performed to obtain the current reconstruction result corresponding to original data 2. This current reconstruction result of original data 2 is then written into memory space 2 as the intermediate reconstruction result corresponding to original data 2. Since data nodes CS1, CS3, and CS4 have already sent the data blocks used to reconstruct original data 2, data nodes CS1, CS3, and CS4 can continue to send data blocks used to reconstruct original data 3.After obtaining the data blocks for reconstructing the original data 2 on data node CS2, data reconstruction can be further performed based on the intermediate results in memory space 2 and the data blocks sent by data node CS2 to obtain the complete original data 2. Similarly, at some point in the future, assuming that data nodes CS1 have obtained the data blocks for reconstructing the original data 3, while data nodes CS2, CS3, and CS4 have not obtained the data blocks for reconstructing the original data 3, a memory space can be allocated in the memory for the original data 3, assuming that the memory space is 2MB to 3MB in the memory, recorded as memory space 3. Then, based on the data blocks for reconstructing the original data 3 on data node CS1, data reconstruction can be performed on the original data 3 to obtain the current reconstruction result corresponding to the original data 3, and the current reconstruction result corresponding to the original data 3 is written into memory space 3 as the intermediate reconstruction result corresponding to the original data 3. In step S16, assuming that the minimum number of data blocks required to reconstruct the original data is k, if the number of data blocks obtained for reconstructing the original data is less than k, it means that the original data cannot be completely reconstructed at present. Therefore, The process can return to step S12 and continue to acquire data blocks for reconstructing the original data. If the number of acquired data blocks for reconstructing the original data reaches k, it indicates that the original data can now be completely reconstructed. The current reconstruction result of the original data can be determined as the reconstructed original data, and the memory allocated for the original data can be released, thereby reducing memory usage. Continuing with Figure 6, taking original data 1 as an example, if the minimum number k of data blocks for reconstructing original data 1 is 4, then data blocks for reconstructing original data 1 on data nodes CS1, CS2, CS3, and CS4 must be acquired to completely reconstruct original data 1. Assuming that at a certain moment, data blocks for reconstructing original data 1 on data nodes CS1, CS2, and CS3 have been acquired, the process returns to step S12 and continues to acquire data blocks for reconstructing original data 1 on data nodes CS4. If all data blocks used to reconstruct the original data 1 have been acquired on data nodes CS1, CS2, CS3, and CS4, the original data h can be completely reconstructed and the memory allocated for the original data 1 can be released. In some embodiments, if the transmission performance of a data node 104 is continuously poor while the transmission performance of another data node 104 is continuously good, more and more original data may be partially reconstructed.Referring to Figure 7, taking two data nodes CS1 and CS2 as an example, data node CS1 consistently has good transmission performance, while data node CS2 consistently has poor transmission performance. At time t3, data node CS2 has only completed transmitting the data block used to reconstruct original data 1, while data node CS1 has already completed transmitting the data block used to reconstruct original data 3. Data node CS2's transmission progress lags behind data node CS1 by two data blocks. At time t6, data node CS2 has only completed transmitting the data block used to reconstruct original data 2, while data node CS1 has already completed transmitting the data block used to reconstruct original data 6. Data node CS2's transmission progress lags behind data node CS1 by four data blocks. Similarly, at time t8, data node CS2's transmission progress lags behind data node CS1 by five data blocks. If data node 1's transmission is not restricted, the difference between data node CS1's and data node CS2's transmission progress will continue to grow. This requires constantly reserving new memory to store the intermediate reconstruction results obtained for data blocks already transmitted by data node CS1, resulting in excessive memory overhead. To address this issue, the maximum difference in the number of data blocks sent by multiple data nodes 104 can be obtained. If this maximum difference is greater than a first preset threshold, data block requests from the first data node that has sent the most data blocks among the multiple data nodes 104 are stopped, causing the first data node to stop sending data blocks on its own node. Still using data nodes CS1 and CS2 in Figure 7 as an example, if data node CS1 has sent the most data blocks among the multiple data nodes 104, and data node CS2 has sent the fewest data blocks among the multiple data nodes 104, then the number of data blocks sent by data node CS1, p1, and the number of data blocks sent by data node CS2, p2, can be obtained. The maximum difference can be expressed as p1 - p2. If the value of p1 - p2 is greater than the first preset threshold, it indicates that the data node CS1 and the data node CS2 are significantly out of step in their data block transmission progress. To reduce memory usage, the data node CS1 can stop sending data requests. Specifically, it can stop requesting data blocks from CS1, causing CS1 to stop sending data blocks. This allows CS2 to gradually catch up with CS1's data block transmission progress, thereby gradually freeing up the memory occupied by the partially reconstructed original data.Furthermore, after stopping data block requests from the first data node that has sent the most data blocks among the multiple data nodes 104, if the maximum number difference drops from a first preset threshold to below a second preset threshold, data block requests can be continued from the first data node, allowing the first data node to continue sending data blocks on its own node. The second preset threshold is less than the first preset threshold. If the maximum number difference drops from the first preset threshold to below the second preset threshold, this indicates that data node CS2 has gradually caught up with the data block transmission progress of data node CS1 to a certain extent, the memory occupied by the partially reconstructed portion of the original data has been released, and the memory space usage has returned to a tolerable range. Therefore, data requests can be continued to be sent to data node CS1, that is, data block requests can be continued from data node CS1, allowing data node CS1 to continue sending data blocks. In the above embodiment, both the first preset threshold and the second preset threshold can be fixed values ​​set by the user. Alternatively, the first preset threshold and the second preset threshold can be set based on the total memory of data node 104, for example, they can be positively correlated with the total memory of data node 104. Referring to FIG8 , an embodiment of the present disclosure further provides a data reconstruction method, which is applied to a distributed storage system 10 including multiple data nodes 104. Each of the multiple data nodes 104 stores data blocks for reconstructing original data, with each piece of original data being reconstructed from multiple data blocks stored on the multiple data nodes. The method comprises: Step S22: If the node stores a data block for reconstructing the original data, sending the data block to a reconstruction node; wherein the node and other data nodes in the distributed storage system asynchronously send data blocks for reconstructing the same piece of original data, such that upon receiving each data block, the reconstruction node partially reconstructs the original data based on the currently received data block until the original data is reconstructed; and Step S24: If the node is configured to receive the reconstructed original data, obtaining the reconstructed original data returned by the reconstruction node. The method of the present embodiment of the disclosure can be executed by any data node 104 in the distributed storage system 10. In this embodiment, each data node 104 can asynchronously send its own data blocks. Once the previous data block on the node has been sent, the current data block on the node can be sent. Assume that the i-th data block on the data node 104 is used to reconstruct the i-th copy of original data, and the i+1-th data block is used to reconstruct the i+1-th copy of original data.When any data node 104 transmits the (i+1)th data block, it can transmit the (i+1)th data block as soon as the (i)th data block on its own node is transmitted, without having to wait for other data nodes 104 to transmit the data blocks used to reconstruct the (i)th copy of the original data. This improves data block transmission efficiency. After obtaining any data block used to reconstruct the original data, the reconstruction node can perform data reconstruction based on the currently obtained data block and the intermediate reconstruction result of the original data reconstructed from the data block, obtaining the current reconstruction result of the original data reconstructed from the data block. The intermediate reconstruction result of the original data is obtained by partially reconstructing the data blocks previously received by the reconstruction node (i.e., previously sent by each data node 104) for reconstructing the original data. In some embodiments, the multiple data nodes correspond one-to-one to multiple asynchronous queues, and the asynchronous queue corresponding to each data node is used to cache data requests received by the data node, wherein the data requests received by the data node are used to request data blocks stored on the data node. The method further includes: obtaining a data request sent by the reconstruction node; caching the data request in the asynchronous queue corresponding to the node; and sending the data block to the reconstruction node includes: weighting the current data request in the asynchronous queue and sending the data block requested by the current data request to the reconstruction node. In some embodiments, each piece of original data is a data block in the data to be reconstructed, and sending the data block to the reconstruction node includes: sequentially sending the data blocks stored on the node to the reconstruction node according to the position of the original data corresponding to the data block stored on the node in the data to be reconstructed. Specific details of the embodiments of the present disclosure are described in the aforementioned embodiments of the method performed by the reconstruction node and are not further described here. The embodiments of the present disclosure also provide a computer device comprising at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method described in any of the aforementioned embodiments when executing the program. FIG9 shows a more specific hardware structure diagram of a computing device provided by an embodiment of the present disclosure. The device may include a processor 20, a memory 22, an input / output interface 24, a communication interface 26, and a bus 28. The processor 20, the memory 22, the input / output interface 24, and the communication interface 26 are communicatively connected to each other within the device via the bus 28.The processor 20 can be implemented using a general-purpose central processing unit, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure. The processor 20 may also include a graphics card, such as an Nvidia Titan X graphics card or an 1080Ti graphics card. The memory 22 can be implemented using read-only memory (ROM), random access memory (RAM), a static storage device, a dynamic storage device, or the like. The memory 22 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of the present disclosure via software or firmware, the relevant program code is stored in the memory 22 and invoked and executed by the processor 20. The input / output interface 24 is used to connect to an input / output module to implement information input and output. The input / output module can be configured as a component within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., while output devices may include a display, speaker, vibrator, indicator light, etc. The communication interface 26 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can achieve communication via wired means (e.g., USB, network cable, etc.) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth, etc.). The bus 28 includes a path for transmitting information between the various components of the device (e.g., processor 20, memory 22, input / output interface 24, and communication interface 26). It should be noted that although the above device only shows the processor 20, memory 22, input / output interface 24, communication interface 26, and bus 28, in actual implementation, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will appreciate that the above device may only include the components necessary to implement the embodiments of the present disclosure, and does not necessarily include all the components shown in the figures.The present disclosure also provides a distributed storage system 10. Referring to FIG. 1 , the distributed storage system 10 includes a metadata node 102 and multiple data nodes 104. The multiple data nodes 104 store data blocks used to reconstruct original data, and each piece of original data is reconstructed from multiple data blocks stored on the multiple data nodes 104. The metadata node 102 is configured to distribute data reconstruction tasks to reconstruction nodes among the multiple data nodes 104. The data node 104 is configured to, in response to the data reconstruction task, execute the method performed by the reconstruction node in any embodiment of the present disclosure when the node is the reconstruction node, and the method performed by the other data nodes when the node is a data node other than the reconstruction node. The present disclosure also provides a computer program product, including a computer program. When executed by a processor, the computer program implements the method described in any embodiment of the present disclosure. The present disclosure also provides a computer-readable storage medium, storing a computer program. When executed by a processor, the program implements the method described in any of the aforementioned embodiments. Computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves. As can be seen from the above description of the embodiments, those skilled in the art will clearly understand that the embodiments of the present disclosure can be implemented using software and a necessary general-purpose hardware platform.Based on this understanding, the technical solutions of the embodiments of the present disclosure, or the portions that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or portions thereof. The systems, devices, modules, or units described in the above embodiments can be implemented by a computer device or entity, or by a product having certain functions. A typical implementation device is a computer, which can be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices. The various embodiments of the present disclosure are described in a progressive manner. Reference will be made to the common and similar portions of the various embodiments. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are described briefly because they are generally similar to the method embodiments. For relevant details, refer to the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware components. Alternatively, some or all of the modules can be selected based on actual needs to achieve the objectives of the embodiments. Persons of ordinary skill in the art can understand and implement the embodiments without inventive effort. The above description is merely a specific implementation of the embodiments of the present disclosure. It should be noted that persons of ordinary skill in the art can make various improvements and modifications without departing from the principles of the embodiments of the present disclosure. Such improvements and modifications should also be considered within the scope of protection of the embodiments of the present disclosure.

Claims

Claims 1. A data reconstruction method, applied to a distributed storage system comprising multiple data nodes, each data node storing a data block for reconstructing original data, wherein each piece of original data is reconstructed from multiple data blocks stored on the multiple data nodes; the method comprising: Acquire multiple data blocks currently asynchronously sent by the multiple data nodes; Data reconstruction is performed based on each data block and an intermediate reconstruction result of the original data reconstructed by the data block to obtain a current reconstruction result of the original data reconstructed by the data block; the intermediate reconstruction result of the original data is obtained by partially reconstructing the data blocks historically sent by at least one data node for reconstructing the original data; for any piece of original data, if the number of data blocks obtained for reconstructing the original data is less than the minimum number of data blocks required to reconstruct the original data, return to the step of obtaining the multiple data blocks currently sent by the multiple data nodes.

2. The method according to claim 1, further comprising: For any piece of original data, when a data block used to reconstruct the original data is first acquired, memory is allocated for the original data; The original data is reconstructed based on the data block obtained for the first time to obtain a current reconstruction result of the original data. The current reconstruction result of the original data is used as an intermediate reconstruction result of the original data and stored in a memory allocated for the original data.

3. The method according to claim 1 or 2, further comprising: If the number of data blocks obtained for reconstructing the original data reaches the minimum number of data blocks required to reconstruct the original data, the current reconstruction result of the original data is determined as the reconstructed original data, and the memory allocated for the original data is released.

4. The method according to any one of claims 1 to 3, further comprising: Obtaining a maximum number difference of data blocks sent by the plurality of data nodes; If the maximum number difference is greater than a first preset threshold, stop requesting data blocks from a first data node that has sent the most data blocks among the multiple data nodes.

5. The method according to claim 4, further comprising: after stopping requesting data blocks from the first data node that has sent the most data blocks among the plurality of data nodes; If the maximum number difference drops from the first preset threshold to below a second preset threshold, continue to request data blocks from the first data node, and the second preset threshold is less than the first preset threshold.

6. A data reconstruction method, applied to a distributed storage system comprising a plurality of data nodes, wherein each of the plurality of data nodes stores data blocks for reconstructing original data, wherein each piece of original data is reconstructed from a plurality of data blocks stored on the plurality of data nodes; the method comprising: If the node stores a data block for reconstructing the original data, the node sends the data block to the reconstruction node; wherein the node and other data nodes in the distributed storage system asynchronously send data blocks for reconstructing the same original data, so that each time the reconstruction node receives a data block, it partially reconstructs the original data based on the currently received data block until the original data is reconstructed; and if the node is used to receive the reconstructed original data, the node obtains the reconstructed data returned by the reconstruction node. Describe the original data.

7. The method according to claim 6, wherein the plurality of data nodes correspond one-to-one to the plurality of asynchronous queues, the asynchronous queue corresponding to each data node is used to cache data requests received by the data node, and the data requests received by the data node are used to request data blocks stored on the data node; The method further includes: obtaining a data request sent by the reconstruction node; caching the data request in an asynchronous queue corresponding to the node; and sending the data block to the reconstruction node, including: weighting the current data request in the asynchronous queue, and sending the data block requested by the current data request to the reconstruction node.

8. The method according to claim 6, wherein each piece of original data is a data block in the data to be reconstructed, and sending the data block to the reconstruction node comprises: The data blocks stored on the node are sent to the reconstruction node in sequence according to the positions of the original data corresponding to the data blocks stored on the node in the data to be reconstructed.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 9 when executing the program.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 9.

12. A distributed storage system comprising a metadata node and a plurality of data nodes, wherein the plurality of data nodes store data blocks for reconstructing original data, and each piece of original data is reconstructed from a plurality of data blocks stored on the plurality of data nodes; The metadata node is used to distribute data reconstruction tasks to a reconstruction node among the multiple data nodes; the data node is used to, when the node is a reconstruction node, respond to the data reconstruction task and execute the method described in any one of claims 1 to 5, and, when the node is a data node other than the reconstruction node, execute the method described in any one of claims 6 to 8.

Images