A data processing method, apparatus and medium

Abstract

The application discloses a data processing method, device and medium, and relates to the technical field of data storage. According to an initial coupling structure corresponding to a multi-layer coupling code, data blocks and check blocks obtained after division of to-be-processed data are encoded to obtain sub-blocks; the initial coupling structure is partitioned, and a unit coupling structure is used to replace the initial coupling structure of each partition in an enumeration manner; the unit coupling structure after replacement is used to synchronously calculate read data average time overhead for recovering a single block, and a coupling structure with minimum time overhead is selected as a target coupling structure; the sub-blocks are coupled and encoded according to the target coupling structure to be stored in a corresponding storage device; when the storage device is invalid, the sub-blocks required for decoding are read according to the target coupling structure to recover data. The application takes actual time consumption of recovered data as a screening basis, reduces non-continuous reading, reduces read data time overhead for recovering invalid data, and improves decoding efficiency.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data processing method, apparatus and medium. Background Technology

[0002] Traditional Redundant Array of Independent Disks Level 6 (RAID 6) encoding requires reading all data from the remaining disks when repairing a single failed disk, resulting in significant network transmission overhead. To address this, minimal storage regeneration codes, such as Clay codes, divide each stripe cell into multiple sub-blocks and encode them in pairs according to a coupled structure, reducing the amount of data read during the repair of a single stripe cell. However, due to the non-contiguous input / output characteristics of hard disk drives (HDDs), decoding and recovering partial stripe cells requires reading a large number of non-contiguous sub-blocks from each HDD. Furthermore, as the number of data disks increases, the number of non-contiguous reads increases dramatically. Since the random read performance of HDDs is far inferior to its sequential read performance, this leads to significant data read latency during decoding and recovery, resulting in low decoding efficiency.

[0003] Therefore, how to reduce the time overhead during decoding and recovery to improve decoding efficiency is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a data processing method, apparatus, and medium to solve the problems of extremely high data reading time overhead and low decoding efficiency when decoding and recovering data.

[0005] To address the aforementioned technical problems, this application provides a data processing method, comprising: Based on the initial coupling structure corresponding to the multi-layer coupled code, the data blocks and check blocks after the data to be processed are divided are encoded to obtain each sub-block; The initial coupling structure is partitioned, and the initial coupling structure of each partition is replaced by a unit coupling structure in an enumerated manner; The average time cost of reading data for a single block is calculated synchronously for the replaced unit coupling structure, and the coupling structure with the minimum time cost is selected as the target coupling structure; each sub-block is coupled and encoded according to the target coupling structure and stored in the corresponding storage device. If the storage device fails, the data of the failed storage device is recovered by reading the sub-blocks required for decoding from the remaining storage devices according to the target coupling structure.

[0006] On the one hand, the unit coupling structure includes at least one or more of the following: unidirectional structure, cross structure, and symmetrical structure; The unit coupling structure acts on an even number of paired nodes, each pair of nodes including a left node and a right node, and the pairs of nodes are arranged in sequence. The unidirectional structure is a coupled structure in which the left node in each pair of nodes is connected to the right node in the next adjacent pair of nodes along the same direction, and the left node in the last pair of nodes is connected to the right node in the first pair of nodes; the number of pairs of nodes in the unidirectional structure is an integer greater than one. The cross structure consists of the left node in each pair of nodes being connected to the right node in a pair of nodes in the same direction, forming a coupled structure in which adjacent connections intersect; the number of pairs of nodes in the cross structure is an integer greater than two. The symmetrical structure is a coupled structure in which the left node of the first pair of nodes is connected to the right node of the last pair of nodes, and the left node of the last pair of nodes is connected to the right node of the first pair of nodes, and the two pairs of nodes in symmetrical positions are connected to each other from the outside to the inside; the number of pairs of nodes in the symmetrical structure is an even number greater than two.

[0007] On the other hand, partitioning the initial coupling structure includes: Each block is divided into multiple groups in order, and each sub-block layer in each group is divided into the corresponding first partition in order. Correspondingly, the initial coupling structure of each partition is replaced by an enumerated unit coupling structure, including: Starting from the last group, the coupling structures in each of the first partitions are replaced one by one by the unit coupling structures in an enumerated manner. Two adjacent sibling partitions are sequentially combined into the next level partition and replaced, until the combined partition covers all sub-block layers within the group.

[0008] On the other hand, two adjacent sibling partitions are sequentially combined into the next level partition and replaced, until the combined partition covers all sub-block layers within the group, including: After the replacement traversal of each of the first partitions in the group is completed, two adjacent first partitions are combined in sequence to form a second partition, and the coupling structure in each of the second partitions is replaced one by one by the unit coupling structure in an enumeration manner. Adjacent sibling partitions are sequentially combined into the next level partition and replaced, until the combined partition covers all sub-block layers within the group.

[0009] On the other hand, the average time cost of recovering read data for a single block is calculated synchronously for the replaced unit coupling structure, including: Obtain the addressing latency cost and unit data transfer time cost of the storage device; Based on the addressing delay time cost and the unit data transmission time cost, determine the read data time overhead when recovering each block separately; The average read data time cost of each block is determined as the average read data time cost.

[0010] On the other hand, determining the read data time overhead for recovering each block based on the addressing delay time cost and the unit data transmission time cost includes: Each block is identified as a block to be recovered in turn; Obtain the current block to be recovered, and determine the sub-blocks required for decoding when recovering the current block to be recovered, as well as the distribution locations of the sub-blocks required for decoding on the other storage devices, based on the replaced coupling structure; Obtain the first storage device from the remaining storage devices; according to the distribution location, group the consecutive sub-blocks in the sub-blocks required for decoding on the first storage device into the same consecutive read segment to obtain at least one consecutive read segment; The number of consecutive read segments is determined as the number of discrete reads of the first storage device; The amount of data read by the first storage device is determined based on the amount of data in the sub-blocks contained in each of the consecutive read segments; The addressing time overhead of the first storage device is determined based on the number of discrete reads and the addressing latency time cost. The transmission time overhead of the first storage device is determined based on the amount of data read and the unit data transmission time cost. The read time cost of the first storage device is determined based on the addressing time cost and the transmission time cost. The read time overhead of each of the other storage devices is summed to obtain the read data time overhead when recovering the block to be recovered.

[0011] On the other hand, recovering data from the failed storage device by reading the sub-blocks required for decoding from the remaining storage devices according to the target coupling structure includes: Determine the target sub-block layer containing the sub-blocks in the failed data block that were not involved in coupled coding; Read all sub-blocks in the target sub-block layer of the remaining storage devices, and sub-blocks that are in the same coupling structure as the sub-blocks in the target sub-block layer and are located in other sub-block layers, and use them as the target sub-blocks; The target sub-blocks are coupled and verified to recover the data in the failed storage device by restoring the sub-blocks required for decoding.

[0012] On the other hand, coupling encoding of each sub-block according to the target coupling structure for storage to the corresponding storage device includes: If this is the first encoding, the target coupling structure is recorded so that each sub-block can be coupled and encoded according to the target coupling structure; If it is not the first encoding, then the recorded target coupling structure is obtained; Coupled sub-blocks are obtained by coupling encoding two mutually coupled sub-blocks according to the target coupling structure; Each coupled sub-block corresponding to the same block is stored sequentially to the corresponding storage device.

[0013] To address the aforementioned technical problems, this application also provides a data processing apparatus, comprising: Memory, used to store computer programs; A processor for implementing the steps of the data processing method as described above when executing the computer program.

[0014] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method described above.

[0015] The beneficial effect of this application lies in providing a data processing method. First, based on the initial coupling structure corresponding to the multi-layer coupling code, the data blocks and check blocks after dividing the data to be processed are encoded to obtain each sub-block. When recovering a single failed block, only a portion of the sub-blocks in the remaining blocks need to be read, without reading all the data. This minimizes the amount of data read for recovery without increasing redundant storage space, balancing storage efficiency and fault tolerance. Second, the initial coupling structure is partitioned, and the initial coupling structure of each partition is replaced by a unit coupling structure using an enumeration method. This makes the initial coupling structure itself adjustable and optimizable, rather than using a fixed coupling relationship. Furthermore, by using an enumeration method for the unit coupling structure, which is not infinitely enumerated, the computational load is reduced. Furthermore, the average read time cost for recovering a single block is calculated synchronously for the replaced coupling structure, and the coupling structure with the lowest time cost is selected as the target coupling structure. The actual data recovery time is used as the direct basis for coupling structure selection, avoiding the problems of large-scale non-continuous reads and high read latency in conventional solutions. Moreover, the average read time cost is related to parameters such as the addressing latency and data transfer rate of the storage device, allowing for the selection of a matching target coupling structure for storage devices with different performance characteristics. Further, each sub-block is coupled and encoded according to the target coupling structure and stored in the corresponding storage device. The encoded data is directly organized and stored according to the structure with the optimal time cost. Subsequent failure recovery can directly utilize this target coupling structure, eliminating the need for data rearrangement or structural adjustments during the recovery phase and preventing the introduction of additional overhead. Finally, when a storage device fails, the data of the failed storage device is recovered by reading the sub-blocks required for decoding from the other storage devices according to the target coupling structure. The location of the sub-blocks to be read is directly determined by the target coupling structure, ensuring accurate positioning. The amount of data read and the number of discrete reads are both in a combination with the minimum time overhead. While ensuring complete data recovery, the time overhead of data recovery is shortened, and the decoding and recovery efficiency of the storage system is improved.

[0016] Secondly, the three unit coupling structures cover the typical forms of coupling pairing relationships with a limited number of basic structural types, providing a limited candidate set for enumeration and replacement. On the one hand, this ensures that the replacement process can be exhaustively traversed and the screening results can be reproduced. On the other hand, different unit coupling structures correspond to different decoding and reading modes, providing a structural basis for selecting coupling structures with more continuous reading and lower time overhead under different storage device parameters. The method of pairwise grouping and dividing the first partition by sub-block layer is adapted to the construction characteristics of pairwise coupling of multi-layer coupling codes, making the replacement within each partition a structurally self-consistent local replacement. The search for the global coupling structure is decomposed into the enumeration of a limited number of unit coupling structures within each partition, converging what could have been an exponential global search into a local traversal within the partition. This ensures that the coupling structure with the minimum time overhead can be selected while controlling the computational scale of the screening process and reducing computational resources. The stepwise combination replacement method allows the screening process to cover various coupling structure combinations from fine-grained to coarse-grained. The replacement of fine-grained partitions ensures fine adjustment of local reading continuity, while the replacement of coarse-grained partitions ensures overall structural optimization across sub-block layers. The reading time of the storage device is determined by two parameters: address latency time cost and unit data transfer time cost. The average time cost of each block as the recovery object is calculated, so that the screening target of the coupled structure changes from indirect indicators such as the amount of data read and the number of reads to the direct indicator of the actual time spent recovering data. The measurement result is consistent with the actual reading behavior of mechanical hard drives.

[0017] In addition, this application also provides a data processing apparatus and medium that have the same beneficial effects as the data processing method described above. Attached Figure Description

[0018] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the need to read sub-blocks when recovering a single block after encoding a multi-layer coupling code according to an initial coupling structure, as provided in an embodiment of this application. Figure 3 A schematic diagram illustrating the need to read sub-blocks when recovering a single block after encoding a multi-layer coupling code according to the initial coupling structure, as provided in an embodiment of this application. Figure 4 This is a schematic diagram of an optimized scheme for recovering a single block in a conventional scheme, which uses Gray sequence rearrangement of sub-blocks. Figure 5 This is a schematic diagram illustrating an optimized scheme for recovering a single block by rearranging sub-blocks using the Gray sequence in the conventional scheme. Figure 6 This is a schematic diagram of a unit coupling structure with M=2 provided in an embodiment of this application; Figure 7 This application provides a schematic diagram of a unit coupling structure with M=4 as an embodiment. Figure 8 This application provides a schematic diagram of a coupling structure selected after replacement with a unit coupling structure; Figure 9 This application provides a schematic diagram of the sub-blocks that need to be read when recovering a single stripe unit from failure. Figure 10 A structural diagram of a data processing apparatus provided in an embodiment of this application; Figure 11 This is a structural diagram of another data processing apparatus provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0021] The core of this application is to provide a data processing method, apparatus, and medium to solve the problems of extremely high data reading time overhead and low decoding efficiency during data decoding and recovery.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] RAID 6 is a disk array encoding technique with dual parity, employing two different forms of parity codes. RAID 6 requires at least K+2 disks (where K represents the number of data disks), with two disks used to store the first and second parity codes respectively, enabling recovery of data from any two failed disks. Traditional RAID 6 encoding methods require reading all data from the remaining disks to repair a single failed disk, resulting in significant network transmission overhead. Minimum Storage Regenerating codes (MSR codes) were proposed as an encoding / decoding method that achieves the lowest possible data transfer during single-disk repair. Clay codes are currently the most advanced known MSR codes.

[0024] Clay codes are constructed based on paired coupling between sub-blocks in different stacking layers. The encoding steps of Clay codes are as follows: a) For a RAID stripe with an even number of K data units, each stripe unit is divided into multiple sub-blocks in sequence; b) All stripe units are divided into multiple sub-block layers, and the K sub-blocks within each layer are encoded using RAID 6 encoding, usually using (K+2, K) Reed-Solomon code (RS code), to obtain two parity sub-blocks for that sub-block layer; c) All sub-blocks in all sub-block layers after RAID 6 encoding are divided into coupling structures in order of arrangement, and the two coupled sub-blocks are encoded using coupling codes to obtain the coupled sub-blocks after coupling calculation.

[0025] When it is necessary to decode and recover a single stripe unit, Clay code only needs to read half of the remaining stripe unit sub-blocks to complete the decoding. The required decoding sub-block layer is determined by the coupling structure of the stripe unit to be recovered.

[0026] To minimize storage overhead during repair, Clay code divides each stripe unit into multiple sub-blocks and couples the calculations, requiring the reading of a large number of non-sequentially placed sub-blocks during the repair of a single stripe unit. For RAID 6 arrays composed of hard disk drives (HDDs), this will severely impact decoding and repair performance.

[0027] Clay code does not fully consider the input / output (I / O) characteristics of HDD disks, and cannot achieve the optimal decoding and recovery efficiency (from a time overhead perspective) for HDD disks forming a RAID array. Especially for larger K values, the decoding process for recovering some nodes requires reading multiple non-contiguous sub-blocks from all HDD disks; for example, when the number of data units in a RAID 6 stripe is K=14, in the worst case, Clay code decoding and recovery of a single stripe unit requires reading 128 non-contiguous sub-blocks from all other HDD disks. Since HDD disks have very poor read performance in non-contiguous I / O scenarios, this will cause a significant data read time delay during decoding and recovery. Therefore, the data processing method provided in this application can solve the above-mentioned technical problems.

[0028] To facilitate understanding of the technical solution of this application, the terms involved in this embodiment will be explained first.

[0029] Storage devices refer to the physical devices used in a storage system for persistently storing data. These can be HDD disks, and multiple storage devices can form a redundant array, such as a RAID 6 array consisting of K data disks and 2 parity disks, capable of tolerating the failure of any two storage devices. This application is particularly applicable to large-scale arrays composed of mechanical hard drives (HDDs) designed for cold data storage scenarios. The total time spent reading data from an HDD consists of three parts: seek time, rotational latency, and data transfer time. For a single read of a small amount of data, the time overhead is mainly seek time and rotational latency; therefore, the random read performance of an HDD is far inferior to its sequential read performance.

[0030] Data blocks and parity blocks: After the data to be processed is organized into a stripe, the K original data obtained from the division each correspond to a data block (i.e., the data stripe unit in the stripe); the parity block is the block corresponding to the redundant data obtained after performing parity encoding on the data block (i.e., the parity stripe unit in the stripe). RAID 6 stripes contain two parity blocks, and the data blocks and parity blocks total K+2 blocks, which are stored in the corresponding K+2 storage devices.

[0031] Sub-blocks and sub-block layers: Each block is divided into smaller granular data units, called sub-blocks; sub-blocks in the same order across all blocks constitute the same sub-block layer. Sub-block partitioning is a prerequisite for constructing multi-layer coupled codes, enabling the recovery of a single block to be achieved by reading only a portion of the sub-blocks from the remaining blocks, rather than all the data.

[0032] Multilayer coupled codes, also known as Clay codes, are a type of minimal storage regeneration code. Their construction is based on paired coupling between sub-blocks in different sub-block layers. Each sub-block layer uses parity coding such as (K+2, K)RS codes, and two sub-blocks across different layers are paired according to a coupling structure and then encoded using coupled codes. Multilayer coupled codes have been theoretically proven to have the minimum data read volume when decoding and recovering a single block.

[0033] Initial coupling structure: refers to the coupling structure originally used by the multilayer coupling code to determine the pairing relationship between each sub-block. The initial coupling structure determines which sub-block layers and additional sub-blocks need to be read when recovering a certain block.

[0034] Unit coupling structure: refers to the basic coupling structure unit used to replace the initial coupling structure. The embodiments of this application provide three types of unit coupling structures: unidirectional structure, cross structure and symmetrical structure.

[0035] Coupled coding: refers to the coding operation of jointly calculating two paired sub-blocks according to the coupled structure to obtain coupled sub-blocks. Its inverse process is coupled decoding. Coupled coding makes it possible to deduce the remaining sub-blocks from some sub-blocks by means of the coupling relationship when recovering a single block.

[0036] Average read time cost of a single block: refers to the average time cost of reading the sub-blocks required for decoding from the remaining storage devices when each block in the stripe is used as the recovery object; this time cost is composed of the product of the number of discrete reads and the addressing latency time cost, and the product of the amount of data read and the unit data transfer time cost.

[0037] Figure 1 A flowchart of a data processing method provided in an embodiment of this application is shown below. Figure 1 As shown, the method includes: S11: Based on the initial coupling structure corresponding to the multi-layer coupling code, the data blocks and check blocks after the data to be processed are divided are encoded to obtain each sub-block; S12: Divide the initial coupling structure into partitions and replace the initial coupling structure of each partition with a unit coupling structure in an enumerated manner; S13: Simultaneously calculate the average time cost of reading data for a single block after the replacement unit coupling structure, and select the coupling structure with the minimum time cost as the target coupling structure; perform coupling encoding on each sub-block according to the target coupling structure, and store it to the corresponding storage device; S14: If a storage device fails, the data of the failed storage device is recovered by reading the sub-blocks required for decoding from the remaining storage devices according to the target coupling structure.

[0038] Specifically, in step S11, the upper-layer application sends out K data sets to be stored. The RAID 6 storage system first organizes the K data sets into a RAID 6 stripe, where the K data sets correspond to K data stripe units. For all K data stripe units, each stripe unit is sequentially divided into... Sub-blocks. Sub-blocks in the same order within all blocks constitute the same sub-block layer; for each sub-block layer, K sub-blocks are subjected to (K+2, K) RS code for check encoding, resulting in two check sub-blocks for that sub-block layer. The check sub-blocks of all sub-block layers thus constitute two check blocks. Taking K=4 as an example, each stripe contains K+2=6 blocks, sequentially labeled C0 to C5, and each block is divided into multiple sub-blocks. At this point, the initial coupling structure determines the pairing relationship between sub-blocks in each sub-block layer.

[0039] Figure 2 This is a schematic diagram illustrating the need to read sub-blocks when recovering a single block after encoding a multi-layer coupling code according to an initial coupling structure, as provided in an embodiment of this application. Figure 3 A schematic diagram illustrating the need to read sub-blocks when recovering a single block after encoding a multi-layer coupling code according to the initial coupling structure, as provided in the embodiments of this application, is shown below. Figure 2 , Figure 3 As shown, the nodes marked by the grid are the data to be recovered, and the other marked nodes are the sub-blocks to be read for decoding. Figure 4 This diagram illustrates an optimized scheme for recovering a single block using Gray sequence rearrangement of sub-blocks within a conventional approach. Figure 5 This is a schematic diagram illustrating an optimized scheme for recovering a single block using a Gray sequence rearrangement of sub-blocks within the conventional approach, as shown below. Figure 4 , 5 As shown, in order to achieve the minimum amount of data read during repair, the multi-layer coupling code divides each stripe unit into multiple sub-blocks and performs coupled calculations, which means that a large number of non-sequentially placed sub-blocks need to be read during the repair of a single stripe unit. Figure 2 , 3 As can be seen from points 4 and 5, regardless of the initial coupling structure or the Gray sequence rearrangement scheme, the sub-blocks that need to be read when recovering a partial block are not discontinuously distributed on the storage device. This is precisely the problem that this application aims to optimize in subsequent steps.

[0040] In step S12, the initial coupling structure is divided into multiple partitions according to the grouping of blocks and the order of sub-block layers. Then, various unit coupling structures are used to replace the initial coupling structure in each partition one by one in an enumerated manner, thereby generating a series of candidate coupling structures based on the initial coupling structure. Through partition replacement, the coupling structure itself becomes an adjustable and optimizable object; by limiting the types of unit coupling structures for enumeration, the search space for replacement is finite, avoiding infinite enumeration and reducing the amount of computation. Regarding the enumeration method, it is possible to first target the large partitions and then gradually narrow down to the corresponding unit coupling structures under the smallest partitions, or to first target the smallest partitions and then gradually expand to the corresponding unit coupling structures under the largest partitions. Different unit coupling structures are set for replacement in each partition to realize the candidate coupling structures for each partition.

[0041] In step S13, after each replacement, the average time cost of recovering a single block of data is calculated for the replaced coupling structure. After traversing all feasible replacements, the coupling structure with the minimum average time cost of data read is determined as the target coupling structure. Since the average time cost of data read is determined by the addressing latency cost and the unit data transfer time cost of the storage device, the actual time spent recovering data is used as the screening criterion, so that the selected target coupling structure matches the actual read characteristics of the storage device. For storage devices of different models and with different performance parameters, a suitable target coupling structure can be selected.

[0042] In step S14, according to the pairing relationship determined by the target coupling structure, the two mutually coupled sub-blocks are encoded using a coupling code to obtain coupled sub-blocks; the sub-blocks corresponding to the same block are written sequentially to the corresponding storage devices. Thus, the encoded data is directly organized and stored according to the target coupling structure with optimal time overhead, and this structure can be directly used for each subsequent failure recovery, without the need for data rearrangement or structural adjustment during the recovery phase.

[0043] In step S15, when a single storage device fails, the sub-blocks to be read and their locations in the remaining storage devices can be directly determined based on the recorded target coupling structure. After reading, the data in the failed storage device is recovered through coupling decoding and verification decoding. Since the location of the sub-block to be read is directly determined by the target coupling structure, the positioning is accurate, and the amount of data read and the number of discrete reads are both in the combination with the minimum time cost determined in the screening stage. This ensures complete data recovery while shortening the time cost of data recovery.

[0044] This application provides a data processing method. First, based on the initial coupling structure corresponding to the multi-layer coupling code, the data blocks and check blocks after dividing the data to be processed are encoded to obtain sub-blocks. When recovering a single failed block, only a portion of the sub-blocks in the remaining blocks need to be read, without reading all the data. This minimizes the amount of data to be read for recovery without increasing redundant storage space, balancing storage efficiency and fault tolerance. Second, the initial coupling structure is partitioned, and the initial coupling structure of each partition is replaced by a unit coupling structure using an enumeration method. This makes the initial coupling structure itself adjustable and optimizable, rather than using a fixed coupling relationship. Furthermore, by enumerating the unit coupling structure, the computational load is reduced, rather than infinitely. Furthermore, the average read time cost for recovering a single block is calculated synchronously for the replaced coupling structure, and the coupling structure with the lowest time cost is selected as the target coupling structure. The actual data recovery time is used as the direct basis for coupling structure selection, avoiding the problems of large-scale non-continuous reads and high read latency in conventional solutions. Moreover, the average read time cost is related to parameters such as the addressing latency and data transfer rate of the storage device, allowing for the selection of a matching target coupling structure for storage devices with different performance characteristics. Further, each sub-block is coupled and encoded according to the target coupling structure and stored in the corresponding storage device. The encoded data is directly organized and stored according to the structure with the optimal time cost. Subsequent failure recovery can directly utilize this target coupling structure, eliminating the need for data rearrangement or structural adjustments during the recovery phase and preventing the introduction of additional overhead. Finally, when a storage device fails, the data of the failed storage device is recovered by reading the sub-blocks required for decoding from the other storage devices according to the target coupling structure. The location of the sub-blocks to be read is directly determined by the target coupling structure, ensuring accurate positioning. The amount of data read and the number of discrete reads are both in a combination with the minimum time overhead. While ensuring complete data recovery, the time overhead of data recovery is shortened, and the decoding and recovery efficiency of the storage system is improved.

[0045] In some embodiments, the unit coupling structure includes at least one or more of the following: unidirectional structure, cross structure, and symmetrical structure; In this structure, the unit coupling structure acts on an even number of paired nodes, each pair of nodes including a left node and a right node, and the nodes in each pair are arranged in sequence. A unidirectional structure is a coupled structure in which the left node in each pair of nodes is connected to the right node in the next adjacent pair of nodes in the same direction, and the left node in the last pair of nodes is connected to the right node in the first pair of nodes; the number of pairs of nodes in a unidirectional structure is an integer greater than one. A cross structure is a coupled structure in which the left node of each pair of nodes is connected to the right node of a pair of nodes in the same direction, forming an intersecting connection between adjacent nodes; the number of pairs of nodes in a cross structure is an integer greater than two. A symmetrical structure is a coupled structure in which the left node of the first pair of nodes is connected to the right node of the last pair of nodes, and the left node of the last pair of nodes is connected to the right node of the first pair of nodes, and the two pairs of nodes in symmetrical positions are connected to each other from the outside to the inside; the number of pairs of nodes in a symmetrical structure is an even number greater than two.

[0046] Specifically, node pairing involves combining all nodes into pairs. In each pair, the left node becomes the left node, and the right node becomes the right node. This explains... Figure 2 There are only connections between C0-C1, C2-C3, and C4-C5; there are no connections between pairs (such as C1-C2).

[0047] The coupling structure is drawn at the sub-block level. In a K=4 Clay code, each node has 8 sub-blocks (R0~R7), so within a node pair: the 8 sub-blocks in the left column are denoted as... The 8 sub-blocks in the right column are denoted as .

[0048] For 2M pairs of nodes, they are sequentially denoted as follows: arrive ; and represent the left node as The right-hand node is represented as The connection method of the unidirectional structure is as follows: connect ,or connect ,in i The value ranges from 0 to M-1, where M is any integer greater than 1. The connection method of the cross structure is as follows: connect ,or connect ,in i The value ranges from 0 to M-1, where M is any integer greater than 2. The connection method for symmetrical structures is as follows: connect , connect ,or connect , connect Where M is any even number greater than 2, i Values ​​range from 0 to . Figure 6 This application provides a schematic diagram of a unit coupling structure with M=2, as shown in the embodiment. Figure 6 As shown, from left to right, they are unidirectional structure, cross structure, and symmetrical structure.

[0049] Taking M=4 as an example, Figure 7 This application provides a schematic diagram of a unit coupling structure with M=4, as shown in the embodiment. Figure 7 As shown, from left to right, the structures are unidirectional, intersecting, and symmetrical. Unidirectional structure (i=0~M-1=0~3, connecting...) ): That is, the top 4 rows of the left column are shifted 4 rows to connect to the bottom 4 rows of the right column, and the 4 connecting lines are parallel and in the same direction.

[0050] Cross structure (i=0~M-1=0~3, connection) ): That is, the lines connecting row 0 to row 7, row 1 to row 6, and so on, all intersect at the center, with spans of 7, 5, 3, and 1 rows respectively.

[0051] Symmetrical structure (M / 2=2, 3M / 2=6, i=0~M / 2-1=0~1): Upper part: lower half: Specifically, this includes: That is: the upper 4 rows are shifted 2 rows internally and connected (to the lower right), and the lower 4 rows are mirrored and connected (to the lower left), making them symmetrical.

[0052] It's important to note that the essence of the three structures lies in the row distribution of the coupled sub-blocks on the disk, which directly determines the number of discrete reads corresponding to the time overhead formula. In a unidirectional structure, the coupled sub-blocks are concentrated in a single block (top 4 rows of the left column / bottom 4 rows of the right column), making it easiest for the sub-blocks to be read during recovery to form a continuous region. x =1, only one seek overhead P is incurred. Symmetrical structure, coupled sub-blocks are divided into upper and lower segments (lines 0-1 and 6-7, lines 2-3 and 4-5) → Reading is usually divided into two segments, i.e. x =2. The cross-connection structure, with a connection span ranging from 1 to 7 rows, requires the most scattered sub-block row numbers for recovery, resulting in the highest number of discrete reads. Different node pairs employ varying degrees of coupling tightness, causing the seek count to double pairwise. These three optional unit coupling structures are proposed to reduce the product of discrete read counts and rotational latency, thereby lowering the average repair time, while maintaining decoding capability, a constant data read volume (y), and 4 units per disk.

[0053] The three unit coupling structures provided in this embodiment cover the typical forms of coupling pairing relationships with a limited number of basic structural types, providing a limited candidate set for enumeration and replacement. On the one hand, this ensures that the replacement process can be exhaustively traversed and the screening results can be reproduced. On the other hand, different unit coupling structures correspond to different decoding and reading modes, providing a structural basis for screening coupling structures with more continuous reading and lower time overhead under different storage device parameters.

[0054] In some embodiments, partitioning the initial coupling structure includes: Each block is divided into multiple groups in order, and each sub-block layer in each group is divided into the corresponding first partition in order. Correspondingly, the initial coupling structure of each partition is replaced by an enumerated unit coupling structure, including: Starting from the last group, the coupling structure in each first partition is replaced one by one by the coupling structure of each unit in an enumerated manner. The two adjacent sibling partitions are combined sequentially to form the next level partition and then replaced, until the combined partition covers all sub-block layers within the group.

[0055] Specifically, each block is divided into multiple groups in pairs according to the order. For example, when K=4, C0 and C1 are in one group, C2 and C3 are in another, and C4 and C5 are in yet another. Then, each sub-block layer within each group is divided into corresponding first partitions according to the order. Each first partition covers the coupling structures on several consecutive sub-block layers within the group. Three basic coupling structure replacements are attempted for each first partition. Next, two adjacent first partitions are combined into two second partitions (each containing 4 layers), and the three basic coupling structure replacements are attempted for each second partition. Finally, two adjacent second partitions are combined into one third partition (containing 8 layers), and the three basic coupling structure replacements are attempted for the third partition.

[0056] By employing a hierarchical partitioning and progressive merging enumeration method, all possible replacement schemes for the initial coupling structure of traditional Clay codes are systematically traversed. Starting with the smallest partition and then gradually merging it into larger partitions, this fine-to-coarse enumeration strategy ensures that all feasible replacement schemes for the coupling structure are covered without omission.

[0057] The pairwise grouping and sub-block layer division of the first partition provided in this embodiment are compatible with the construction characteristics of pairwise coupling of multi-layer coupled codes, so that the replacement in each partition is a structurally self-consistent local replacement; the search of the global coupling structure is decomposed into the enumeration of a finite number of unit coupling structures in each partition, and the originally potentially exponential global search is converged into a local traversal within the partition. While ensuring that the coupling structure with the minimum time cost can be selected, the computational scale of the selection process is controlled and computational resources are reduced.

[0058] In some embodiments, two adjacent sibling partitions are sequentially combined into a next-level partition and replaced until the combined partition covers all sub-block layers within the group, including: After the replacement traversal of each first partition within the group is completed, two adjacent first partitions are combined in order to form a second partition. The coupling structure in each second partition is replaced one by one by the coupling structure of each unit in an enumeration manner. Adjacent sibling partitions are sequentially combined into the next level partition and replaced, until the combined partition covers all sub-block layers within the group.

[0059] Specifically, after the replacement traversal of each first partition within a group is completed, adjacent first partitions are sequentially combined into second partitions. The coupling structures in each second partition are then replaced one by one using an enumerated approach with each unit coupling structure. Subsequently, adjacent sibling partitions are sequentially combined into next-level partitions and the replacement is performed, until the resulting partitions cover all sub-block layers within the group. Since the node scale covered by the next-level partition is twice that of the previous-level partition, the number of node pairs in the unit coupling structure used for replacement is also increased accordingly. That is, different scales of unidirectional structures, cross structures, and symmetrical structures are used to adapt to different levels of partitions.

[0060] Taking K=4 and each block divided into 8 sub-blocks as an example, a group is first divided into several first partitions according to the sub-block layer order and then enumerated and replaced one by one. After the first partition is traversed, two adjacent first partitions are combined into a second partition, and the unit coupling structure with more node pairs is used to continue enumerating and replacing. This process is repeated level by level until the partition covers all 8 sub-block layers of the group, and then the process is repeated in the next group.

[0061] The step-by-step replacement method provided in this embodiment enables the screening process to cover various coupling structure combinations from fine-grained to coarse-grained. The replacement of fine-grained partitions ensures fine adjustment of local read continuity, while the replacement of coarse-grained partitions ensures overall structural optimization across sub-block layers.

[0062] In some embodiments, the average time cost of recovering a single block of read data is synchronously calculated for the replaced cell coupling structure, including: Obtain the addressing latency cost and unit data transfer time cost of the storage device; The read data time cost for recovering each block is determined based on the addressing delay time cost and the unit data transfer time cost. The average read data time cost for each block is determined as the average read data time cost.

[0063] Specifically, the physical meaning and derivation of the two time cost parameters are as follows: The total time for a hard disk drive to read data consists of three parts: seek time, rotational latency, and data transfer time. The sum of the seek time and rotational latency of a single read operation on the storage device is denoted as the address latency time cost. P The time taken to read a unit size of data is denoted as the unit data transfer time cost. R The total read time for the storage device to continuously read y units of data at a time is: P + y × RFor single read operations involving small amounts of data, the time overhead primarily consists of seek time and rotational latency. Let the seek time and rotational latency of a single HDD read operation be denoted as... P The time to read one unit of data is denoted as R .

[0064] The bandwidth performance of sequential read (one addressing plus multiple consecutive data reads) is 255MB / s, and the performance of random read (one addressing plus 4kB data reads) is 150 input / output operations per second. Based on the random read performance, the total time for a single random read is approximately 1000ms ÷ 150 ≈ 6.67ms. The transmission time of the 4kB data is negligible, so the addressing latency cost P is approximately 6.60ms. Dividing 128kB into units, the unit data transmission time cost R, based on the sequential read bandwidth, is approximately 128kB ÷ 255MB / s ≈ 0.49ms.

[0065] This invention relates to a multilayer coupled code encoding method that achieves optimal average time cost for decoding and recovering a single stripe unit, given a seek delay time cost P and a data transmission time cost R. Furthermore, given P and R, and without subsequent modifications, the process of finding the time-optimal coupling structure described in this application only needs to be executed once; once the optimal coupling structure is determined, it does not need to be executed again during each batch of stripe data encoding.

[0066] This embodiment uses two parameters—address latency cost and unit data transfer time cost—to determine the read time of a storage device, and averages the time cost of recovering each block. This transforms the selection target of the coupled structure from indirect indicators such as the amount of data read and the number of reads to the direct indicator of the actual time spent recovering data. The measurement results are consistent with the actual read behavior of a mechanical hard drive.

[0067] In some embodiments, determining the read data time overhead for recovering each block separately, based on the addressing latency time cost and the unit data transfer time cost, includes: Each block is identified as a block to be recovered in turn; Obtain the current block to be recovered, and determine the sub-blocks required for decoding when recovering the current block to be recovered, as well as the distribution of the sub-blocks required for decoding on the other storage devices, based on the replaced coupling structure; Obtain the first storage device from the remaining storage devices; according to the distribution location, group the consecutive sub-blocks in the sub-blocks required for decoding on the first storage device into the same consecutive read segment to obtain at least one consecutive read segment; The number of consecutive read segments is determined as the number of discrete reads of the first storage device; The amount of data read from the first storage device is determined based on the amount of data in the sub-blocks contained in each consecutive read segment; The addressing time overhead of the first storage device is determined based on the number of discrete reads and the addressing latency cost. The transmission time overhead of the first storage device is determined based on the amount of data read and the cost of unit data transmission time. The read time cost of the first storage device is determined based on the addressing time cost and the transmission time cost. The read time overhead of each of the other storage devices is summed to obtain the read data time overhead when recovering the block to be recovered.

[0068] Specifically, regarding the number of discrete reads, in Figure 2 For example, there are 6 nodes C0-C5, each with 8 sub-blocks R0-R7 containing Clay codes. If C0 fails, recovering it requires reading data from the remaining 5 surviving nodes (C1-C5). Figure 2 As can be seen, each surviving node has exactly four sub-blocks concentrated in the R0-R3 section. This means that when recovering C0, the amount of data read from each disk is four units, corresponding to y=4 in the formula above. Furthermore, these four sub-blocks R0-R3 are contiguous on the disk, allowing for continuous reading in a single seek, meaning the discrete read count is 1. Combined with... Figure 2 For node C2, the data volume y read per disk is 4 units, but the sub-blocks read are distributed in 2 segments, resulting in 2 discrete read operations x. Similarly, for node C4, the sub-blocks read are distributed in 4 segments, resulting in 4 discrete read operations x. Each additional segment of discontinuity requires one more seek operation and rotational delay for the read / write head. Therefore, the larger x is, the more times the fixed overhead P is incurred. In other words, for the same amount of data read, the time cost is different.

[0069] Based on this, the formula for the read data time overhead when recovering the block to be recovered is: ; in, Indicates recovery block C i From block C j The number of discrete reads required to retrieve data from the storage device. This indicates the corresponding amount of data read. The outer summation is... i Traverse all K+2 units in the strip, indicating that assuming the first... i This scenario involves a single cell failure. Restoring cells at different locations may require reading different data (the reading modes for restoring data cells, P-check cells, and Q-check cells differ). The inner summation is... j Iterate through all disks except the failed ones, accumulating the sum of the sums from each surviving disk. j The overhead of reading data. Assuming that any one of the K+2 units fails with equal probability, the average cost of recovering a single block is obtained by averaging the total cost of all failure scenarios.

[0070] by Figure 2 For example, the decoding and recovery of C1, C3, and C5 correspond one-to-one with C0, C2, and C4. The average time cost for decoding and recovering a single block under this initial coupled structure is derived as follows: Each block recovery requires reading data from the other five storage devices. The cost of recovering C0 is 5×(P+4R), the cost of recovering C2 is 5×(2P+4R), and the cost of recovering C4 is 5×(4P+4R). The costs for C1, C3, and C5 are the same. Therefore, the average time cost is [2×5×(P+4R) + 2×5×(2P+4R) + 2×5×(4P+4R)]÷6 = (35P+60R)÷3. Substituting P=6.60ms and R=0.49ms, the average time cost is approximately 86.8ms.

[0071] The average time cost of each candidate coupling structure after replacement is calculated in the same manner, and the one with the minimum cost is selected. Under the parameter configuration of this example, the target coupling structure selected by this application is adopted. Figure 8 This application provides a schematic diagram of a coupling structure selected after replacement with a unit coupling structure, as shown in the embodiment of the present application. Figure 8 As shown, starting from the rightmost stripe unit group, the coupling structures in the original partition are replaced using three basic coupling structures in an enumerated manner. After replacement, the average time cost of decoding and restoring data for a single stripe unit is calculated. After all coupling structure replacement operations within the smallest partition have been traversed, two adjacent smallest partitions in this stripe unit group are sequentially combined into a larger partition. The coupling structures of the larger partition are then replaced using the three basic coupling structures, and the average time cost of decoding and restoring a single stripe unit is calculated simultaneously. Using the basic coupling structures, the process is repeated from the rightmost stripe unit group to the leftmost stripe unit group, traversing different partitions in the same stripe group, and then merging two smaller partitions into a larger partition. The average time cost of decoding and restoring a single stripe unit is calculated simultaneously during the enumeration of replacements. After enumerating all feasible replacements, the coupling structure with the lowest average time cost is selected as the filtering result.

[0072] by Figure 4 The average time cost for optimizing the coupling structure and recovering the data from a single stripe unit using Clay code decoding is: ;pass Figure 8 The coupled structure of the multi-layer coupled code, when individual stripe units fail, requires reading sub-blocks for decoding. Figure 9 This application provides a schematic diagram of the sub-blocks that need to be read when recovering from the failure of a single stripe unit, as shown in the embodiment of the present application. Figure 9As shown, K=4, and the stripe cells at positions C0 to C5 are considered failed, from top left to bottom right. Grid nodes represent sub-blocks of the failed stripe cell, and solid nodes represent sub-blocks that need to be read for decoding. The average time cost for recovering data from a single stripe cell is: Comparing traditional Clay code, optimized Clay code, and the method proposed in this application, the average time cost for decoding and recovering a single stripe unit under the example HDD disk and settings is as follows: , , Under the parameter configuration of the current example, compared with traditional Clay codes, the average time cost of decoding and recovering a single block of data is reduced by about 32% compared with the initial coupled structure and by about 9% compared with the optimized scheme using Gray sequences; and as K increases, the number of non-contiguous sub-blocks that need to be read under the initial coupled structure increases sharply, and the performance gain of the method in this application will be more significant.

[0073] This embodiment uses the merging of consecutive read segments as the basis for determining the number of discrete reads and the amount of data read. It restores the physical read operation of merging consecutive sub-blocks on the hard disk into a single addressing plus continuous transmission, making the calculation of time overhead correspond item by item to the actual read process. The method of calculating the addressing time overhead and the transmission time overhead separately and then summing them makes the action path of each type of time cost traceable, ensuring the accuracy of the time overhead comparison between different candidate coupling structures, thereby ensuring the reliability of the target coupling structure selection results.

[0074] In some embodiments, recovering data from a failed storage device by reading the sub-blocks required for decoding from the remaining storage devices according to the target coupling structure includes: Determine the target sub-block layer containing the sub-blocks in the failed data block that were not involved in coupled coding; Read all sub-blocks in the target sub-block layer of the remaining storage devices, and sub-blocks that are in the same coupling structure as the sub-blocks in the target sub-block layer and are located in other sub-block layers, and use them as the target sub-blocks; The target sub-blocks are coupled and verified to recover the data from the failed storage device by restoring the sub-blocks required for decoding.

[0075] Specifically, with Figure 8Taking the target coupling structure shown as an example, assuming the storage device containing block C0 fails, the decoding steps are as follows: Based on the recorded target coupling structure, determine that the target sub-block layer containing the sub-blocks in block C0 that did not participate in coupling encoding is sub-block layer R0 to R3. Therefore, all sub-blocks in sub-block layers R0 to R3 of blocks C1 to C5 need to be read. Sub-blocks in the same coupling structure as the sub-blocks in sub-block layers R0 to R3, but located in other sub-block layers, also need to be read; in this example, these are sub-blocks R6 and R7 of block C5. After reading the above target sub-blocks, perform coupling decoding on the two valid coupled sub-blocks to recover the original sub-block data, i.e., Figure 8 The coupled sub-blocks in the upper half of C2 and C3, and the three coupled sub-blocks in C4 and C5 are then used. The coupled decoded sub-blocks and existing data sub-blocks are then used to decode and recover the sub-blocks of the R0 to R3 sub-block layers of block C0 using RS code, and the original sub-blocks of the R0 to R3 sub-block layers of block C1 are simultaneously decoded and recovered. Finally, the original sub-blocks obtained from the decoding of block C1 and the read coupled sub-blocks are used to calculate the coupled sub-blocks of the R4 to R7 sub-block layers in block C0, thus completing the full recovery of the failed block. Figure 9 This shows the sub-blocks that need to be read in the manner described above when blocks C0 to C5 fail.

[0076] The sub-blocks required for decoding provided in this embodiment are uniquely determined by the target coupling structure. The two-step positioning method of first determining the target sub-block layer and then supplementing the sub-blocks located in other sub-block layers in the same coupling structure ensures that the read set is neither omitted nor redundant. The coupling decoding and verification decoding work together and are the inverse processes of the coupling encoding and verification encoding on the encoding side, ensuring the correctness and integrity of the recovered failed data. Moreover, since the target coupling structure has been determined according to the optimal time cost during the screening stage, the number of discrete reads and the amount of data read during decoding are in the combination with the minimum time cost, resulting in low read data latency during the recovery process.

[0077] In some embodiments, coupling encoding of each sub-block according to the target coupling structure for storage to the corresponding storage device includes: If it is the first encoding, the target coupling structure is recorded so that each sub-block can be coupled and encoded according to the target coupling structure; If it is not the first encoding, then obtain the recorded target coupling structure; Coupled sub-blocks are obtained by coupling encoding two mutually coupled sub-blocks according to the target coupling structure; Each coupled sub-block corresponding to the same block is stored sequentially to the corresponding storage device.

[0078] Specifically, for sub-blocks in the target coupling structure that are not paired, they are stored directly in sequence. It should be noted that, given the addressing latency and unit data transfer time costs, and assuming no further modifications, the selection of the target coupling structure only needs to be performed once. Once the target coupling structure is determined, subsequent batches of striped data encoding do not require further selection; the recorded target coupling structure can be directly reused. Sub-blocks within the same block are stored sequentially and consecutively, allowing the reading of adjacent sub-blocks on the same storage device during recovery to be combined into a single sequential read operation involving addressing and continuous transfer, fully leveraging the superior sequential read performance of mechanical hard drives compared to random read performance.

[0079] The mechanism provided in this embodiment, which involves initial encoding screening and recording followed by direct reuse of non-initial encoding, removes the computational overhead of coupled structure screening from the encoding path of each batch of data, resulting in minimal additional computational overhead during runtime.

[0080] In some embodiments, the method further includes: Based on the sub-blocks required for decoding on the storage device, determine the amount of gap data between the corresponding sub-blocks of two adjacent discrete reads; If the product of the gap data volume and the unit data transmission time cost is less than the addressing delay time cost, then two adjacent discrete reads will be merged into one continuous read, and the range of the continuous read will cover the sub-block corresponding to the gap data volume. Update the discrete number of reads and the amount of data read from the storage device to determine the read time overhead.

[0081] Specifically, based on the sub-blocks required for decoding on the storage device, the gap data amount between the corresponding sub-blocks of two adjacent discrete reads is determined; if the product of the gap data amount and the unit data transfer time cost is less than the addressing latency time cost, then the two adjacent discrete reads are merged into one continuous read, and the range of the continuous read covers the sub-blocks corresponding to the gap data amount, and the gap data is discarded after reading; then the number of discrete reads and the amount of data read for the storage device are updated to determine the read time overhead.

[0082] The derivation of the merging condition is as follows: Let the gap data volume be g units. Without merging, two adjacent discrete reads require two addressing latency costs. Merging into a single continuous read only incurs one addressing latency cost, but requires the additional transmission of g units of gap data, with an additional transmission time of g×R. Therefore, when g×R is less than P, the total time consumed by the merged read is lower, i.e., the merging condition is g less than P÷R. In the above embodiment, P is approximately 6.60ms and R is approximately 0.49ms, so P÷R is approximately 13.5, meaning that when the gap does not exceed 13 units, the merged read is superior to two discrete reads. Since the addressing latency cost of a hard disk drive is usually much higher than the transmission time cost of a small amount of data, this merging condition is easily met under actual parameters. This merging mechanism can be applied to the actual reading process in the decoding and recovery stage, as well as to the time cost calculation in the coupled structure screening stage. That is, the number of discrete reads and the amount of data read are determined according to the actual reading pattern after merging, making the screening results closer to the actual time consumption.

[0083] This embodiment uses the comparison between addressing delay time cost and transmission time cost as the merging criterion to transform some discrete reads into continuous reads. Under the given coupling structure, this further reduces the number of addressing attempts and lowers the read data time overhead when recovering failed data. At the same time, the merged read mode is fed back into the time overhead calculation, so that the selection of the coupling structure is consistent with the actual read behavior.

[0084] The foregoing has described in detail various embodiments corresponding to the data processing method. Based on this, this application also discloses a data processing apparatus corresponding to the above-described method. Figure 10 This is a structural diagram of a data processing apparatus provided in an embodiment of this application. Figure 10 As shown, the data processing device includes: Encoding module 11 is used to encode the data blocks and check blocks after the data to be processed is divided according to the initial coupling structure corresponding to the multi-layer coupling code to obtain each sub-block; Replacement module 12 is used to partition the initial coupling structure and replace the initial coupling structure of each partition with a unit coupling structure in an enumerated manner. Module 13 is selected to synchronously calculate the average time cost of reading data for restoring a single block of the replaced unit coupling structure, and select the coupling structure with the minimum time cost as the target coupling structure; the sub-blocks are coupled and encoded according to the target coupling structure and stored in the corresponding storage device. Decoding module 14 is used to recover the data of the failed storage device by reading the sub-blocks required for decoding from the other storage devices according to the target coupling structure if the storage device fails.

[0085] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments described in the method part for the embodiments of the device part, and will not be repeated here.

[0086] For a description of the data processing apparatus provided in this application, please refer to the above method embodiments. This application will not repeat the description here, as it has the same beneficial effects as the above data processing method.

[0087] Figure 11 A structural diagram of another data processing apparatus provided in the embodiments of this application is shown below. Figure 11 As shown, the device includes: Memory 21 is used to store computer programs; Processor 22 is used to implement the steps of a data processing method when executing a computer program.

[0088] The data processing device provided in this embodiment may include, but is not limited to, a tablet computer, a laptop computer, or a desktop computer.

[0089] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array. The processor 22 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 22 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 22 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0090] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 21 is used to store at least the following computer program 211, which, after being loaded and executed by the processor 22, is capable of implementing the relevant steps of the data processing method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. The operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, the data involved in the data processing method, etc.

[0091] In some embodiments, the data processing device may further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.

[0092] Those skilled in the field can understand, Figure 11 The structure shown does not constitute a limitation on the data processing device and may include more or fewer components than illustrated.

[0093] The processor 22 implements the data processing method provided in any of the above embodiments by calling instructions stored in the memory 21.

[0094] For a description of the data processing apparatus provided in this application, please refer to the above method embodiments. This application will not repeat the description here, as it has the same beneficial effects as the above data processing method.

[0095] Furthermore, this application also provides a computer-readable storage medium storing a computer program, which, when executed by processor 22, implements the steps of the data processing method described above.

[0096] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments. This application will not repeat the description here, as it has the same beneficial effects as the above data processing method.

[0098] Furthermore, this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of a data processing method.

[0099] The data processing method, apparatus, and medium provided in this application have been described in detail above. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

[0100] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

Claims

1. A data processing method, characterized in that, include: Based on the initial coupling structure corresponding to the multi-layer coupled code, the data blocks and check blocks after the data to be processed are divided are encoded to obtain each sub-block; The initial coupling structure is partitioned, and the initial coupling structure of each partition is replaced by a unit coupling structure in an enumerated manner; The average time cost of reading data for a single block is calculated synchronously for the replaced unit coupling structure, and the coupling structure with the minimum time cost is selected as the target coupling structure. Each sub-block is coupled and encoded according to the target coupling structure, and then stored in the corresponding storage device; If the storage device fails, the data of the failed storage device is recovered by reading the sub-blocks required for decoding from the remaining storage devices according to the target coupling structure.

2. The data processing method according to claim 1, characterized in that, The unit coupling structure includes at least one or more of the following: unidirectional structure, cross structure, and symmetrical structure; The unit coupling structure acts on an even number of paired nodes, each pair of nodes including a left node and a right node, and the pairs of nodes are arranged in sequence. The unidirectional structure is a coupled structure in which the left node in each pair of nodes is connected to the right node in the next adjacent pair of nodes along the same direction, and the left node in the last pair of nodes is connected to the right node in the first pair of nodes; the number of pairs of nodes in the unidirectional structure is an integer greater than one. The cross structure consists of the left node in each pair of nodes being connected to the right node in a pair of nodes in the same direction, forming a coupled structure in which adjacent connections intersect; the number of pairs of nodes in the cross structure is an integer greater than two. The symmetrical structure is a coupled structure in which the left node of the first pair of nodes is connected to the right node of the last pair of nodes, and the left node of the last pair of nodes is connected to the right node of the first pair of nodes, and the two pairs of nodes in symmetrical positions are connected to each other from the outside to the inside; the number of pairs of nodes in the symmetrical structure is an even number greater than two.

3. The data processing method according to claim 2, characterized in that, Partitioning the initial coupling structure includes: Each block is divided into multiple groups in order, and each sub-block layer in each group is divided into the corresponding first partition in order. Correspondingly, the initial coupling structure of each partition is replaced by an enumerated unit coupling structure, including: Starting from the last group, the coupling structures in each of the first partitions are replaced one by one by the unit coupling structures in an enumerated manner. Two adjacent sibling partitions are sequentially combined into the next level partition and replaced, until the combined partition covers all sub-block layers within the group.

4. The data processing method according to claim 3, characterized in that, Adjacent sibling partitions are sequentially combined into the next level partition and replaced, until the combined partition covers all sub-block layers within the group, including: After the replacement traversal of each of the first partitions in the group is completed, two adjacent first partitions are combined in sequence to form a second partition, and the coupling structure in each of the second partitions is replaced one by one by the unit coupling structure in an enumeration manner. Adjacent sibling partitions are sequentially combined into the next level partition and replaced, until the combined partition covers all sub-block layers within the group.

5. The data processing method according to claim 3 or 4, characterized in that, The average time cost of recovering read data for a single block is calculated synchronously for the replaced unit coupling structure, including: Obtain the addressing latency cost and unit data transfer time cost of the storage device; Based on the addressing delay time cost and the unit data transmission time cost, determine the read data time overhead when recovering each block separately; The average read data time cost of each block is determined as the average read data time cost.

6. The data processing method according to claim 5, characterized in that, The step of determining the read data time overhead for recovering each block based on the addressing delay time cost and the unit data transmission time cost includes: Each block is identified as a block to be recovered in turn; Obtain the current block to be recovered, and determine the sub-blocks required for decoding when recovering the current block to be recovered, as well as the distribution locations of the sub-blocks required for decoding on the other storage devices, based on the replaced coupling structure; Obtain the first storage device from the remaining storage devices; according to the distribution location, group the consecutive sub-blocks in the sub-blocks required for decoding on the first storage device into the same continuous read segment to obtain at least one continuous read segment; The number of consecutive read segments is determined as the number of discrete reads of the first storage device; The amount of data read by the first storage device is determined based on the amount of data in the sub-blocks contained in each of the consecutive read segments; The addressing time overhead of the first storage device is determined based on the number of discrete reads and the addressing latency time cost. The transmission time overhead of the first storage device is determined based on the amount of data read and the unit data transmission time cost. The read time cost of the first storage device is determined based on the addressing time cost and the transmission time cost. The read time overhead of each of the other storage devices is summed to obtain the read data time overhead when recovering the block to be recovered.

7. The data processing method according to claim 1, characterized in that, To recover data from a failed storage device by reading the sub-blocks required for decoding from the remaining storage devices according to the target coupling structure, the method includes: Determine the target sub-block layer containing the sub-blocks in the failed data block that were not involved in coupled coding; Read all sub-blocks in the target sub-block layer of the remaining storage devices, and sub-blocks that are in the same coupling structure as the sub-blocks in the target sub-block layer and are located in other sub-block layers, and use them as the target sub-blocks; The target sub-blocks are coupled and verified to recover the data in the failed storage device by restoring the sub-blocks required for decoding.

8. The data processing method according to claim 1, characterized in that, According to the target coupling structure, each sub-block is coupled and encoded for storage in the corresponding storage device, including: If this is the first encoding, the target coupling structure is recorded so that each sub-block can be coupled and encoded according to the target coupling structure; If it is not the first encoding, then the recorded target coupling structure is obtained; Coupled sub-blocks are obtained by coupling encoding two mutually coupled sub-blocks according to the target coupling structure; Each coupled sub-block corresponding to the same block is stored sequentially to the corresponding storage device.

9. A data processing apparatus, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 8.

Images