A method and apparatus for cross-frame sensing invalid data recovery based on erasure coding

Abstract

This application belongs to the field of computer storage technology, specifically disclosing a cross-rack sensing invalid data reclamation method and apparatus based on erasure coding. The method includes: determining the cross-rack update network traffic generated by simulating the reclamation of invalid data blocks in each erasure coding stripe based on the invalid data blocks and valid check blocks stored in each rack; dividing each erasure coding stripe into filling stripes and reclamation stripes based on the cross-rack update network traffic corresponding to each erasure coding stripe; migrating the valid data blocks of each reclamation stripe to the storage nodes where the invalid data blocks in each filling stripe are located, and releasing the storage space of each reclamation stripe; and updating the valid check blocks of each filling stripe. This application achieves a more scientific and efficient invalid data reclamation method, effectively reducing network traffic throughout the invalid data reclamation process, improving invalid data reclamation efficiency, while ensuring system storage efficiency.

Description

Technical Field

[0001] This application belongs to the field of computer storage technology, and more specifically, relates to a method and apparatus for cross-rack sensing invalid data recovery based on erasure coding. Background Technology

[0002] Compared to replication mechanisms, erasure coding can achieve high data reliability while maintaining low storage overhead, and is therefore widely deployed in today's cloud storage systems. In erasure coding-based cloud storage systems, deletion operations generate invalid data. As invalid data accumulates, the cloud storage system must periodically invoke a reclamation mechanism to release storage space for invalid data to ensure storage efficiency. However, to ensure the encoding consistency of data blocks and check blocks within an erasure coding stripe, erasure coding-based cloud storage systems cannot directly delete invalid data blocks like replication mechanisms. Therefore, how to efficiently reclaim invalid data in cloud storage systems has always been a key research focus in the industry.

[0003] Existing data reclamation mechanisms offer a recoding-based method that directly reassembles the remaining valid data blocks in an old stripe containing invalid data blocks into a new erasure coding stripe to maximize data availability and storage efficiency. However, this method introduces significant inter-node network traffic during stripe reassembly. Furthermore, the recoding reclamation technique ignores the multi-layered rack network architecture of cloud storage systems, further introducing substantial cross-rack network traffic. This results in excessive network traffic during invalid data reclamation and low reclamation efficiency.

[0004] Therefore, how to better realize the reclamation of invalid data in cloud storage systems based on erasure coding has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this application is to better realize the invalid data reclamation of cloud storage systems based on erasure coding, aiming to solve the problems of excessive network traffic and low efficiency in the invalid data reclamation process in existing technologies.

[0006] To achieve the above objectives, in a first aspect, this application provides a cross-architecture sensing invalid data retrieval method based on erasure coding, comprising:

[0007] Based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, determine the cross-rack update network traffic generated by each erasure code stripe simulating the recovery of the invalid data blocks;

[0008] Based on the cross-rack update network traffic corresponding to each erasure code stripe, each erasure code stripe is divided into a fill stripe and a reclaim stripe; the cross-rack update network traffic corresponding to the fill stripe is no greater than the cross-rack update network traffic corresponding to the reclaim stripe.

[0009] The valid data blocks of each of the reclaimed strips are migrated to the storage node where the invalid data blocks of each of the filling strips are located, and the storage space of each of the reclaimed strips is released;

[0010] Update the valid check blocks for each of the filling strips.

[0011] Optionally, determining the cross-rack update network traffic generated by simulating the reclamation of invalid data blocks for each erasure coding stripe stored in each rack, based on the invalid data blocks and valid check blocks of each erasure coding stripe stored in each rack, includes:

[0012] Determine the collection rack corresponding to each erasure code stripe from each of the racks;

[0013] Based on the distribution of invalid data blocks and valid parity blocks in each erasure coding stripe and the collection rack corresponding to each erasure coding stripe, the target cross-rack update network traffic generated by executing a preset recycling strategy for each erasure coding stripe is determined; the preset recycling strategy is used to execute the strategy of transmitting all invalid data blocks of each erasure coding stripe to the corresponding collection rack to overwrite the valid parity blocks, and updating the remaining valid parity blocks in the corresponding collection rack;

[0014] Based on the target cross-rack update network traffic, the cross-rack update network traffic generated by each erasure code stripe simulating the reclamation of the invalid data block is obtained.

[0015] Optionally, determining the collection rack corresponding to each erasure code stripe from the various racks includes:

[0016] Obtain the distribution information of the number of valid parity blocks stored in each rack for each erasure code stripe;

[0017] The rack that has the largest number of valid check blocks for storing erasure code stripes is determined as the collection rack corresponding to each erasure code stripe.

[0018] Optionally, the step of dividing each erasure code stripe into a filling stripe and a recycling stripe based on the cross-rack update network traffic corresponding to each erasure code stripe includes:

[0019] Sort the cross-rack update network traffic corresponding to each erasure code stripe in ascending order;

[0020] Starting with the erasure code stripe with the smallest sequence number, select each erasure code stripe in sequence until it is determined that the total number of invalid data blocks in all selected erasure code stripes is not less than the total number of valid data blocks in all unselected erasure code stripes, then stop the erasure code stripe selection process.

[0021] All selected erasure code stripes are designated as fill stripes, and all unselected erasure code stripes are designated as recycle stripes.

[0022] Optionally, after sorting the cross-rack update network traffic corresponding to each erasure code stripe in ascending order, the method further includes:

[0023] Starting with the erasure code stripe with the largest sequence number, select each erasure code stripe in reverse order until it is determined that the total number of all valid data blocks in all selected erasure code stripes is not greater than the total number of invalid data blocks in all unselected erasure code stripes, and then stop the erasure code stripe selection process;

[0024] All selected erasure code stripes are designated as the recycled stripes, and all unselected erasure code stripes are designated as the fill stripes.

[0025] Optionally, migrating the valid data blocks of each of the reclaimed strips to the storage node where the invalid data blocks of each of the filling strips reside includes:

[0026] Following a strategy that prioritizes data migration within the same rack, valid data blocks from each of the reclaimed strips are migrated to the storage nodes where invalid data blocks from each of the filling strips reside.

[0027] Secondly, this application provides a cross-architecture sensing invalid data recovery device based on erasure coding, comprising:

[0028] The processing module is used to determine the cross-rack update network traffic generated by each erasure code stripe simulating the recovery of the invalid data blocks based on the invalid data blocks and valid check blocks stored in each rack.

[0029] The partitioning module is used to partition each erasure code stripe into a fill stripe and a reclaimed stripe based on the cross-rack update network traffic corresponding to each erasure code stripe; the cross-rack update network traffic corresponding to the fill stripe is not greater than the cross-rack update network traffic corresponding to the reclaimed stripe;

[0030] The recycling module is used to migrate the valid data blocks of each of the recycled strips to the storage nodes where the invalid data blocks of each of the filled strips are located, and to release the storage space of each of the recycled strips;

[0031] The update module is used to update the valid verification blocks of each of the filling strips.

[0032] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0033] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0034] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0035] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0036] The cross-rack sensing invalid data reclamation method and apparatus based on erasure coding provided in this application treats invalid data blocks as data blocks that need to be updated. It simulates and calculates the cross-rack update network traffic generated by each erasure coding stripe for reclamation of invalid data blocks. Based on the sorting of these cross-rack update network traffic, the stripes with lower cross-rack update network traffic in each erasure coding stripe are preferentially assigned as filling stripes, and the rest are assigned as reclamation stripes. The valid data blocks of each reclamation stripe are migrated to the filling stripe with low update traffic to cover their invalid data blocks, thereby releasing the storage space of each reclamation stripe. This achieves a more scientific and efficient invalid data reclamation method, which can effectively reduce the network traffic in the entire invalid data reclamation process, improve the invalid data reclamation efficiency, and ensure the storage efficiency of the system. Attached Figure Description

[0037] Figure 1 This is one of the flowcharts of the cross-frame sensing invalid data recovery method based on erasure coding provided in this application;

[0038] Figure 2 This is the second flowchart of the cross-frame sensing invalid data recovery method based on erasure coding provided in the embodiments of this application;

[0039] Figure 3 This is a schematic diagram of the cross-rack sensing invalid data recovery device based on erasure coding provided in the embodiments of this application;

[0040] Figure 4This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0043] First, the technical terms involved in the embodiments of this application will be introduced.

[0044] (1) Erasure code stripe

[0045] Erasure coding technology can divide the original data into several data blocks, and then perform specific matrix operations on these data blocks to generate several check blocks. These data blocks and check blocks together form an erasure code stripe, which can also be described as an erasure code stripe.

[0046] For example, in an erasure coding-based In a storage system, a file is divided into Each original data block is encoded into a coding matrix. A total of data blocks, of which ,in, The number of check blocks. A set of m total data blocks is called an "erasing code stripe". An erasing code stripe is the smallest coding unit in erasing codes. Typically, each data block in an erasing code stripe is stored on different storage nodes, and the system can tolerate the failure of any m nodes. If no more than m nodes are affected... If one node fails, then the remaining nodes can be used. Damaged data was recovered from each node.

[0047] (2) Frame

[0048] A rack is the physical support structure for storage devices (such as hard drives and servers), typically used in data centers or large storage systems. In erasure coding-based distributed storage systems, multiple racks can form a cluster, with each rack containing multiple storage nodes (such as servers).

[0049] The embodiments of this application are described below with reference to the accompanying drawings.

[0050] Figure 1 This is one of the flowcharts illustrating the cross-architecture sensing invalid data retrieval method based on erasure coding provided in this application, such as... Figure 1 As shown, it includes:

[0051] Step S1: Based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, determine the cross-rack update network traffic generated by simulating the reclamation of invalid data blocks for each erasure code stripe;

[0052] Step S2: Based on the cross-rack update network traffic corresponding to each erasure code stripe, divide each erasure code stripe into a fill stripe and a reclaim stripe; the cross-rack update network traffic corresponding to the fill stripe is no greater than the cross-rack update network traffic corresponding to the reclaim stripe.

[0053] Step S3: Migrate the valid data blocks of each reclaimed strip to the storage node where the invalid data blocks of each filled strip are located, and release the storage space of each reclaimed strip;

[0054] Step S4: Update the valid check blocks for each fill strip.

[0055] Specifically, the invalid data block described in this application generally refers to the data block corresponding to the data to be deleted as indicated by the deletion request input by the front-end user. When data is deleted by the user, the storage system does not immediately delete the data on the disk, but marks the deleted data as invalid, thereby generating an invalid data block to ensure the coding consistency of erasure coding stripes.

[0056] The cross-rack update network traffic described in this application refers to the amount of data that needs to be transferred between storage nodes on different racks when the system performs erasure code stripe data update operations.

[0057] Specifically, when a data block needs to be updated, due to the encoding characteristics of erasure coding, all related check blocks need to be checked and updated. These check blocks may be distributed across different racks. Therefore, the update operation needs to read the relevant check blocks from storage nodes on other racks via the network and write the updated check blocks back to the corresponding storage nodes. The amount of data transferred in this process constitutes the cross-rack update network traffic.

[0058] The recycling stripe described in this application refers to the erasure code stripe used for subsequent migration of valid data blocks.

[0059] The padding strip described in this application refers to the erasure code strip containing invalid data blocks that need to be covered during the migration of valid data blocks in subsequent recycling strips.

[0060] In the embodiments of this application, in step S1, based on the distribution of invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, the invalid data blocks of each erasure code stripe can be regarded as data blocks that need to be updated. The process of updating the relevant check blocks in the erasure code stripe is simulated, and the cross-rack update network traffic generated by simulating the recycling of invalid data blocks in each erasure code stripe is calculated. Thus, the amount of cross-rack update network traffic required to recycle invalid data blocks in each erasure code stripe can be determined.

[0061] In the embodiments of this application, in step S2, the traffic size is sorted according to the cross-rack update network traffic corresponding to each erasure code stripe by a pre-set stripe selection strategy, and then the erasure code stripe with the smallest cross-rack update network traffic is selected as the fill stripe, thereby dividing each erasure code stripe into fill stripe and reclaimed stripe.

[0062] Understandably, by selectively choosing erasure coding stripes with lower cross-rack update network traffic as filler stripes to overwrite subsequent invalid data, it is possible to ensure that the cross-rack update network traffic generated during the process of reclaiming invalid data blocks is lower. This can reduce the cross-rack update network traffic during the invalid data reclamation process, thereby reducing the network traffic of the entire invalid data reclamation process.

[0063] Furthermore, in the embodiments of this application, in step S3, based on the determination of the filling strips and the reclaimed strips, the valid data blocks of each reclaimed strip are migrated to the storage nodes where the invalid data blocks in each filling strip are located, thus completing the overwriting of invalid data blocks by valid data blocks. Since the valid data blocks in the reclaimed strips are migrated out, the storage space of the reclaimed strips can be released. Therefore, the storage space occupied by the original valid data blocks in each reclaimed strip can be released, thereby completing the reclamation of invalid data.

[0064] Meanwhile, invalid data blocks in the fill strip are overwritten by valid data blocks in the reclaimed strip, therefore, the check blocks in the fill strip need to be updated.

[0065] Specifically, in the embodiments of this application, in step S4, in order to ensure the encoding consistency of erasure code stripes, when new data is migrated into each padding strip and a valid data block is updated, it is also necessary to update each valid check block in the corresponding padding strip according to the updated valid data block, and write the updated check block back to the corresponding storage node, so as to ensure the storage efficiency of the system.

[0066] The cross-rack awareness invalid data reclamation method based on erasure coding in this application embodiment treats invalid data blocks as data blocks that need to be updated. It simulates and calculates the cross-rack update network traffic generated by each erasure coding stripe for reclamation of invalid data blocks. Based on the sorting of these cross-rack update network traffic, the stripes with lower cross-rack update network traffic in each erasure coding stripe are preferentially classified as filling stripes, and the rest are classified as reclamation stripes. The valid data blocks of each reclamation stripe are migrated to the filling stripe with low update traffic to cover their invalid data blocks, thereby releasing the storage space of each reclamation stripe. This achieves a more scientific and efficient invalid data reclamation method, which can effectively reduce the network traffic in the entire invalid data reclamation process, improve the invalid data reclamation efficiency, and ensure the storage efficiency of the system.

[0067] Based on the above embodiments, as an optional embodiment, step S1, based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, determines the cross-rack update network traffic generated by simulating the reclamation of invalid data blocks for each erasure code stripe, including:

[0068] Identify the collection rack corresponding to each erasure code stripe from each rack;

[0069] Based on the distribution of invalid data blocks and valid parity blocks in each erasure code stripe and the collection rack corresponding to each erasure code stripe, the target cross-rack update network traffic generated by executing the preset recycling strategy for each erasure code stripe is determined; the preset recycling strategy is used to execute the strategy of transmitting all invalid data blocks of each erasure code stripe to the corresponding collection rack to overwrite the valid parity blocks, and updating the remaining valid parity blocks in the corresponding collection rack.

[0070] Based on the target cross-rack update network traffic, the cross-rack update network traffic generated by each erasure code stripe simulating the reclamation of invalid data blocks is obtained.

[0071] Specifically, the collection rack described in the embodiments of this application refers to the target rack set from each rack to receive invalid data blocks in the erasure code strip during the process of simulating the recovery of invalid data blocks in each erasure code stripe.

[0072] The target cross-rack update network traffic described in the embodiments of this application refers to the cross-rack update network traffic generated during the process of each erasure code stripe simulating the transmission of all its invalid data blocks to the corresponding collection rack to overwrite the valid check blocks stored therein, and updating the remaining valid check blocks in the corresponding collection rack.

[0073] In the embodiments of this application, it is necessary to first determine the collection rack corresponding to each erasure code stripe from each rack.

[0074] Based on the above embodiments, as an optional embodiment, determining the collection rack corresponding to each erasure code stripe from various racks includes:

[0075] Obtain the distribution information of the number of valid parity blocks stored in each rack for each erasure code stripe;

[0076] The rack that has the largest number of valid parity blocks for storing erasure code stripes is determined as the collection rack corresponding to each erasure code stripe.

[0077] Specifically, in the embodiments of this application, assuming that invalid data blocks are updated, the distribution information of the number of valid check blocks stored in each rack of each erasure code stripe is obtained based on the distribution of invalid data blocks and valid check blocks in each erasure code stripe. Then, the rack with the largest number of valid check blocks is selected from each rack as the collection rack, thereby obtaining the collection rack corresponding to each erasure code stripe.

[0078] The method in this application selects a suitable collection rack based on the invalid data block recovery status and the number distribution of valid check blocks in each erasure coding stripe, so that a suitable stripe division method can be selected in the subsequent data recovery and update stage, which helps to further reduce cross-rack update network traffic.

[0079] Furthermore, in the embodiments of this application, for each erasure code stripe, all invalid blocks can be transmitted to the corresponding collection rack according to a preset recycling strategy. The collection rack then updates the remaining valid check blocks, thereby calculating the target cross-rack update network traffic corresponding to each erasure code stripe based on the data transmission and update process.

[0080] Finally, the target cross-rack update network traffic corresponding to each erasure code stripe is used as the cross-rack update network traffic generated by each erasure code stripe to simulate the reclamation of invalid data blocks.

[0081] The method in this application embodiment updates data by simulating invalid data blocks as update data and adopts an erasure coding data collaborative update method to predict the cross-rack update network traffic generated by simulating the recycling of invalid data blocks for each erasure coding stripe. This provides a technical basis for the subsequent division of recycling stripes and filling stripes, and ensures that the cross-rack update network traffic is minimized during the data recycling process.

[0082] Based on the above embodiments, as an optional embodiment, based on the cross-rack update network traffic corresponding to each erasure coding stripe, each erasure coding stripe is divided into a filling stripe and a recycling stripe, including:

[0083] Sort the cross-rack update network traffic corresponding to each erasure code stripe in ascending order;

[0084] Starting with the erasure code stripe with the smallest sequence number, select each erasure code stripe in sequence until it is determined that the total number of invalid data blocks in all selected erasure code stripes is not less than the total number of valid data blocks in all unselected erasure code stripes, then stop the erasure code stripe selection process.

[0085] All selected erasure coding stripes are classified as fill stripes, and all unselected erasure coding stripes are classified as recycle stripes.

[0086] Specifically, in the embodiments of this application, there are two ways to divide the erasure coding stripes into filling stripes and recycling stripes. The first method is to sort them according to the size of the cross-rack update network traffic and then filter them sequentially from the side of the erasure coding stripe with the least traffic.

[0087] More specifically, the cross-rack update network traffic corresponding to each erasure code stripe is sorted in ascending order. Thus, the erasure code stripe with the smallest cross-rack update network traffic corresponds to the smallest sequence number, and the erasure code stripe with the largest cross-rack update network traffic corresponds to the largest sequence number.

[0088] Furthermore, in this embodiment, starting with the erasure code stripe with the smallest sequence number, each erasure code stripe is selected sequentially. For each selected stripe, the difference between the number of invalid data blocks in all selected stripes and the number of valid data blocks in the remaining unselected stripes is calculated. This process continues until it is determined that the total number of invalid data blocks in all selected erasure code stripes is not less than the total number of valid data blocks in all unselected erasure code stripes. This indicates that all valid data blocks in the subsequently reclaimed stripes can be completely migrated out without any valid data blocks failing to migrate, thus preventing the storage space of the reclaimed stripes from being released. At this point, the erasure code stripe selection process is stopped.

[0089] Therefore, all selected erasure coding stripes can be classified as fill stripes, while all remaining unselected erasure coding stripes can be classified as recycle stripes.

[0090] The method in this application embodiment sorts the cross-rack update network traffic corresponding to each erasure code stripe in ascending order. Starting from the erasure code stripe with the smallest sequence number, the stripe with smaller cross-rack update network traffic is selected as the filling stripe based on the relationship between the number of invalid data blocks in the filling stripe and the number of valid data blocks in the reclaimed stripe. The remaining stripes are used as reclaimed stripes to facilitate efficient reclamation of invalid data in subsequent operations and reduce cross-rack update network traffic during the reclamation process.

[0091] Based on the above embodiments, as an optional embodiment, after sorting the cross-rack update network traffic corresponding to each erasure code stripe in ascending order, the method further includes:

[0092] Starting with the erasure code stripe with the largest sequence number, select each erasure code stripe in reverse order until it is determined that the total number of all valid data blocks in all selected erasure code stripes is not greater than the total number of invalid data blocks in all unselected erasure code stripes, and then stop the erasure code stripe selection process;

[0093] All selected erasure coding stripes are classified as reclaimed stripes, and all unselected erasure coding stripes are classified as fill stripes.

[0094] Specifically, in the embodiments of this application, there are two implementation methods for dividing each erasure code stripe into filling stripes and recycling stripes. The second method can be the opposite of the first one, that is, sorting according to the size of cross-rack update network traffic and filtering sequentially from the side of the erasure code stripe with the largest traffic.

[0095] More specifically, after sorting the cross-rack update network traffic corresponding to each erasure code stripe in ascending order, the erasure code stripes are selected sequentially, starting with the stripe with the largest sequence number. During this process, for each selected stripe, the difference between the total number of valid data blocks in all selected stripes and the total number of invalid data blocks in all remaining unselected stripes is calculated. This continues until it is determined that the total number of valid data blocks in all selected erasure code stripes is no greater than the total number of invalid data blocks in all unselected erasure code stripes, ensuring that all valid data blocks in subsequently reclaimed stripes can be completely migrated out and the storage space of the reclaimed stripes can be released. At this point, the erasure code stripe selection process stops.

[0096] Therefore, all selected erasure coding stripes can be classified as recycled stripes, while all remaining unselected erasure coding stripes can be classified as fill stripes.

[0097] The method in this application embodiment sorts the cross-rack update network traffic corresponding to each erasure code stripe in ascending order. Starting from the erasure code stripe with the largest sequence number, the stripe with larger cross-rack update network traffic is selected as the recycling stripe based on the relationship between the number of invalid data blocks in the filling stripe and the number of valid data blocks in the reclaimed stripe. The remaining stripes are used as filling stripes. This reduces the cross-rack update network traffic during the recycling process and facilitates efficient recycling of invalid data in the subsequent process.

[0098] Based on the above embodiments, as an optional embodiment, migrating the valid data blocks of each recycled strip to the storage node where the invalid data blocks of each filled strip reside includes:

[0099] Following a strategy that prioritizes data migration within the same rack, valid data blocks from each reclaimed strip are migrated to the storage nodes containing invalid data blocks from each filled strip.

[0100] Specifically, in the embodiments of this application, valid data blocks in the reclaimed strip are migrated to the storage location where invalid data blocks in the filling strip are located. For valid data blocks in the reclaimed strip within the same rack, they are preferentially migrated to the node location within the same rack where invalid data blocks in the filling strip are located, thereby further avoiding cross-rack update network traffic during data migration.

[0101] Figure 2 This is the second flowchart illustrating the cross-architecture sensing invalid data recovery method based on erasure coding provided in this application embodiment. Figure 2 As shown, in one specific embodiment of this application, there are five chains in the cluster. The storage system uses erasure coding stripes, each containing invalid data blocks. Blocks with the same numerical number belong to the same erasure coding stripe, which are designated as erasure coding stripes S1, S2, S3, S4, and S5. The valid data blocks, valid parity blocks, and invalid data blocks in these five erasure coding stripes are distributed across storage nodes in five racks, denoted as R1, R2, R3, R4, and R5. As shown in the figure, the number of valid data blocks in these erasure coding stripes is 2, 3, 2, 3, and 2 respectively, represented as 2 (S1), 3 (S2), 2 (S3), 3 (S4), and 2 (S5).

[0102] Furthermore, following the cross-rack update network traffic calculation method in the aforementioned embodiment, for erasure coding stripe S1, the collection rack R4 with the most valid check blocks is determined. Two invalid data blocks in R3 and R5 are considered as updated data and are transmitted to the corresponding two valid check blocks in R4. Therefore, the cross-rack update network traffic corresponding to erasure coding stripe S1 can be quantified as 2 blocks. Similarly, for erasure coding stripe S2, its corresponding cross-rack update network traffic can be quantified as 1 block; for erasure coding stripe S3, its corresponding cross-rack update network traffic as 3; for erasure coding stripe S4, its corresponding cross-rack update network traffic as 4; and for erasure coding stripe S5, its corresponding cross-rack update network traffic as 2. Thus, the cross-rack update network traffic of these stripes, sorted in ascending order, can be represented as: 1 (S2), 2 (S5), 2 (S4), 2 (S1), 3 (S3).

[0103] Reference Figure 2It can be seen that, through three rounds of selection, all the erasure coding stripes selected sequentially are represented as fiS={S2, S5, S4}, and all the remaining unselected erasure coding stripes are represented as reS={S1, S3}. The total number of invalid data blocks in the erasure coding stripes {S2, S5, S4} is 4, which satisfies the condition that it is not less than the total number of valid data blocks in all the unselected erasure coding stripes {S1, S3}, which is also 4. Therefore, the erasure coding stripes {S2, S5, S4} with less cross-rack update traffic are selected as the fill stripes, and the remaining erasure coding stripes {S1, S3} are selected as the reclaimed stripes.

[0104] Furthermore, valid data blocks in the reclaimed stripes {S1, S3} are migrated to the storage location where invalid data blocks in the filling stripes {S2, S5, S4} reside. For example, valid data blocks in reclaimed stripe S1 in R1 can be migrated within the rack via intra-rack links, avoiding cross-rack update network traffic and thus overwriting invalid data blocks in filling stripe S5 in R1; valid data blocks in reclaimed stripe S1 in R2 are migrated via cross-rack links, overwriting another invalid data block in filling stripe S5 in R1, and so on. This process continues until the storage space of reclaimed stripes {S1, S3} in each rack is freed up, thereby completing the reclamation of invalid data blocks from the five erasure coding stripes.

[0105] The invalid data reclamation method in this application, targeting cross-rack network architecture, employs a data padding algorithm to migrate valid data from one part of the erasure coding stripe to cover the location of invalid data in another part of the erasure coding stripe. This reduces cross-rack update network traffic. The migration process is completed within the same rack whenever possible. Finally, the storage space of the reclaimed stripe is released, and the parity blocks in the padding stripe are updated. Through the cross-rack update network traffic sorting and intra-rack migration strategies in this method, the cross-rack update network traffic generated during invalid data reclamation can be significantly reduced, improving the efficiency of invalid data reclamation.

[0106] The cross-rack sensing invalid data recovery device based on erasure coding provided in this application is described below. The cross-rack sensing invalid data recovery device based on erasure coding described below can be referred to in correspondence with the cross-rack sensing invalid data recovery method based on erasure coding described above.

[0107] Figure 3 This is a schematic diagram of the cross-rack sensing invalid data recovery device based on erasure coding provided in the embodiments of this application, as shown below. Figure 3 As shown, it includes:

[0108] Processing module 10 is used to determine the cross-rack update network traffic generated by simulating the reclamation of invalid data blocks for each erasure code stripe based on the invalid data blocks and valid check blocks stored in each rack.

[0109] The partitioning module 20 is used to divide each erasure code stripe into a fill stripe and a reclaim stripe based on the cross-rack update network traffic corresponding to each erasure code stripe; the cross-rack update network traffic corresponding to the fill stripe is no greater than the cross-rack update network traffic corresponding to the reclaim stripe;

[0110] The recycling module 30 is used to migrate the valid data blocks of each recycled strip to the storage node where the invalid data blocks of each filled strip are located, and to release the storage space of each recycled strip.

[0111] Update module 40 is used to update the valid check blocks of each fill strip.

[0112] It is understood that the detailed functional implementation of each of the above units / modules can be found in the description in the aforementioned method embodiments, and will not be repeated here.

[0113] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0114] The cross-rack sensing invalid data reclamation device based on erasure coding in this application embodiment treats invalid data blocks as data blocks that need to be updated. It simulates and calculates the cross-rack update network traffic generated by each erasure coding stripe for reclamation of invalid data blocks. Based on the sorting of these cross-rack update network traffic, the stripes with lower cross-rack update network traffic in each erasure coding stripe are preferentially classified as filling stripes, and the rest are classified as reclamation stripes. The valid data blocks of each reclamation stripe are migrated to the filling stripe with low update traffic to cover their invalid data blocks, thereby releasing the storage space of each reclamation stripe. This achieves a more scientific and efficient invalid data reclamation method, which can effectively reduce the network traffic in the entire invalid data reclamation process, improve the invalid data reclamation efficiency, and ensure the storage efficiency of the system.

[0115] Based on the methods in the above embodiments, this application provides an electronic device, such as... Figure 4As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the methods in the above embodiments.

[0116] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0117] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0118] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0119] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0120] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0121] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0122] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0123] It should be understood that expressions such as “comprising” and “may include” used in this application indicate the existence of the disclosed functions, operations, or constituent elements, and do not limit one or more additional functions, operations, and constituent elements. In this application, terms such as “comprising” and / or “having” are to be interpreted as indicating a particular characteristic, number, operation, constituent element, component, or combination thereof, but not to exclude the existence or possibility of adding one or more other characteristics, numbers, operations, constituent elements, components, or combinations thereof.

[0124] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for cross-architecture sensing invalid data retrieval based on erasure coding, characterized in that, include: Based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, determine the cross-rack update network traffic generated by each erasure code stripe simulating the recovery of the invalid data blocks; Based on the cross-rack update network traffic corresponding to each erasure code stripe, each erasure code stripe is divided into a fill stripe and a reclaim stripe; the cross-rack update network traffic corresponding to the fill stripe is no greater than the cross-rack update network traffic corresponding to the reclaim stripe. The valid data blocks of each of the reclaimed strips are migrated to the storage node where the invalid data blocks of each of the filling strips are located, and the storage space of each of the reclaimed strips is released; Update the valid check blocks for each of the aforementioned fill strips; The step of determining the cross-rack update network traffic generated by simulating the reclamation of invalid data blocks for each erasure coding stripe stored in each rack, based on the invalid data blocks and valid check blocks of each erasure coding stripe stored in each rack, includes: Determine the collection rack corresponding to each erasure code stripe from each of the racks; Based on the distribution of invalid data blocks and valid parity blocks in each erasure coding stripe and the collection rack corresponding to each erasure coding stripe, the target cross-rack update network traffic generated by executing a preset recycling strategy for each erasure coding stripe is determined; the preset recycling strategy is used to execute the strategy of transmitting all invalid data blocks of each erasure coding stripe to the corresponding collection rack to overwrite the valid parity blocks, and updating the remaining valid parity blocks in the corresponding collection rack; Based on the target cross-rack update network traffic, the cross-rack update network traffic generated by each erasure code stripe simulating the reclamation of the invalid data block is obtained; The step of determining the collection rack corresponding to each erasure code stripe from the various racks includes: Obtain the distribution information of the number of valid parity blocks stored in each rack for each erasure code stripe; The rack that has the largest number of valid check blocks for storing erasure code stripes is determined as the collection rack corresponding to each erasure code stripe.

2. The cross-frame sensing invalid data recovery method based on erasure coding according to claim 1, characterized in that, The step of dividing each erasure code stripe into fill stripes and reclaimed stripes based on the cross-rack update network traffic corresponding to each erasure code stripe includes: Sort the cross-rack update network traffic corresponding to each erasure code stripe in ascending order; Starting with the erasure code stripe with the smallest sequence number, select each erasure code stripe in sequence until it is determined that the total number of invalid data blocks in all selected erasure code stripes is not less than the total number of valid data blocks in all unselected erasure code stripes, then stop the erasure code stripe selection process. All selected erasure code stripes are designated as fill stripes, and all unselected erasure code stripes are designated as recycle stripes.

3. The cross-frame sensing invalid data recovery method based on erasure coding according to claim 2, characterized in that, After sorting the cross-rack update network traffic corresponding to each erasure code stripe in ascending order, the method further includes: Starting with the erasure code stripe with the largest sequence number, select each erasure code stripe in reverse order until it is determined that the total number of all valid data blocks in all selected erasure code stripes is not greater than the total number of invalid data blocks in all unselected erasure code stripes, and then stop the erasure code stripe selection process; All selected erasure code stripes are designated as the recycled stripes, and all unselected erasure code stripes are designated as the fill stripes.

4. The cross-frame sensing invalid data retrieval method based on erasure coding according to any one of claims 1-3, characterized in that, The step of migrating valid data blocks from each of the reclaimed strips to the storage node where invalid data blocks from each of the filling strips reside includes: Following a strategy that prioritizes data migration within the same rack, valid data blocks from each of the reclaimed strips are migrated to the storage nodes where invalid data blocks from each of the filling strips reside.

5. A cross-rack sensing invalid data retrieval device based on erasure coding, characterized in that, include: The processing module is used to determine the cross-rack update network traffic generated by each erasure code stripe simulating the recovery of the invalid data blocks based on the invalid data blocks and valid check blocks stored in each rack. The partitioning module is used to partition each erasure code stripe into a fill stripe and a reclaimed stripe based on the cross-rack update network traffic corresponding to each erasure code stripe; the cross-rack update network traffic corresponding to the fill stripe is not greater than the cross-rack update network traffic corresponding to the reclaimed stripe; The recycling module is used to migrate the valid data blocks of each of the recycled strips to the storage nodes where the invalid data blocks of each of the filled strips are located, and to release the storage space of each of the recycled strips; The update module is used to update the valid verification blocks of each of the filling strips; The step of determining the cross-rack update network traffic generated by simulating the reclamation of invalid data blocks for each erasure coding stripe stored in each rack, based on the invalid data blocks and valid check blocks of each erasure coding stripe stored in each rack, includes: Determine the collection rack corresponding to each erasure code stripe from each of the racks; Based on the distribution of invalid data blocks and valid parity blocks in each erasure coding stripe and the collection rack corresponding to each erasure coding stripe, the target cross-rack update network traffic generated by executing a preset recycling strategy for each erasure coding stripe is determined; the preset recycling strategy is used to execute the strategy of transmitting all invalid data blocks of each erasure coding stripe to the corresponding collection rack to overwrite the valid parity blocks, and updating the remaining valid parity blocks in the corresponding collection rack; Based on the target cross-rack update network traffic, the cross-rack update network traffic generated by each erasure code stripe simulating the reclamation of the invalid data block is obtained; The step of determining the collection rack corresponding to each erasure code stripe from the various racks includes: Obtain the distribution information of the number of valid parity blocks stored in each rack for each erasure code stripe; The rack that has the largest number of valid check blocks for storing erasure code stripes is determined as the collection rack corresponding to each erasure code stripe.

6. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-4.

8. A computer program product, characterized in that, When the computer program product is run on a processor, the processor causes the processor to perform the method as described in any one of claims 1-4.

Images