Cross-frame sensing invalid data recovery method and device based on erasure codes

Abstract

The invention belongs to the technical field of computer storage, and particularly discloses a cross-frame perception invalid data recovery method and device based on erasure codes. The method comprises the following steps: based on an invalid data block and an effective check block of each erasure code strip stored in each rack, determining cross-rack update network traffic generated by each erasure code strip for simulating and recycling the invalid data block; dividing each erasure code strip into a filling strip and a recycling strip based on the cross-frame updating network traffic corresponding to each erasure code strip; migrating the valid data block of each recovery strip to the storage node where the invalid data block in each filling strip is located, and releasing the storage space of each recovery strip; and updating the effective verification block of each filling strip. By means of the invalid data recovery method and device, a more scientific and efficient invalid data recovery mode is achieved, and the storage efficiency of the system can be guaranteed while the network flow in the whole invalid data recovery process is effectively reduced and the invalid data recovery efficiency is improved.

Description

Technical Field

[0001] The present application belongs to the field of computer storage technology, and more specifically, to a method and device for recovering invalid data across racks based on erasure codes. Background Art

[0002] Compared with the replication mechanism, erasure coding can achieve high data reliability while ensuring low storage overhead, so it is widely deployed in today's cloud storage systems. In cloud storage systems based on erasure coding, deletion operations will generate invalid data. As invalid data increases, cloud storage systems must regularly call recycling mechanisms to release storage space for invalid data to ensure storage efficiency. However, in order to ensure the encoding consistency of data blocks and check blocks in erasure code stripes, cloud storage systems based on erasure coding cannot directly delete invalid data blocks like replication mechanisms. Therefore, how to efficiently recycle invalid data in cloud storage systems has always been a focus of research in the industry.

[0003] The existing recycling mechanism provides a method for recoding data recycling, which directly reorganizes the remaining valid data blocks in the old stripe with invalid data blocks into a new erasure code stripe to maximize data availability and storage efficiency. However, this method will introduce a large amount of inter-node network traffic during the stripe reorganization process. At the same time, the recoding recycling technology also ignores the multi-layer rack network architecture of the cloud storage system, further introducing a large amount of cross-rack network traffic, resulting in excessive network traffic during the invalid data recovery process and low invalid data recovery efficiency.

[0004] Therefore, how to better realize invalid data recovery in cloud storage systems based on erasure codes has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0005] In view of the defects of the prior art, the purpose of this application is to better realize the invalid data recovery of the cloud storage system based on erasure code, aiming to solve the problems of excessive network traffic and low invalid data recovery efficiency in the invalid data recovery process in the prior art.

[0006] To achieve the above objectives, in a first aspect, the present application provides a cross-frame-aware invalid data recovery method based on erasure codes, comprising: Based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, determining the cross-rack update network traffic generated by simulating the recovery of the invalid data blocks for each erasure code stripe; Based on the cross-rack update network traffic corresponding to each of the erasure code stripes, each of the erasure code stripes is divided into a filling stripe and a recycling stripe; the cross-rack update network traffic corresponding to the filling stripe is not greater than the cross-rack update network traffic corresponding to the recycling stripe; Migrating valid data blocks of each of the recycled stripes to storage nodes where invalid data blocks in each of the filled stripes are located, and releasing storage space of each of the recycled stripes; The valid check blocks of each of the filling strips are updated.

[0007] Optionally, the determining, based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, the cross-rack update network traffic generated by simulating the recovery of the invalid data blocks of each erasure code stripe comprises: Determine a collection rack corresponding to each of the erasure code stripes from each of the racks; Based on the distribution of invalid data blocks and valid check blocks in each of the erasure code stripes and the collection rack corresponding to each of the erasure code stripes, determine the target cross-rack update network traffic generated by each of the erasure code stripes executing a preset recycling strategy; the preset recycling strategy is used to execute a strategy of transmitting all invalid data blocks of each of the erasure code stripes to the corresponding collection rack to cover the valid check blocks, and updating the remaining valid check blocks in the corresponding collection rack; According to the target cross-rack update network traffic, the cross-rack update network traffic generated by each erasure code stripe simulating the recovery of the invalid data block is obtained.

[0008] Optionally, determining a collection rack corresponding to each erasure code stripe from each rack includes: Obtaining the quantity distribution information of the valid check blocks stored in each of the racks for each of the erasure code stripes; Determine that a rack storing the erasure code stripe with the largest number of valid check blocks is a collection rack corresponding to each of the erasure code stripes.

[0009] Optionally, the dividing each erasure code stripe into a filling stripe and a recycling stripe based on the cross-rack updating network traffic corresponding to each erasure code stripe comprises: Sorting the cross-rack update network traffic corresponding to each of the erasure code stripes in ascending order; Starting from the erasure code stripe with the smallest serial number, select each erasure code stripe in order until it is determined that the total number of all invalid data blocks in all the selected erasure code stripes is not less than the total number of valid data blocks in all the unselected erasure code stripes, and then stop the erasure code stripe selection process; All the selected erasure code stripes are divided into the filling stripes, and all the unselected erasure code stripes are divided into the recycling stripes.

[0010] Optionally, after sorting the cross-rack update network traffic corresponding to each erasure code stripe in ascending order, the method further includes: Starting from the erasure code stripe with the largest sequence number, select each erasure code stripe in reverse order until it is determined that the total number of all valid data blocks in all the selected erasure code stripes is not greater than the total number of invalid data blocks in all the unselected erasure code stripes, and then stop the erasure code stripe selection process; All the selected erasure code stripes are divided into the recycling stripes, and all the unselected erasure code stripes are divided into the filling stripes.

[0011] Optionally, the migrating the valid data blocks of each of the reclaimed stripes to the storage nodes where the invalid data blocks in each of the filled stripes are located includes: According to the strategy of completing data migration in the same rack first, the valid data blocks of each of the reclaimed stripes are migrated to the storage nodes where the invalid data blocks in each of the filled stripes are located.

[0012] In a second aspect, the present application provides a cross-frame-aware invalid data recovery device based on erasure codes, comprising: A processing module, configured to determine, based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, the cross-rack update network traffic generated by simulating the recovery of the invalid data blocks of each erasure code stripe; A partitioning module is used to partition each of the erasure code stripes into a filling stripe and a recycling stripe based on the cross-rack update network traffic corresponding to each of the erasure code stripes; the cross-rack update network traffic corresponding to the filling stripe is not greater than the cross-rack update network traffic corresponding to the recycling stripe; A recycling module, used to migrate the valid data blocks of each of the recycled stripes to the storage nodes where the invalid data blocks in each of the filled stripes are located, and release the storage space of each of the recycled stripes; An updating module is used to update the valid check blocks of each of the filling strips.

[0013] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0015] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0016] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art: The cross-rack-aware invalid data recovery method and device based on erasure codes provided in the present application treats invalid data blocks as data blocks that need to be updated, simulates and calculates the cross-rack update network traffic generated by each erasure code stripe for recovering invalid data blocks, and according to the sorting of these cross-rack update network traffics, preferentially divides the stripes with lower cross-rack update network traffic in each erasure code stripe as filling stripes, and the rest are divided into recovery stripes, and uses data filling to migrate the valid data blocks of each recovery stripe to the filling stripe with low update traffic to cover its invalid data blocks, thereby releasing the storage space of each recovery stripe, realizing a more scientific and efficient invalid data recovery method, which can effectively reduce the network traffic in the entire invalid data recovery process, improve the invalid data recovery efficiency, and ensure the storage efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is one of the flow charts of the cross-frame-aware invalid data recovery method based on erasure code provided by this application; Figure 2 This is the second flow chart of the cross-frame-aware invalid data recovery method based on erasure code provided in the embodiment of the present application; Figure 3 It is a structural diagram of a cross-frame-aware invalid data recovery device based on erasure codes provided in an embodiment of the present application; Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0020] First, the technical terms involved in the embodiments of the present application are introduced.

[0021] (1) Erasure Code Stripe Through erasure coding technology, the original data can be divided into several data blocks, and then several check blocks are generated by performing specific matrix operations on these data blocks. These data blocks and the check blocks form an erasure code stripe, which can also be described as an erasure code stripe.

[0022] For example, in an erasure code based In the storage system, a file is divided into The original data blocks are encoded into Total data blocks, of which, ,in, is the number of check blocks, The collection of total data blocks is called an "erasure code stripe". An erasure code stripe is the smallest coding unit in the erasure code. Usually, each data block in an erasure code stripe is stored in a different storage node. The system can tolerate the failure of any m nodes. If a node fails, the remaining Repair the invalid data in each node.

[0023] (2) Rack A rack is a physical support structure for storage devices (such as hard disks, servers, etc.), and is usually used in data centers or large storage systems. In a distributed storage system based on erasure codes, multiple racks can form a cluster, and each rack contains multiple storage nodes (such as servers).

[0024] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0025] Figure 1 This is one of the flow charts of the cross-frame-aware invalid data recovery method based on erasure code provided by this application, such as Figure 1 As shown, including: Step S1, based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, determining the cross-rack update network traffic generated by simulating the recovery of invalid data blocks of each erasure code stripe; Step S2, based on the cross-rack update network traffic corresponding to each erasure code stripe, each erasure code stripe is divided into a filling stripe and a recycling stripe; the cross-rack update network traffic corresponding to the filling stripe is not greater than the cross-rack update network traffic corresponding to the recycling stripe; Step S3, migrating the valid data blocks of each reclaimed stripe to the storage nodes where the invalid data blocks in each filled stripe are located, and releasing the storage space of each reclaimed stripe; Step S4, updating the valid check blocks of each filling stripe.

[0026] Specifically, the invalid data blocks described in the embodiments of the present application generally refer to the data blocks corresponding to the data deleted as indicated by the deletion request input by the front-end user. When the data is deleted by the user, the storage system will not delete the data in the disk immediately, but mark the deleted data as invalid, thereby generating invalid data blocks to ensure the coding consistency of the erasure code stripe.

[0027] The cross-rack update network traffic described in the embodiment of the present application refers to the amount of data that needs to be transmitted between storage nodes on different racks when the system performs an erasure code stripe data update operation.

[0028] Specifically, when a data block needs to be updated, due to the encoding characteristics of the erasure code, all related check blocks need to be updated. These check blocks may be distributed on different racks. Therefore, the update operation needs to read the relevant check blocks from the storage nodes on other racks through the network and write the updated check blocks back to the corresponding storage nodes. The amount of data transmitted in this process constitutes the cross-rack update network traffic.

[0029] The reclaimed stripe described in the embodiment of the present application refers to an erasure code stripe corresponding to the subsequent execution of valid data block migration.

[0030] The padding stripe described in the embodiment of the present application refers to an erasure code stripe where invalid data blocks that need to be covered by the migration of valid data blocks in the subsequent recovery stripe are located.

[0031] In an embodiment of the present application, in step S1, according to the distribution of invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, the invalid data blocks of each erasure code stripe can be regarded as data blocks that need to be updated, and the process of checking and updating the relevant check blocks in the erasure code stripe is simulated, and the cross-rack update network traffic generated by the simulated recovery of invalid data blocks for each erasure code stripe is calculated, thereby determining the size of the cross-rack update network traffic required for the recovery of invalid data blocks in each erasure code stripe.

[0032] In an embodiment of the present application, in step S2, a pre-set stripe selection strategy is used to sort the traffic size according to the cross-rack update network traffic corresponding to each erasure code stripe, and then the erasure code stripe with smaller cross-rack update network traffic is selected as the filling stripe as much as possible, thereby dividing each erasure code stripe into filling stripes and recycling stripes.

[0033] It can be understood that by selectively selecting erasure code stripes with smaller cross-rack update network traffic as filling strips for performing subsequent invalid data overwriting, it is possible to ensure that the cross-rack update network traffic generated in the process is smaller while recovering invalid data blocks. This can reduce the cross-rack update network traffic of the system in the process of recovering invalid data, thereby reducing the network traffic of the entire system in the invalid data recovery process.

[0034] Further, in an embodiment of the present application, in step S3, on the basis of determining the filling stripe and the recycling stripe, the valid data blocks of each recycling stripe are migrated to the storage node where the invalid data blocks in each filling stripe are located, that is, the coverage of the invalid data blocks by the valid data blocks is completed. Since the valid data blocks in the recycling stripe are migrated out, the storage space of the recycling stripe can be released. Therefore, the storage space originally occupied by the valid data blocks in each recycling stripe can be released, thereby completing the recycling of invalid data.

[0035] At the same time, the invalid data blocks in the filling stripe are overwritten by the valid data blocks in the recycled stripe, so the check blocks in the filling stripe need to be updated.

[0036] Specifically, in an embodiment of the present application, in step S4, in order to ensure the coding consistency of the erasure code stripes, when new data is migrated into each padding stripe and the valid data block is updated, it is also necessary to update each valid check block in the corresponding padding stripe according to the updated valid data block, and write the updated check block back to the corresponding storage node, so as to ensure the storage efficiency of the system.

[0037] The cross-rack-aware invalid data recovery method based on erasure codes in the embodiment of the present application regards invalid data blocks as data blocks that need to be updated, simulates and calculates the cross-rack update network traffic generated by each erasure code stripe for recovering invalid data blocks, and according to the sorting of these cross-rack update network traffics, prioritizes the stripes with lower cross-rack update network traffic in each erasure code stripe as filling stripes, and the rest are divided into recovery stripes, and uses data filling to migrate the valid data blocks of each recovery stripe to the filling stripe with low update traffic to cover its invalid data blocks, thereby releasing the storage space of each recovery stripe, realizing a more scientific and efficient invalid data recovery method, which can effectively reduce the network traffic in the entire invalid data recovery process, improve the invalid data recovery efficiency, and ensure the storage efficiency of the system.

[0038] Based on the content of the above embodiment, as an optional embodiment, step S1, based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, determines the cross-rack update network traffic generated by simulating the recovery of invalid data blocks of each erasure code stripe, including: Determine the collection rack corresponding to each erasure code stripe from each rack; Based on the distribution of invalid data blocks and valid check blocks in each erasure code stripe and the collection rack corresponding to each erasure code stripe, determine the target cross-rack update network traffic generated by each erasure code stripe executing a preset recycling strategy; the preset recycling strategy is used to execute a strategy of transmitting all invalid data blocks of each erasure code stripe to the corresponding collection rack to cover the valid check blocks, and updating the remaining valid check blocks in the corresponding collection rack; According to the target cross-rack update network traffic, the cross-rack update network traffic generated by each erasure code stripe simulating the recovery of invalid data blocks is obtained.

[0039] Specifically, the collection rack described in the embodiment of the present application refers to a target rack set from each rack for receiving invalid data blocks in an erasure code stripe during the process of simulating the recovery of invalid data blocks in each erasure code stripe.

[0040] The target cross-rack update network traffic described in the embodiment of the present application refers to the cross-rack update network traffic generated during the process of each erasure code stripe simulating the execution of transferring all invalid data blocks in the band to the corresponding collection rack to cover the valid check blocks stored therein, and updating the remaining valid check blocks in the corresponding collection rack.

[0041] In the embodiment of the present application, it is necessary to first determine the collection rack corresponding to each erasure code stripe from each rack.

[0042] Based on the content of the above embodiment, as an optional embodiment, determining the collection rack corresponding to each erasure code stripe from each rack includes: Obtain the number distribution information of valid check blocks stored in each rack for each erasure code stripe; The rack with the largest number of valid check blocks for storing the erasure code stripe is determined as the collection rack corresponding to each erasure code stripe.

[0043] Specifically, in an embodiment of the present application, assuming that an invalid data block is updated, according to the distribution of invalid data blocks and valid check blocks in each erasure code stripe, the number distribution information of valid check blocks stored in each rack of each erasure code stripe is obtained, and then a rack with the largest number of valid check blocks is selected from each rack as a collection rack, so as to obtain the collection rack corresponding to each erasure code stripe.

[0044] The method of the embodiment of the present application selects a suitable collection rack according to the recovery status of invalid data blocks and the number distribution of valid check blocks in each erasure code stripe, so as to select a suitable stripe division method in the subsequent data recovery and update stage, which is beneficial to further reduce the cross-rack update network traffic.

[0045] Furthermore, in an embodiment of the present application, for each erasure code stripe, all invalid blocks can be transmitted to the corresponding collection rack according to a preset recovery strategy, and the collection rack updates the remaining valid check blocks, thereby calculating the target cross-rack update network traffic corresponding to each erasure code stripe based on the data transmission and update process.

[0046] Finally, the target cross-rack update network traffic corresponding to each erasure code stripe is taken as the cross-rack update network traffic generated by each erasure code stripe to simulate the recovery of invalid data blocks.

[0047] The method of the embodiment of the present application performs data updating by simulating invalid data blocks as update data, adopts the method of collaborative updating of erasure code data, predicts the cross-rack update network traffic generated by each erasure code stripe simulating the recovery of invalid data blocks, provides a technical basis for the subsequent division of recovery stripes and filling stripes, and ensures that the cross-rack update network traffic is minimized during the data recovery process.

[0048] Based on the content of the above embodiment, as an optional embodiment, each erasure code stripe is divided into a filling stripe and a recycling stripe based on the cross-rack update network traffic corresponding to each erasure code stripe, including: Sort the cross-rack update network traffic corresponding to each erasure code stripe from small to large; Starting from the erasure code stripe with the smallest serial number, select each erasure code stripe in order until it is determined that the total number of all invalid data blocks in all the selected erasure code stripes is not less than the total number of valid data blocks in all the unselected erasure code stripes, and then stop the erasure code stripe selection process; All selected erasure code stripes are divided into filling stripes, and all unselected erasure code stripes are divided into recycling stripes.

[0049] Specifically, in the embodiment of the present application, there are two implementation methods for dividing each erasure code stripe into a filling stripe and a recycling stripe. The first is to sort by the size of the cross-rack update network traffic and select in sequence from the side of the erasure code stripe with the smallest traffic.

[0050] More specifically, the cross-rack update network traffic corresponding to each erasure code stripe is sorted in ascending order, so that the erasure code stripe with the smallest cross-rack update network traffic corresponds to the smallest sequence number, and the erasure code stripe with the largest cross-rack update network traffic corresponds to the largest sequence number.

[0051] Furthermore, in this embodiment, starting from the erasure code stripe with the smallest serial number, each erasure code stripe is selected in order, and each time an erasure code stripe is selected, the difference between the number of invalid data blocks in all selected stripes and the number of valid data blocks in the remaining unselected stripes is calculated. Until it is determined that the total number of all invalid data blocks in all selected erasure code stripes is not less than the total number of valid data blocks in all unselected erasure code stripes, it means that all valid data blocks in the subsequent recovered stripes can be completely migrated out, and there will be no situation where there are valid data blocks that cannot be migrated out and the storage space of the recovered stripe cannot be released. At this time, the erasure code stripe selection process is stopped.

[0052] Therefore, all selected erasure code stripes may be divided into filling stripes, and all remaining erasure code stripes that are not selected may be divided into recycling stripes.

[0053] The method of the embodiment of the present application sorts the cross-rack update network traffic corresponding to each erasure code stripe in ascending order, starting from the erasure code stripe with the smallest sequence number, and selects the stripe with smaller cross-rack update network traffic as the filling stripe and the remaining stripes as the recovery stripe according to the relationship between the number of invalid data blocks in the filling stripe and the valid data blocks in the recovery stripe, so as to perform efficient recovery of invalid data later and reduce the cross-rack update network traffic during the recovery process.

[0054] Based on the content of the above embodiment, as an optional embodiment, after sorting the cross-rack update network traffic corresponding to each erasure code stripe in ascending order, the method further includes: Starting from the erasure code stripe with the largest sequence number, select each erasure code stripe in reverse order until it is determined that the total number of all valid data blocks in all the selected erasure code stripes is not greater than the total number of invalid data blocks in all the unselected erasure code stripes, and then stop the erasure code stripe selection process; All selected erasure code stripes are divided into recycling stripes, and all unselected erasure code stripes are divided into filling stripes.

[0055] Specifically, in the embodiments of the present application, there are two implementation methods for dividing each erasure code stripe into a filling stripe and a recycling stripe. The second implementation method can be the opposite of the first implementation method, that is, sorting by the size of the cross-rack update network traffic and screening in sequence from the side of the erasure code stripe with the largest traffic.

[0056] More specifically, after sorting the cross-frame update network traffic corresponding to each erasure code stripe in ascending order, further, starting from the erasure code stripe with the largest serial number, each erasure code stripe is selected in sequence. During the process, each time an erasure code stripe is selected, the difference between the total number of all valid data blocks in all selected stripes and the total number of invalid data blocks in all remaining unselected stripes is calculated. Until it is determined that the total number of all valid data blocks in all selected erasure code stripes is not greater than the total number of invalid data blocks in all unselected erasure code stripes, to ensure that all valid data blocks in the subsequent recycled stripes can be completely migrated out and the storage space of the recycled stripes can be released. At this point, the erasure code stripe selection process is stopped.

[0057] Therefore, all the selected erasure code stripes can be divided into recycling stripes, and all the remaining erasure code stripes that are not selected can be divided into filling stripes.

[0058] The method of the embodiment of the present application, by sorting the cross-rack update network traffic corresponding to each erasure code stripe in ascending order, starting from the erasure code stripe with the largest sequence number, and according to the relationship between the number of invalid data blocks in the padding stripe and the valid data blocks in the recovery stripe, selects the stripe with larger cross-rack update network traffic as the recovery stripe, and the remaining stripes as the padding stripes, so as to reduce the cross-rack update network traffic in the recovery process and facilitate the subsequent efficient execution of invalid data recovery.

[0059] Based on the content of the above embodiment, as an optional embodiment, migrating valid data blocks of each reclaimed stripe to the storage node where the invalid data blocks in each filled stripe are located includes: According to the strategy of completing data migration in the same rack first, the valid data blocks of each reclaimed stripe are migrated to the storage nodes where the invalid data blocks in each filled stripe are located.

[0060] Specifically, in an embodiment of the present application, all valid data blocks in the recycled stripe are migrated to the storage location where the invalid data blocks in the filling stripe are located. For valid data blocks in the recycled stripe in the same rack, they are preferentially migrated to the node location where the invalid data blocks in the filling stripe in the same rack are located, thereby further avoiding the generation of cross-rack update network traffic during data migration.

[0061] Figure 2 This is the second flow chart of the cross-frame perception invalid data recovery method based on erasure code provided in the embodiment of the present application, such as Figure 2 As shown, in a specific embodiment of the present application, there are five The erasure code stripes encoded by the storage system have invalid data blocks in each erasure code stripe. Blocks with the same digital number belong to the same erasure code stripe, which includes erasure code stripes S1, S2, S3, S4 and S5. The valid data blocks, valid check blocks and invalid data blocks in the five erasure code stripes are distributed on the storage nodes of the five racks, where the five racks are represented as R1, R2, R3, R4 and R5. As can be seen from the figure, the number of valid data blocks in these erasure code stripes is 2, 3, 2, 3, 2, respectively, represented as 2 (S1), 3 (S2), 2 (S3), 3 (S4), 2 (S5).

[0062] Further, according to the cross-rack update network traffic calculation method in the aforementioned embodiment, for the erasure code stripe S1, it is determined that the collection rack is R4 with the largest number of valid check blocks, and a total of 2 invalid data blocks in R3 and R5 are regarded as updated data, which are respectively transmitted to the corresponding 2 valid check blocks in R4. The cross-rack update network traffic corresponding to the erasure code stripe S1 can be quantified as 2 blocks. Similarly, for the erasure code stripe S2, its corresponding cross-rack update network traffic can be quantified as 1 block; the erasure code stripe S3 can be quantified as 3; the erasure code stripe S4 can be quantified as 4; the erasure code stripe S5 can be quantified as 2. Therefore, the cross-rack update network traffic of these stripes is sorted in ascending order, which can be expressed as: 1 (S2), 2 (S5), 2 (S4), 2(S1), 3 (S3).

[0063] Reference Figure 2 It can be seen that after three rounds of selection, all erasure code stripes selected in sequence are represented as fiS={S2, S5, S4}, and all remaining erasure code stripes not selected are represented as reS={S1, S3}. Among them, the total number of all invalid data blocks in the erasure code stripes {S2, S5, S4} is 4, which is not less than the total number of valid data blocks in all erasure code stripes {S1, S3} that have not been selected. Therefore, the erasure code stripes {S2, S5, S4} with less cross-rack update traffic are selected as filling stripes, and the remaining erasure code stripes {S1, S3} are used as recycling stripes.

[0064] Furthermore, the valid data blocks in the recycled stripes {S1, S3} are migrated to the storage location where the invalid data blocks in the filling stripes {S2, S5, S4} are located. For example, the valid data blocks of the recycled stripe S1 in R1 can be migrated within the rack according to the intra-rack link to avoid the generation of cross-rack update network traffic, thereby overwriting the invalid data blocks of the filling stripe S5 in R1; the valid data blocks of the recycled stripe S1 in R2 are migrated through the cross-rack link to overwrite another invalid data block of the filling stripe S5 in R1, and so on. By analogy, the storage space of the recycled stripes {S1, S3} in each rack can be released eventually, thereby completing the recycling of invalid data blocks in the 5 erasure code stripes.

[0065] The invalid data recovery method of the embodiment of the present application adopts a data filling algorithm to migrate the valid data in a part of the erasure code stripe to the location of the invalid data in another part of the erasure code stripe for the cross-rack network architecture, so as to reduce the cross-rack update network traffic. The migration process is completed in the same rack as much as possible, and finally the storage space of the recovery stripe is released and the check block in the filling stripe is updated. Through the cross-rack update network traffic sorting and intra-rack migration strategy in this method, the cross-rack update network traffic generated during the invalid data recovery process of the system can be greatly reduced, and the recovery efficiency of invalid data can be improved.

[0066] The cross-frame aware invalid data recovery device based on erasure codes provided in the present application is described below. The cross-frame aware invalid data recovery device based on erasure codes described below and the cross-frame aware invalid data recovery method based on erasure codes described above can be referenced to each other.

[0067] Figure 3 is a structural diagram of a cross-frame-aware invalid data recovery device based on erasure codes provided in an embodiment of the present application, such as Figure 3 As shown, including: The processing module 10 is used to determine the cross-rack update network traffic generated by simulating the recovery of invalid data blocks of each erasure code stripe based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack; A partitioning module 20 is used to partition each erasure code stripe into a filling stripe and a recycling stripe based on the cross-rack update network traffic corresponding to each erasure code stripe; the cross-rack update network traffic corresponding to the filling stripe is not greater than the cross-rack update network traffic corresponding to the recycling stripe; The recycling module 30 is used to migrate the valid data blocks of each recycled stripe to the storage nodes where the invalid data blocks in each filled stripe are located, and release the storage space of each recycled stripe; The updating module 40 is used to update the valid check blocks of each filling stripe.

[0068] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0069] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0070] The cross-rack-aware invalid data recovery device based on erasure codes in the embodiment of the present application treats invalid data blocks as data blocks that need to be updated, simulates and calculates the cross-rack update network traffic generated by each erasure code stripe for recovering invalid data blocks, and according to the sorting of these cross-rack update network traffics, prioritizes the stripes with lower cross-rack update network traffic in each erasure code stripe as filling stripes, and the rest are divided into recovery stripes, and uses data filling to migrate the valid data blocks of each recovery stripe to the filling stripe with low update traffic to cover its invalid data blocks, thereby releasing the storage space of each recovery stripe, realizing a more scientific and efficient invalid data recovery method, which can effectively reduce the network traffic in the entire invalid data recovery process, improve the invalid data recovery efficiency, and ensure the storage efficiency of the system.

[0071] Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, such as Figure 4 As shown, the electronic device may include: a processor (Processor) 410, a communication interface (Communications Interface) 420, a memory (Memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the method in the above embodiment.

[0072] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0073] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0074] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0075] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0076] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0077] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0078] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0079] It should be understood that expressions such as "including" and "may include" that may be used in the present application indicate the existence of the disclosed functions, operations, or constituent elements, and do not limit one or more additional functions, operations, and constituent elements. In the present application, terms such as "including" and / or "having" may be interpreted as indicating specific characteristics, numbers, operations, constituent elements, components, or combinations thereof, but may not be interpreted as excluding the existence or possibility of adding one or more other characteristics, numbers, operations, constituent elements, components, or combinations thereof.

[0080] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A cross-frame-aware invalid data recovery method based on erasure codes, characterized in that: include: Based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, determining the cross-rack update network traffic generated by simulating the recovery of the invalid data blocks for each erasure code stripe; Based on the cross-rack update network traffic corresponding to each of the erasure code stripes, each of the erasure code stripes is divided into a filling stripe and a recycling stripe; the cross-rack update network traffic corresponding to the filling stripe is not greater than the cross-rack update network traffic corresponding to the recycling stripe; Migrating valid data blocks of each of the recycled stripes to storage nodes where invalid data blocks in each of the filled stripes are located, and releasing storage space of each of the recycled stripes; The valid check blocks of each of the filling strips are updated.

2. The method for recovering invalid data across racks based on erasure codes according to claim 1, characterized in that: The determining, based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, cross-rack update network traffic generated by simulating the recovery of the invalid data blocks of each erasure code stripe comprises: Determine a collection rack corresponding to each of the erasure code stripes from each of the racks; Based on the distribution of invalid data blocks and valid check blocks in each of the erasure code stripes and the collection rack corresponding to each of the erasure code stripes, determine the target cross-rack update network traffic generated by each of the erasure code stripes executing a preset recycling strategy; the preset recycling strategy is used to execute a strategy of transmitting all invalid data blocks of each of the erasure code stripes to the corresponding collection rack to cover the valid check blocks, and updating the remaining valid check blocks in the corresponding collection rack; According to the target cross-rack update network traffic, the cross-rack update network traffic generated by each erasure code stripe simulating the recovery of the invalid data block is obtained.

3. The method for recovering invalid data across racks based on erasure codes according to claim 2 is characterized in that: The step of determining the collection rack corresponding to each erasure code stripe from each of the racks includes: Obtaining the quantity distribution information of the valid check blocks stored in each of the racks for each of the erasure code stripes; Determine that a rack storing the erasure code stripe with the largest number of valid check blocks is a collection rack corresponding to each of the erasure code stripes.

4. The cross-frame aware invalid data recovery method based on erasure coding according to claim 1 is characterized in that: The method of dividing each erasure code stripe into a filling stripe and a recycling stripe based on the cross-rack updating network traffic corresponding to each erasure code stripe comprises: Sorting the cross-rack update network traffic corresponding to each of the erasure code stripes in ascending order; Starting from the erasure code stripe with the smallest serial number, select each erasure code stripe in order until it is determined that the total number of all invalid data blocks in all the selected erasure code stripes is not less than the total number of valid data blocks in all the unselected erasure code stripes, and then stop the erasure code stripe selection process; All the selected erasure code stripes are divided into the filling stripes, and all the unselected erasure code stripes are divided into the recycling stripes.

5. The method for recovering invalid data across racks based on erasure codes according to claim 4 is characterized in that: After sorting the cross-rack update network traffic corresponding to each of the erasure code stripes in ascending order, the method further includes: Starting from the erasure code stripe with the largest sequence number, select each erasure code stripe in reverse order until it is determined that the total number of all valid data blocks in all the selected erasure code stripes is not greater than the total number of invalid data blocks in all the unselected erasure code stripes, and then stop the erasure code stripe selection process; All the selected erasure code stripes are divided into the recycling stripes, and all the unselected erasure code stripes are divided into the filling stripes.

6. The method for recovering invalid data across racks based on erasure codes according to any one of claims 1 to 5, characterized in that: The step of migrating the valid data blocks of each of the recycled stripes to the storage nodes where the invalid data blocks in each of the filled stripes are located includes: According to the strategy of completing data migration in the same rack first, the valid data blocks of each of the reclaimed stripes are migrated to the storage nodes where the invalid data blocks in each of the filled stripes are located.

7. A cross-frame aware invalid data recovery device based on erasure code, characterized in that: include: A processing module, configured to determine, based on the invalid data blocks and valid check blocks of each erasure code stripe stored in each rack, the cross-rack update network traffic generated by simulating the recovery of the invalid data blocks of each erasure code stripe; A partitioning module is used to partition each of the erasure code stripes into a filling stripe and a recycling stripe based on the cross-rack update network traffic corresponding to each of the erasure code stripes; the cross-rack update network traffic corresponding to the filling stripe is not greater than the cross-rack update network traffic corresponding to the recycling stripe; A recycling module, used to migrate the valid data blocks of each of the recycled stripes to the storage nodes where the invalid data blocks in each of the filled stripes are located, and release the storage space of each of the recycled stripes; An updating module is used to update the valid check blocks of each of the filling strips.

8. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

Images