Method, device, equipment and storage medium for reducing network overhead of erasure code storage

Abstract

A method, device, equipment and storage medium for reducing network overhead of erasure code storage, the method comprising: adjusting a delayed elimination queue according to the identification of the stripe of each write or delete operation each time a stripe is written or deleted; determining a replica data processing strategy for each full stripe according to the position of the identification of the full stripe in the delayed elimination queue each time an aggregation task is performed; after aggregating the replica data of the full stripe, if the replica data processing strategy is a deletion strategy, the replica data is deleted; if the replica data processing strategy is a non-deletion strategy, the replica data is not deleted. Through the present application, it is possible to identify whether a stripe belongs to a hotspot stripe. If a stripe belongs to a hotspot stripe, its replica data will not be deleted after the stripe is fully aggregated, thereby greatly reducing the network overhead generated by the hotspot stripe when restoring replica data or updating verification data.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a method, device, equipment and storage medium for reducing network overhead of erasure code storage. Background Art

[0002] Nowadays, the amount of data is growing explosively. In order to improve the reliability of data storage, replicas and erasure codes are two commonly used redundancy strategies in the field of distributed storage. For the erasure code redundant storage strategy, in order to reduce the performance impact on hard disk writing, the replica write solution came into being. Figure 1 , Figure 1 The flowchart of the erasure code copy writing solution is as follows: Figure 1 As shown in the figure, the main process of the erasure code replica writing scheme is: when the stripe is full, the verification data corresponding to the stripe data is calculated, and then the original data block is written to the hard disk together with the verification data block. When the stripe is not full, it is written as a whole in the form of a copy to the corresponding data shard and the verification shard. There is a background task running on the verification shard, which is responsible for regularly checking the data written to the stripe. When it is detected that the stripe is full, the replica data on the stripe will be aggregated into verification data, and then the replica data will be deleted.

[0003] The erasure code replica writing solution solves the write performance problem when the stripe is not full. At the same time, it cleverly uses background tasks to periodically aggregate the stripe data. After the stripe is full and the replica data is aggregated, the replica data will be deleted indiscriminately. Figure 2 , Figure 2 The following is a schematic diagram of the overwrite process of the erasure code copy writing scheme. Figure 2 As shown, when the data in the stripe is overwritten, and the stripe data has been aggregated and the replica data has been deleted before, refer to Figure 3 , Figure 3 The deletion process diagram of the erasure code copy writing scheme is as follows: Figure 3 As shown in the figure, when the data in the stripe is deleted, and the replica data of the stripe data has been aggregated before and has been deleted, in both cases, the replica data on the check shard has been deleted after the aggregation is completed, but because the data on the stripe has changed, the check shard needs to communicate with the data shard and pull data from the data shard to restore the replica data or update the check data. When a stripe becomes a hot spot for some reason, repeated overwrite and deletion operations will bring very large network overhead.

[0004] In summary, the current erasure code replica writing scheme will indiscriminately delete replica data after the stripe is full and the replica data is aggregated. When a stripe becomes a hot spot for some reason and repeatedly performs overwrite and deletion operations, the verification shard needs to frequently pull data from the data shard to restore the replica data or update the verification data, which will generate very large network overhead. Summary of the invention

[0005] The present application provides a method, apparatus, device and storage medium for reducing the network overhead of erasure code storage, and aims to solve the following technical problem: the current erasure code replica writing scheme will indiscriminately delete the replica data after the stripe is full and the replica data is aggregated. When a stripe becomes a hot spot for some reason and is repeatedly overwritten or deleted, the check shard needs to frequently pull data from the data shard to restore the replica data or update the check data, which will generate very large network overhead.

[0006] In a first aspect, an embodiment of the present application provides a method for reducing the network overhead of an erasure code storage, and the method for reducing the network overhead of an erasure code storage comprises:

[0007] Each time a stripe is written or deleted, a delayed elimination queue is adjusted according to the stripe identifier of each write or delete operation, the delayed elimination queue is used to store the stripe identifier of the write or delete operation, the delayed elimination queue includes an elimination sentinel, the elimination sentinel is used to indicate the identifier of a stripe in the delayed elimination queue, the delayed elimination queue takes the first one to join the queue as the tail and takes the last one to join the queue as the head;

[0008] Each time an aggregation task is executed, for each full stripe, the replica data processing strategy is determined according to the position of the full stripe identifier in the delayed elimination queue;

[0009] After the replica data of the fully written stripe is aggregated, if the replica data processing policy is a deletion policy, the replica data is deleted; if the replica data processing policy is a non-deletion policy, the replica data is not deleted.

[0010] Optionally, each time a write or delete operation is performed on a stripe, adjusting the delayed elimination queue according to an identifier of the stripe of each write or delete operation includes:

[0011] Each time a stripe is written or deleted, the identifier of the current stripe to be written or deleted is obtained;

[0012] When the identifier of the current stripe is not in the delayed elimination queue, the identifier of the current stripe is added to the head of the delayed elimination queue;

[0013] When the identifier of the current stripe is in the delayed elimination queue, it is detected whether the position of the identifier of the current stripe in the delayed elimination queue is the head of the queue, and whether the identifier of the current stripe is an elimination sentinel;

[0014] If it is not the head of the team and is not an eliminated sentinel, move the marker of the current strip to the head of the team;

[0015] If it is not the head of the team and is an elimination sentinel, the next stripe mark of the current stripe towards the end of the team is set as the new elimination sentinel, and the current stripe mark is moved to the head of the team;

[0016] If it is the head of the team and is an elimination sentinel, the next stripe mark of the current stripe towards the end of the team is set as the new elimination sentinel;

[0017] If it is the head of the queue and not an elimination sentinel, the delayed elimination queue will not be adjusted.

[0018] Optionally, determining the replica data processing strategy according to the position of the identifier of the full stripe in the delayed elimination queue includes:

[0019] If the identifier of the full stripe is in the delayed elimination queue and the position of the identifier of the full stripe in the delayed elimination queue is in the first range, the replica data processing strategy is determined to be a non-deletion strategy, and the first range is the position range from before the sentinel is eliminated to the head of the queue;

[0020] If the identifier of the full stripe is in the delayed elimination queue and the position of the identifier of the full stripe in the delayed elimination queue is in the second range, or the identifier of the full stripe is not in the delayed elimination queue, then the copy data processing strategy is determined to be the deletion strategy, and the second range is the position range from the elimination sentinel and after the elimination sentinel to the end of the queue.

[0021] Optionally, determining the replica data processing strategy according to the position of the identifier of the fully written stripe in the delayed elimination queue further includes:

[0022] Check whether the current disk usage is less than the preset disk usage;

[0023] If the current disk usage is less than the preset disk usage, the replica data processing strategy is determined according to the position of the identifier of the full stripe in the delayed elimination queue.

[0024] Optionally, after detecting whether the current disk usage is less than a preset disk usage, the method further includes:

[0025] If the current disk usage is greater than or equal to the preset disk usage, the copy data processing strategy is determined to be a deletion strategy.

[0026] Optionally, when the identifier of the current stripe is not in the delayed elimination queue, adding the identifier of the current stripe to the head of the delayed elimination queue includes:

[0027] When the identifier of the current stripe is not in the delayed elimination queue, detecting whether the number of stripe identifiers stored in the delayed elimination queue is greater than a preset number after the identifier of the current stripe is added to the head of the delayed elimination queue;

[0028] If yes, after deleting the stripe identifier stored at the tail of the delayed elimination queue, the identifier of the current stripe is added to the head of the delayed elimination queue;

[0029] If not, the identifier of the current strip is added to the head of the delayed elimination queue.

[0030] Optionally, after the replica data of the fully written stripe is aggregated, if the replica data processing policy is a deletion policy, the replica data is deleted; if the replica data processing policy is a non-deletion policy, the replica data is not deleted, and then the replica data is not deleted, including:

[0031] At the end of each aggregation task execution, delete the identifiers of all stripes in the delayed elimination queue that are in the range from the elimination sentinel to the tail of the queue, and set the head of the delayed elimination queue to the new elimination sentinel.

[0032] In a second aspect, an embodiment of the present application provides a device for reducing the network overhead of an erasure code storage, and the device for reducing the network overhead of an erasure code storage includes:

[0033] An adjustment module is used to adjust the delayed elimination queue according to the identification of the stripe of each write or delete operation each time the stripe is written or deleted, the delayed elimination queue is used to store the identification of the stripe of the write or delete operation, the delayed elimination queue includes an elimination sentinel, and the elimination sentinel is used to indicate the identification of a stripe in the delayed elimination queue, and the delayed elimination queue takes the first one to join the queue as the tail and the last one to join the queue as the head;

[0034] A determination module, used for determining a replica data processing strategy for each full stripe according to the position of the identifier of the full stripe in the delayed elimination queue each time an aggregation task is executed;

[0035] The processing module is used for deleting the replica data after aggregating the replica data of the fully written stripe if the replica data processing policy is a deletion policy, and not deleting the replica data if the replica data processing policy is a non-deletion policy.

[0036] In a third aspect, an embodiment of the present application provides a device for reducing the network overhead of an erasure code storage device, the device comprising a processor, a memory, and an erasure code storage network overhead reduction program stored in the memory and executable by the processor, wherein when the erasure code storage network overhead reduction program is executed by the processor, the steps of the method for reducing the network overhead of an erasure code storage device as described above are implemented.

[0037] In a fourth aspect, an embodiment of the present application provides a storage medium, on which is stored a program for reducing the storage network overhead of an erasure code. When the program for reducing the storage network overhead of an erasure code is executed by a processor, the steps of the method for reducing the storage network overhead of an erasure code as described above are implemented.

[0038] The beneficial effects brought by the technical solution provided by the embodiments of the present application include:

[0039] In an embodiment of the present application, each time a write or delete operation is performed on a stripe, a delayed elimination queue is adjusted according to the identifier of the stripe of each write or delete operation, the delayed elimination queue is used to store the identifier of the stripe of the write or delete operation, the delayed elimination queue includes an elimination sentinel, the elimination sentinel is used to indicate the identifier of a stripe in the delayed elimination queue, the delayed elimination queue has the first to join the queue as the tail and the last to join the queue as the head; each time an aggregation task is executed, for each full stripe, a replica data processing strategy is determined according to the position of the identifier of the full stripe in the delayed elimination queue; after aggregating the replica data of the full stripe, if the replica data processing strategy is a deletion strategy, the replica data is deleted, and if the replica data processing strategy is a non-deletion strategy, the replica data is not deleted. In the embodiment of the present application, by setting a delayed elimination queue and an elimination sentinel, each time a write or delete operation is performed on a stripe, the delayed elimination queue is adjusted according to the identifier of the stripe of each write or delete operation, so that for stripes that frequently perform write or delete operations, their identifiers will be adjusted as much as possible to the range from before the elimination sentinel to the head of the queue, and for stripes that seldom perform write or delete operations, their identifiers will be adjusted as much as possible to the range from the elimination sentinel and after the elimination sentinel to the tail of the queue. It can be seen that the delayed elimination queue is divided into a hot stripe area and a non-hot stripe area by the elimination sentinel. After each aggregation of the full stripe, the replica data processing strategy is determined according to the position of the identifier of the full stripe in the delayed elimination queue. If the full stripe belongs to a hot stripe, the replica data processing strategy is determined to be a non-deletion strategy, and the replica data is not deleted after the replica data aggregation is completed, so that the replica data of the hot stripe will not be deleted, and the check slice does not need to pull data from the data slice to restore the replica data or update the check data, thereby greatly reducing the network overhead generated by the hot stripe when restoring the replica data or updating the check data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the process of writing a replica for erasure coding;

[0041] Figure 2 This is a schematic diagram of the overwrite write process of the erasure code replica write solution;

[0042] Figure 3 Schematic diagram of the deletion process of the erasure code replica writing scheme;

[0043] Figure 4 This is a flow chart of an embodiment of a method for reducing network overhead of erasure code storage of the present application;

[0044] Figure 5 This is a schematic diagram of a first delayed elimination queue according to an embodiment of a method for reducing network overhead of erasure code storage of the present application;

[0045] Figure 6 For this application Figure 4 A detailed flow chart of step S10;

[0046] Figure 7 This is a schematic diagram of a second delayed elimination queue according to an embodiment of a method for reducing network overhead of erasure code storage of the present application;

[0047] Figure 8 This is a schematic diagram of a third delayed elimination queue of an embodiment of a method for reducing network overhead of erasure code storage of the present application;

[0048] Fig. 9 This is a schematic diagram of a fourth delayed elimination queue of an embodiment of a method for reducing network overhead of erasure code storage of the present application;

[0049] Fig.10 This is a functional module diagram of an embodiment of an apparatus for reducing network overhead of erasure code storage of the present application;

[0050] Fig.11 A schematic diagram of the hardware structure of an erasure code storage network overhead reduction device involved in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] First, some technical terms in the present application are explained to facilitate those skilled in the art to understand the present application.

[0053] Distributed storage system: a data storage system that stores data in multiple independent devices. Traditional network storage systems use centralized storage servers to store all data. Storage servers become the bottleneck of system performance and the focus of reliability and security, and cannot meet the needs of large-scale storage applications. Distributed network storage systems use an extensible system structure, use multiple storage servers to share storage loads, and use location servers to locate storage information. It not only improves system reliability, availability, and access efficiency, but is also easy to expand.

[0054] Erasure coded data storage: is a data protection method that divides the original data into k original data blocks according to the specified size, and then encodes it into p check data blocks according to the redundancy level, and finally stores the original data blocks and check data blocks in different locations to achieve the purpose of data protection.

[0055] Replica data storage: is a data protection method that treats the original data as a whole and stores n identical copies of the data in n different locations according to the redundancy level to achieve the purpose of data protection.

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0057] In a first aspect, an embodiment of the present application provides a method for reducing network overhead of erasure code storage.

[0058] In one embodiment, referring to Figure 4 , Figure 4 This is a flow chart of an embodiment of a method for reducing network overhead of erasure code storage in this application. Figure 4 As shown in FIG. 1 , the method for reducing the network overhead of erasure code storage includes:

[0059] Step S10, each time a write or delete operation is performed on a stripe, the delayed elimination queue is adjusted according to the identifier of the stripe of each write or delete operation, the delayed elimination queue is used to store the identifier of the stripe of the write or delete operation, the delayed elimination queue includes an elimination sentinel, and the elimination sentinel is used to indicate the identifier of a stripe in the delayed elimination queue, the delayed elimination queue has the first one to join the queue as the tail and the last one to join the queue as the head.

[0060] In this embodiment, the delayed elimination queue includes an elimination sentinel. The elimination sentinel is used to point to the identifier of a stripe in the delayed elimination queue and is used to divide the hotspot stripe area and the non-hotspot stripe area. The delayed elimination queue takes the last one to join the queue as the head and the first one to join the queue as the tail. Each time a write or delete operation is performed on the stripe, the delayed elimination queue is adjusted according to the identifier of the stripe for each write or delete operation. As a result, for stripes that frequently perform write or delete operations, their identifiers will be adjusted to the range from before the elimination sentinel to the head of the team as much as possible. For stripes that perform write or delete operations less frequently, their identifiers will be adjusted to the range from after the elimination sentinel and the elimination sentinel to the tail of the team as much as possible. It can be seen that the delayed elimination queue is divided into hotspot stripe areas and non-hotspot stripe areas by the elimination sentinel. The stripes in the range from before the elimination sentinel to the head of the team are hotspot strips, and the strips in the range from after the elimination sentinel and the elimination sentinel to the tail of the team are non-hotspot strips. Reference Figure 5 , Figure 5 This is a schematic diagram of the first delayed elimination queue of an embodiment of the erasure code storage network overhead reduction method of the present application, as shown in FIG. Figure 5 As shown, six stripe identifiers are added to the delayed elimination queue in the order of writing or deleting the stripes, among which stripe 0 is added first as the tail of the queue, stripe 5 is added last as the head of the queue, and stripe 2 is the elimination sentinel. Then stripe 2, stripe 1 and stripe 0 are non-hotspot strips, and stripe 5, stripe 4 and stripe 3 are hotspot strips.

[0061] Step S20, when each aggregation task is executed, for each full stripe, a replica data processing strategy is determined according to the position of the identifier of the full stripe in the delayed elimination queue.

[0062] In this embodiment, a background task, namely, an aggregation task, is run on the check shard. The aggregation task runs periodically and is responsible for detecting the data written to the stripe. When it is detected that a stripe is full, the replica data processing strategy is determined according to the position of its identifier in the delayed elimination queue. If the full stripe belongs to a hot stripe, the replica data processing strategy can be determined as a non-deletion strategy. If the full stripe belongs to a non-hot stripe, the replica data processing strategy can be determined as a deletion strategy.

[0063] Step S30, after aggregating the replica data of the fully written stripe, if the replica data processing policy is a deletion policy, the replica data is deleted; if the replica data processing policy is a non-deletion policy, the replica data is not deleted.

[0064] In this embodiment, after aggregating the replica data of the full stripe, if the full stripe belongs to a non-hotspot stripe, the replica data processing strategy is determined to be a deletion strategy, and the replica data is deleted accordingly. If the full stripe belongs to a hotspot stripe, the replica data processing strategy is determined to be a non-deletion strategy, and the replica data is not deleted accordingly. For non-hotspot stripes, write or delete operations are not frequently performed. After deleting the replica data, even if write or delete operations are occasionally performed, the network overhead of pulling the replica data is acceptable. For hotspot stripes, write or delete operations are frequently performed, and the replica data will not be deleted. The verification shard does not need to pull data from the data shard, thereby greatly reducing unnecessary network overhead.

[0065] In this embodiment, the delayed elimination queue includes an elimination sentinel, which is used to point to the identifier of a stripe in the delayed elimination queue and is used to divide the hot stripe area and the non-hot stripe area. The delayed elimination queue takes the last one to join the queue as the head and the first one to join the queue as the tail. Through the setting of the delayed elimination queue and the elimination sentinel, each time a write or delete operation is performed on the stripe, the delayed elimination queue is adjusted according to the identifier of the stripe of each write or delete operation, so that for stripes that frequently perform write or delete operations, their identifiers will be adjusted to the range from before the elimination sentinel to the head of the team as much as possible, and for stripes that seldom perform write or delete operations, their identifiers will be adjusted to the range from the elimination sentinel to the end of the team as much as possible. It can be seen that the delayed elimination queue is divided into hot stripe areas and non-hot stripe areas by the elimination sentinel. The strips located in the range from before the elimination sentinel to the head of the team are hot strips, and the strips located in the range from before the elimination sentinel to the head of the team are hot strips. The stripes in the range from the elimination of the sentinel to the end of the queue are non-hotspot strips. Each time an aggregation task is executed, when it is detected that a stripe is full, the replica data processing strategy is determined according to the position of its identifier in the delayed elimination queue. If the full stripe belongs to a hotspot stripe, the replica data processing strategy can be determined as a non-deletion strategy, and the replica data will not be deleted after the replica data aggregation is completed. If the full stripe belongs to a non-hotspot stripe, the replica data processing strategy can be determined as a deletion strategy, and the replica data will be deleted after the replica data aggregation is completed. Therefore, write or delete operations will not be frequently performed on non-hotspot strips. After deleting the replica data, even if write or delete operations are occasionally performed, the network overhead of pulling the replica data is acceptable. Write or delete operations will be frequently performed on hotspot strips, and the replica data will not be deleted. The verification shard does not need to pull data from the data shard, which can greatly reduce unnecessary network overhead.

[0066] Further, in one embodiment, referring to Figure 6 , Figure 6 For this application Figure 4 The detailed flow chart of step S10 is as follows: Figure 6 As shown, step S10 includes:

[0067] Step S101, each time a stripe is written or deleted, an identifier of the current stripe to be written or deleted is obtained;

[0068] Step S102, when the identifier of the current stripe is not in the delayed elimination queue, the identifier of the current stripe is added to the head of the delayed elimination queue;

[0069] Step S103, when the identifier of the current stripe is in the delayed elimination queue, detecting whether the position of the identifier of the current stripe in the delayed elimination queue is the head of the queue, and detecting whether the identifier of the current stripe is an elimination sentinel;

[0070] Step S104, if it is not the team head and is not an eliminated sentinel, move the identifier of the current strip to the team head;

[0071] Step S105, if it is not the team head and is an eliminated sentinel, then the next stripe identifier of the current stripe in the direction of the team tail is set as a new eliminated sentinel, and the current stripe identifier is moved to the team head;

[0072] Step S106, if it is the head of the team and is an eliminated sentinel, then the next stripe identifier of the current stripe toward the end of the team is set as a new eliminated sentinel;

[0073] Step S107: If it is the head of the queue and not an elimination sentinel, the delayed elimination queue is not adjusted.

[0074] In this embodiment, each time a stripe is written or deleted, the identifier of the current stripe is obtained. If the identifier of the current stripe is not in the delayed elimination queue, the identifier of the current stripe is directly added to the head of the delayed elimination queue. If the identifier of the current stripe is already in the delayed elimination queue, the identifier of the current stripe is moved to the head of the team. At this time, if the identifier of the current stripe is also an elimination sentinel, it is necessary to set the identifier of the next stripe in the direction of the tail of the current stripe to the new elimination sentinel, and move the identifier of the current stripe to the head of the team, and continue to refer to Figure 5 For example, the current stripe is stripe 2, and stripe 2 is also the sentinel to be eliminated. Figure 7 , Figure 7 This is a schematic diagram of a second delayed elimination queue in an embodiment of a method for reducing network overhead of erasure code storage of the present application, as shown in FIG. Figure 7 As shown, after stripe 2 moves to the head of the team, stripe 1 is set as the new elimination sentinel. If the identifier of the current stripe is before or after the elimination sentinel, only the identifier of the current stripe is moved to the head of the team, and the position of the elimination sentinel does not need to be changed. The schematic diagrams of the identifier of the current stripe before and after the elimination sentinel are respectively referred to Figure 8 and Fig. 9 , Figure 8This is a schematic diagram of the third delayed elimination queue of an embodiment of the erasure code storage network overhead reduction method of the present application, Fig. 9 This is a schematic diagram of the fourth delayed elimination queue of an embodiment of the method for reducing network overhead of erasure code storage in the present application. If the identifier of the current stripe is at the head of the team and is also an elimination sentinel, the identifier of the next stripe from the current stripe's identifier to the tail of the team is set as the new elimination sentinel. If the identifier of the current stripe is at the head of the team but is not an elimination sentinel, there is no need to adjust the delayed elimination queue.

[0075] Furthermore, in one embodiment, step S20 includes:

[0076] If the identifier of the full stripe is in the delayed elimination queue and the position of the identifier of the full stripe in the delayed elimination queue is in the first range, the replica data processing strategy is determined to be a non-deletion strategy, and the first range is the position range from before the sentinel is eliminated to the head of the queue;

[0077] If the identifier of the full stripe is in the delayed elimination queue and the position of the identifier of the full stripe in the delayed elimination queue is in the second range, or the identifier of the full stripe is not in the delayed elimination queue, then the copy data processing strategy is determined to be the deletion strategy, and the second range is the position range from the elimination sentinel and after the elimination sentinel to the end of the queue.

[0078] In this embodiment, the delayed elimination queue is divided into hotspot stripe areas and non-hotspot stripe areas by the elimination sentinel. If the full stripe is in the first range of the delayed elimination queue, that is, the position range from before the elimination sentinel to the head of the queue, the copy data processing strategy is determined to be a non-deletion strategy. If the full stripe is in the second range of the delayed elimination queue, that is, the position range from the elimination sentinel and after the elimination sentinel to the tail of the queue, the copy data processing strategy is determined to be a deletion strategy. Among them, if the identifier of the full stripe is not in the delayed elimination queue, it means that the stripe has never been added to the delayed elimination queue or has been added but has been squeezed out of the delayed elimination queue, and it also belongs to a non-hotspot stripe. Correspondingly, the copy data processing strategy is also determined to be a deletion strategy.

[0079] Furthermore, in one embodiment, step S20 further includes:

[0080] Check whether the current disk usage is less than the preset disk usage;

[0081] If the current disk usage is less than the preset disk usage, the replica data processing strategy is determined according to the position of the identifier of the full stripe in the delayed elimination queue.

[0082] In this embodiment, since not deleting the replica data will result in a certain amount of storage space being occupied, before determining the corresponding replica data processing strategy, it is possible to detect whether the current disk usage is less than the preset disk usage. If the current disk usage is less than the preset disk usage, it means that the current data storage space pressure is not large. The replica data processing strategy can be determined based on the position of the identifier of the full stripe in the delayed elimination queue.

[0083] Further, in one embodiment, after detecting whether the current disk usage is less than the preset disk usage, the following steps are performed:

[0084] If the current disk usage is greater than or equal to the preset disk usage, the copy data processing strategy is determined to be a deletion strategy.

[0085] In this embodiment, if the current disk usage is greater than or equal to the preset disk usage, it means that the current data storage space pressure is already large. In order to reduce the data storage space pressure, the copy data processing strategy is directly determined as the deletion strategy, and the copy data is deleted after the copy data is aggregated.

[0086] Further, in one embodiment, when the identifier of the current stripe is not in the delayed elimination queue, adding the identifier of the current stripe to the head of the delayed elimination queue includes:

[0087] When the identifier of the current stripe is not in the delayed elimination queue, detecting whether the number of stripe identifiers stored in the delayed elimination queue is greater than a preset number after the identifier of the current stripe is added to the head of the delayed elimination queue;

[0088] If yes, after deleting the stripe identifier stored at the tail of the delayed elimination queue, the identifier of the current stripe is added to the head of the delayed elimination queue;

[0089] If not, the identifier of the current strip is added to the head of the delayed elimination queue.

[0090] In this embodiment, since the delayed elimination queue will occupy a certain amount of data storage space, certain restrictions can be placed on the number of stripe identifiers stored in the delayed elimination queue, or certain restrictions can be placed on the size of the storage space of the delayed elimination queue. When the delayed elimination queue is not full, the identifier of the current stripe can be directly added to the head of the delayed elimination queue. When the delayed elimination queue is full, the stripe identifier stored at the tail of the delayed elimination queue can be squeezed out, and then the identifier of the current stripe can be added to the head of the delayed elimination queue.

[0091] Furthermore, in one embodiment, after step S30, the following steps are included:

[0092] At the end of each aggregation task execution, the identifiers of the strips located at the elimination sentinel and the range after the elimination sentinel to the end of the queue are deleted from the delayed elimination queue, and the head of the delayed elimination queue is set to the new elimination sentinel.

[0093] In this embodiment, since non-hotspot strips rarely perform write or delete operations, at the end of each aggregation task execution, the non-hotspot stripe identifier can be deleted from the delayed elimination queue. The non-hotspot stripe identifier is the stripe identifier located in the range from the elimination sentinel to the tail of the queue, and the head of the delayed elimination queue is set as the new elimination sentinel, and the process returns to execute step S10 to make new adjustments to the delayed elimination queue.

[0094] Furthermore, in one embodiment, the delayed elimination queue runs in memory.

[0095] In this embodiment, each time a stripe is written or deleted, and each time an aggregation task is executed, access to the delayed elimination queue is involved. Running the delayed elimination queue in memory can greatly improve the operating efficiency of the program.

[0096] In a second aspect, an embodiment of the present application also provides a device for reducing network overhead of erasure code storage.

[0097] In one embodiment, referring to Fig.10 , Fig.10 This is a functional module diagram of an embodiment of an apparatus for reducing network overhead of erasure code storage in the present application, as shown in FIG. Fig.10 As shown, the erasure code storage network overhead reduction device includes:

[0098] The adjustment module 10 is used to adjust the delayed elimination queue according to the identification of the stripe of each write or delete operation each time the stripe is written or deleted. The delayed elimination queue is used to store the identification of the stripe of the write or delete operation. The delayed elimination queue includes an elimination sentinel, and the elimination sentinel is used to indicate the identification of a stripe in the delayed elimination queue. The delayed elimination queue takes the first one to join the queue as the tail and the last one to join the queue as the head.

[0099] A determination module 20, configured to determine a replica data processing strategy for each full stripe according to the position of the identifier of the full stripe in the delayed elimination queue each time an aggregation task is executed;

[0100] The processing module 30 is used to delete the replica data after aggregating the replica data of the fully written stripe if the replica data processing policy is a deletion policy, and not delete the replica data if the replica data processing policy is a non-deletion policy.

[0101] Furthermore, in one embodiment, the adjustment module 10 is used to:

[0102] Each time a stripe is written or deleted, the identifier of the current stripe to be written or deleted is obtained;

[0103] When the identifier of the current stripe is not in the delayed elimination queue, the identifier of the current stripe is added to the head of the delayed elimination queue;

[0104] When the identifier of the current stripe is in the delayed elimination queue, it is detected whether the position of the identifier of the current stripe in the delayed elimination queue is the head of the queue, and whether the identifier of the current stripe is an elimination sentinel;

[0105] If it is not the head of the team and is not an eliminated sentinel, move the marker of the current strip to the head of the team;

[0106] If it is not the head of the team and is an elimination sentinel, the next stripe mark of the current stripe towards the end of the team is set as the new elimination sentinel, and the current stripe mark is moved to the head of the team;

[0107] If it is the head of the team and is an elimination sentinel, the next stripe mark of the current stripe towards the end of the team is set as the new elimination sentinel;

[0108] If it is the head of the queue and not an elimination sentinel, the delayed elimination queue will not be adjusted.

[0109] Furthermore, in one embodiment, the determination module 20 is used to:

[0110] If the identifier of the full stripe is in the delayed elimination queue and the position of the identifier of the full stripe in the delayed elimination queue is in the first range, the replica data processing strategy is determined to be a non-deletion strategy, and the first range is the position range from before the sentinel is eliminated to the head of the queue;

[0111] If the identifier of the full stripe is in the delayed elimination queue and the position of the identifier of the full stripe in the delayed elimination queue is in the second range, or the identifier of the full stripe is not in the delayed elimination queue, then the copy data processing strategy is determined to be the deletion strategy, and the second range is the position range from the elimination sentinel and after the elimination sentinel to the end of the queue.

[0112] Furthermore, in one embodiment, the determination module 20 is further configured to:

[0113] Check whether the current disk usage is less than the preset disk usage;

[0114] If the current disk usage is less than the preset disk usage, the replica data processing strategy is determined according to the position of the identifier of the full stripe in the delayed elimination queue.

[0115] Furthermore, in one embodiment, the apparatus for reducing erasure code storage network overhead further includes a deletion strategy determination module, which is used to:

[0116] If the current disk usage is greater than or equal to the preset disk usage, the copy data processing strategy is determined to be a deletion strategy.

[0117] Furthermore, in one embodiment, the apparatus for reducing erasure code storage network overhead further includes a adding module, which is used to:

[0118] When the identifier of the current stripe is not in the delayed elimination queue, detecting whether the number of stripe identifiers stored in the delayed elimination queue is greater than a preset number after the identifier of the current stripe is added to the head of the delayed elimination queue;

[0119] If yes, after deleting the stripe identifier stored at the tail of the delayed elimination queue, the identifier of the current stripe is added to the head of the delayed elimination queue;

[0120] If not, the identifier of the current strip is added to the head of the delayed elimination queue.

[0121] Furthermore, in one embodiment, the apparatus for reducing erasure code storage network overhead further includes a deletion adjustment module, which is used to:

[0122] At the end of each aggregation task execution, delete the identifiers of all stripes in the delayed elimination queue that are in the range from the elimination sentinel to the tail of the queue, and set the head of the delayed elimination queue to the new elimination sentinel.

[0123] Among them, the functional implementation of each module in the above-mentioned erasure code storage network overhead reduction device corresponds to the various steps in the above-mentioned erasure code storage network overhead reduction method embodiment, and its functions and implementation processes will not be repeated here one by one.

[0124] In a third aspect, an embodiment of the present application provides a device for reducing network overhead of erasure code storage.

[0125] Reference Fig.11 , Fig.11 Schematic diagram of the hardware structure of the erasure code storage network overhead reduction device involved in the embodiment of the present application. In the embodiment of the present application, the erasure code storage network overhead reduction device may include a processor, a memory, a communication interface and a communication bus.

[0126] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0127] The communication interface includes an input / output (I / O) interface, a physical interface, and a logical interface, etc., which are used to interconnect the devices inside the erasure code storage network overhead reduction device, and an interface used to interconnect the erasure code storage network overhead reduction device with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber interface, an ATM interface, etc.; the user device can be a display (Display), a keyboard (Keyboard), etc.

[0128] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0129] The processor may be a general-purpose processor, and the general-purpose processor may call the erasure code storage network overhead reduction program stored in the memory, and execute the erasure code storage network overhead reduction method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). Among them, the method executed when the erasure code storage network overhead reduction program is called can refer to the various embodiments of the erasure code storage network overhead reduction method of the present application, which will not be repeated here.

[0130] Those skilled in the art will understand that Fig.11 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0131] In a fourth aspect, an embodiment of the present application also provides a storage medium.

[0132] The storage medium of the present application stores a program for reducing the network overhead of erasure code storage, wherein when the program for reducing the network overhead of erasure code storage is executed by a processor, the steps of the method for reducing the network overhead of erasure code storage are implemented as described above.

[0133] Among them, the method implemented when the erasure code storage network overhead reduction program is executed can refer to the various embodiments of the erasure code storage network overhead reduction method of the present application, and will not be repeated here.

[0134] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0135] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0136] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0137] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0138] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0140] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for reducing network overhead of erasure code storage, characterized in that: The method for reducing the erasure code storage network overhead comprises: Each time a stripe is written or deleted, a delayed elimination queue is adjusted according to the stripe identifier of each write or delete operation, the delayed elimination queue is used to store the stripe identifier of the write or delete operation, the delayed elimination queue includes an elimination sentinel, the elimination sentinel is used to indicate the identifier of a stripe in the delayed elimination queue, the delayed elimination queue takes the first one to join the queue as the tail and takes the last one to join the queue as the head; Each time an aggregation task is executed, for each full stripe, the replica data processing strategy is determined according to the position of the full stripe identifier in the delayed elimination queue; After the replica data of the full stripe is aggregated, if the replica data processing policy is the deletion policy, the replica data is deleted; if the replica data processing policy is the non-deletion policy, the replica data is not deleted; The step of adjusting the delayed elimination queue according to the stripe identifier of each write or delete operation each time the stripe is written or deleted includes: Each time a stripe is written or deleted, the identifier of the current stripe to be written or deleted is obtained; When the identifier of the current stripe is not in the delayed elimination queue, the identifier of the current stripe is added to the head of the delayed elimination queue; When the identifier of the current stripe is in the delayed elimination queue, it is detected whether the position of the identifier of the current stripe in the delayed elimination queue is the head of the queue, and whether the identifier of the current stripe is an elimination sentinel; If it is not the head of the team and is not an eliminated sentinel, move the marker of the current strip to the head of the team; If it is not the head of the team and is an elimination sentinel, the next stripe mark of the current stripe towards the end of the team is set as the new elimination sentinel, and the current stripe mark is moved to the head of the team; If it is the head of the team and is an elimination sentinel, the next stripe mark of the current stripe towards the end of the team is set as the new elimination sentinel; If it is the head of the queue and not an elimination sentinel, the delayed elimination queue will not be adjusted; The step of determining the copy data processing strategy according to the position of the identifier of the full stripe in the delayed elimination queue includes: If the identifier of the full stripe is in the delayed elimination queue and the position of the identifier of the full stripe in the delayed elimination queue is in the first range, the replica data processing strategy is determined to be a non-deletion strategy, and the first range is the position range from before the sentinel is eliminated to the head of the queue; If the identifier of the full stripe is in the delayed elimination queue and the position of the identifier of the full stripe in the delayed elimination queue is in the second range, or the identifier of the full stripe is not in the delayed elimination queue, the replica data processing strategy is determined to be a deletion strategy, and the second range is the position range from the elimination sentinel and after the elimination sentinel to the end of the queue; When the identifier of the current stripe is not in the delayed elimination queue, adding the identifier of the current stripe to the head of the delayed elimination queue includes: When the identifier of the current stripe is not in the delayed elimination queue, detecting whether the number of stripe identifiers stored in the delayed elimination queue is greater than a preset number after the identifier of the current stripe is added to the head of the delayed elimination queue; If yes, after deleting the stripe identifier stored at the tail of the delayed elimination queue, the identifier of the current stripe is added to the head of the delayed elimination queue; If not, the identifier of the current strip is added to the head of the delayed elimination queue.

2. The method for reducing the network overhead of erasure code storage according to claim 1, characterized in that: The step of determining the copy data processing strategy according to the position of the identifier of the fully written stripe in the delayed elimination queue also includes: Check whether the current disk usage is less than the preset disk usage; If the current disk usage is less than the preset disk usage, the replica data processing strategy is determined according to the position of the identifier of the full stripe in the delayed elimination queue.

3. The method for reducing the network overhead of erasure code storage according to claim 2, characterized in that: After detecting whether the current disk usage is less than the preset disk usage, the method further includes: If the current disk usage is greater than or equal to the preset disk usage, the copy data processing strategy is determined to be a deletion strategy.

4. The method for reducing the network overhead of erasure code storage according to claim 1, characterized in that: After the replica data of the fully written stripe is aggregated, if the replica data processing policy is a deletion policy, the replica data is deleted; if the replica data processing policy is a non-deletion policy, the replica data is not deleted, and then the replica data is not deleted, including: At the end of each aggregation task execution, delete the identifiers of all stripes in the delayed elimination queue that are in the range from the elimination sentinel to the tail of the queue, and set the head of the delayed elimination queue to the new elimination sentinel.

5. A device for reducing network overhead of erasure code storage, characterized in that: The erasure code storage network overhead reduction device comprises: An adjustment module is used to adjust the delayed elimination queue according to the identification of the stripe of each write or delete operation each time the stripe is written or deleted, the delayed elimination queue is used to store the identification of the stripe of the write or delete operation, the delayed elimination queue includes an elimination sentinel, and the elimination sentinel is used to indicate the identification of a stripe in the delayed elimination queue, and the delayed elimination queue takes the first one to join the queue as the tail and the last one to join the queue as the head; A determination module, used for determining a replica data processing strategy for each full stripe according to the position of the identifier of the full stripe in the delayed elimination queue each time an aggregation task is executed; A processing module, configured to, after aggregating the replica data of the fully written stripe, delete the replica data if the replica data processing policy is a deletion policy, and not delete the replica data if the replica data processing policy is a non-deletion policy; The adjustment module is further used for: Each time a stripe is written or deleted, the identifier of the current stripe to be written or deleted is obtained; When the identifier of the current stripe is not in the delayed elimination queue, the identifier of the current stripe is added to the head of the delayed elimination queue; When the identifier of the current stripe is in the delayed elimination queue, it is detected whether the position of the identifier of the current stripe in the delayed elimination queue is the head of the queue, and whether the identifier of the current stripe is an elimination sentinel; If it is not the head of the team and is not an eliminated sentinel, move the marker of the current strip to the head of the team; If it is not the head of the team and is an elimination sentinel, the next stripe mark of the current stripe towards the end of the team is set as the new elimination sentinel, and the current stripe mark is moved to the head of the team; If it is the head of the team and is an elimination sentinel, the next stripe mark of the current stripe towards the end of the team is set as the new elimination sentinel; If it is the head of the queue and not an elimination sentinel, the delayed elimination queue will not be adjusted; The determining module is further used for: If the identifier of the full stripe is in the delayed elimination queue and the position of the identifier of the full stripe in the delayed elimination queue is in the first range, the replica data processing strategy is determined to be a non-deletion strategy, and the first range is the position range from before the sentinel is eliminated to the head of the queue; If the identifier of the full stripe is in the delayed elimination queue and the position of the identifier of the full stripe in the delayed elimination queue is in the second range, or the identifier of the full stripe is not in the delayed elimination queue, the replica data processing strategy is determined to be a deletion strategy, and the second range is the position range from the elimination sentinel and after the elimination sentinel to the end of the queue; When the identifier of the current strip is not in the delayed elimination queue, the identifier of the current strip is added to the head of the delayed elimination queue, and is also used for: When the identifier of the current stripe is not in the delayed elimination queue, detecting whether the number of stripe identifiers stored in the delayed elimination queue is greater than a preset number after the identifier of the current stripe is added to the head of the delayed elimination queue; If yes, after deleting the stripe identifier stored at the tail of the delayed elimination queue, the identifier of the current stripe is added to the head of the delayed elimination queue; If not, the identifier of the current strip is added to the head of the delayed elimination queue.

6. A device for reducing network overhead of erasure code storage, characterized in that: The erasure code storage network overhead reduction device includes a processor, a memory, and an erasure code storage network overhead reduction program stored in the memory and executable by the processor, wherein when the erasure code storage network overhead reduction program is executed by the processor, the steps of the erasure code storage network overhead reduction method according to any one of claims 1 to 4 are implemented.

7. A storage medium, characterized in that: The storage medium stores an erasure code storage network overhead reduction program, wherein when the erasure code storage network overhead reduction program is executed by the processor, the steps of the erasure code storage network overhead reduction method as described in any one of claims 1 to 4 are implemented.

Images