A data recycling method, system, device and computer readable storage medium

Abstract

Embodiments of the present application disclose a data recycling method, system, device and computer readable storage medium, the method comprising: in a current scheduling period, obtaining usage information of a storage unit in a storage unit cluster; the usage information comprising a recycling state and a historical recycling time; based on the recycling state, determining at least one first storage unit in the storage unit cluster that is not in a data recycling state; in the case that the recycling state represents a data recycling state, determining a storage unit corresponding to the recycling state as a second storage unit; based on the historical recycling time, determining a target storage unit from the at least one first storage unit; stopping a data processing service of the target storage unit, executing a data recycling operation through the target storage unit, and updating the recycling state of the target storage unit; stopping a data recycling operation of the second storage unit, and resuming a data processing service of the second storage unit. Through the present solution, the garbage recycling efficiency and the storage cluster performance can be improved.

Description

Technical Field

[0001] This invention relates to the field of infrastructure, and more particularly to a data recovery method, system, apparatus, and computer-readable storage medium. Background Technology

[0002] When users use products developed based on distributed storage (such as cloud storage or object storage), deletion is one of the most common operations. In the actual engineering implementation of deletion operations, the storage system generally does not immediately and completely erase the files that users want to delete from the disk, because too many deletion operations will affect the performance of the entire system. Therefore, the common approach of the storage system for user deletion operations is to mark the files that the user wants to delete as "deleted," making them invisible to the user, thus deleting the file from the user's perspective. Furthermore, the storage system records files that are marked as "deleted" but still exist on the storage medium. After a grace period, the storage system will delete these files at a predetermined time (such as in the early morning when access is low) or at a uniform time interval (such as every hour), thereby truly freeing up storage space and realizing garbage collection in the distributed storage system.

[0003] However, the aforementioned garbage collection process is usually carried out spontaneously by the storage logic unit, which is not convenient for unified progress management; and the garbage collection process consumes a lot of system resources. When the storage logic unit executes the garbage collection process, it can easily affect the data service performance of the entire storage cluster, thereby reducing the performance of the storage cluster and the efficiency of garbage collection. Summary of the Invention

[0004] This invention provides a data recycling method, system, apparatus, and computer-readable storage medium that can improve garbage collection efficiency and storage cluster performance.

[0005] The technical solution of this invention is implemented as follows:

[0006] This invention provides a data recycling method, the method comprising:

[0007] Within the current scheduling period, obtain the usage information of storage units in the storage unit cluster; the usage information includes the recycling status and historical recycling time.

[0008] Based on the recycling status, at least one first storage unit in the storage unit cluster that is not in a data recycling state is determined;

[0009] When the recycling state characterizes a data recycling state, the storage unit corresponding to the recycling state is determined as the second storage unit;

[0010] Based on the historical recycling time, a target storage unit is determined from the at least one first storage unit;

[0011] Stop the data processing service of the target storage unit, perform a data reclamation operation through the target storage unit, and update the reclamation status of the target storage unit;

[0012] Stop the data reclamation operation of the second storage unit and resume the data processing service of the second storage unit.

[0013] This invention provides a data recycling system, comprising: a control unit and a storage unit cluster; the control unit is interconnected with the storage units in the storage unit cluster via a preset interface; wherein,

[0014] The control unit is configured to: acquire usage information of storage units in the storage unit cluster within the current scheduling cycle; the usage information includes reclamation status and historical reclamation time; based on the reclamation status, determine at least one first storage unit in the storage unit cluster that is not in a data reclamation state; if the reclamation status indicates that the storage unit is in a data reclamation state, determine the storage unit corresponding to the reclamation status as a second storage unit; based on the historical reclamation time, determine a target storage unit from the at least one first storage unit; send a data reclamation instruction to the target storage unit through the preset interface and update the reclamation status of the target storage unit; send a stop reclamation instruction to the second storage unit through the preset interface and restore the data processing service of the second storage unit.

[0015] The storage unit is configured to, upon receiving the data recycling instruction, stop the data processing service and perform a data recycling operation; upon receiving the stop recycling instruction, stop the data recycling operation and start the data processing service accordingly.

[0016] This invention provides a data recycling device, which includes an acquisition unit, a determination unit, a data recycling unit, and a recovery unit; wherein,

[0017] The acquisition unit is used to acquire the usage information of storage units in the storage unit cluster within the current scheduling period; the usage information includes the recycling status and historical recycling time.

[0018] The determining unit is configured to, based on the reclamation state, determine at least one first storage unit in the storage unit cluster that is not in a data reclamation state; if the reclamation state indicates that the data is in a data reclamation state, determine the storage unit corresponding to the reclamation state as a second storage unit; and determine a target storage unit from the at least one first storage unit based on the historical reclamation time.

[0019] The data recycling unit is used to stop the data processing service of the target storage unit, perform data recycling operations through the target storage unit, and update the recycling status of the target storage unit.

[0020] The recovery unit is used to stop the data reclamation operation of the second storage unit and restore the data processing service of the second storage unit.

[0021] In the above-described apparatus, the determining unit is further configured to use a first storage unit with an empty historical recycling time as the target storage unit; the historical recycling time represents the time of the last data recycling operation; and, when the historical recycling time of at least one first storage unit is not empty, to determine the target storage unit from the at least one first storage unit based on at least one of the storage unit load, the amount of data to be recycled, and the storage unit usage.

[0022] In the above-described apparatus, the determining unit is further configured to perform a weighted summation of at least one of the storage unit load, the amount of data to be recycled, and the storage unit usage according to a preset weight, to obtain a weighted value for each of the at least one first storage units; and to determine the target storage unit from the at least one first storage unit based on the weighted value of each first storage unit.

[0023] In the above-described apparatus, the determining unit is further configured to: when the number of first storage units with the highest weighted value among the at least one first storage unit is greater than a preset number threshold, select the first storage unit with the highest weighted value as a candidate storage unit; and determine the target storage unit based on the historical recycling time of the candidate storage unit.

[0024] In the above-described apparatus, the determining unit is further configured to calculate the difference between the historical reclamation time and the current time of the candidate storage unit; and based on the difference, determine the target storage unit among the candidate storage units.

[0025] In the above-described device, the recovery unit is further configured to send a stop recycling command to the second storage unit through a first preset interface, so as to stop the data recycling operation of the second storage unit; update the recycling status of the at least one second storage unit to be in a non-data recycling state, and update the historical recycling time of the at least one second storage unit; and restore the data processing service of the second storage unit.

[0026] In the above-described apparatus, the determining unit is further configured to determine the storage unit as a first storage unit when the recycling state indicates that the data is not in a data recycling state, the working state indicates that the data is working normally, and the amount of data to be recycled is greater than a preset data amount threshold, thereby determining the at least one first storage unit.

[0027] In the above-described device, the data recycling unit is further configured to send a data recycling instruction to the target storage unit through a second preset interface to stop the data processing service of the target storage unit and perform a data recycling operation through the target storage unit; and to update the recycling status of the target storage unit to be in a data recycling state.

[0028] In the above-mentioned device, the data recycling unit is further configured to enter the next scheduling cycle after a preset waiting time, and to perform data recycling processing on the storage unit cluster through the next scheduling cycle, thereby realizing data recycling scheduling of the storage unit cluster through at least one scheduling cycle.

[0029] This invention provides a data recycling device, the data recycling device comprising:

[0030] Memory, used to store executable instructions;

[0031] When a processor executes executable instructions stored in the memory, the processor executes the data recycling method applied to the terminal when the executable instructions are executed.

[0032] This invention provides a storage medium storing executable instructions, which, when executed, cause a processor to perform a data reclamation method as described in this invention.

[0033] This invention provides a data reclamation method, system, apparatus, and computer-readable storage medium. The method includes: acquiring usage information of storage units in a storage unit cluster during a current scheduling period; the usage information includes reclamation status and historical reclamation time; determining at least one first storage unit in the storage unit cluster that is not in a data reclamation state based on the reclamation status; if the reclamation status indicates that the storage unit is in a data reclamation state, determining the storage unit corresponding to the reclamation status as a second storage unit; determining a target storage unit from the at least one first storage unit based on the historical reclamation time; stopping the data processing service of the target storage unit, performing a data reclamation operation through the target storage unit, and updating the reclamation status of the target storage unit; stopping the data reclamation operation of the second storage unit, and restoring the data processing service of the second storage unit. In the above scheme, during the current scheduling cycle, the target storage unit and the second storage unit are determined based on the recycling status. After stopping the data processing service of the target storage unit, the data recycling operation can be performed using all the resources of the target storage unit. Subsequently, the data service of the storage unit can be restored according to the recycling status, thereby realizing unified scheduling of data recycling operations of storage units, making full use of the system resources of the target storage unit, and improving garbage collection efficiency. In addition, it can reduce the impact of the target storage unit running garbage collection operations on the data service performance of the storage cluster, thereby also improving the performance of the storage cluster. Attached Figure Description

[0034] Figure 1 A flowchart illustrating a data recovery method provided in an embodiment of the present invention. Figure 1 ;

[0035] Figure 2 This invention provides a flowchart illustrating a data recovery method. Figure 2 ;

[0036] Figure 3 This invention provides a flowchart illustrating a data recovery method. Figure 3 ;

[0037] Figure 4 This invention provides a flowchart illustrating a data recovery method. Figure 4 ;

[0038] Figure 5 This invention provides a flowchart illustrating a data recovery method. Figure 5 ;

[0039] Figure 6 This is a schematic diagram of the structure of a data recovery device provided in an embodiment of the present invention;

[0040] Figure 7This is a schematic diagram of another data recovery device provided in an embodiment of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0042] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The method of the embodiments of the present invention can be executed by a data recycling device; in some embodiments, the data recycling device can be a terminal or a server. Figure 1 A flowchart illustrating a data recovery method provided in an embodiment of the present invention. Figure 1 , will combine Figure 1 The steps shown are explained.

[0043] S101. During the current scheduling period, obtain the usage information of the storage units in the storage unit cluster; the usage information includes the recycling status and historical recycling time.

[0044] In an embodiment of the present invention, within the current scheduling period, the usage information of the storage units in the storage unit cluster is obtained, including the recycling status and historical recycling time.

[0045] In an embodiment of the present invention, the control unit collects usage information of all storage units in the storage unit cluster during the current scheduling cycle. This usage information includes the garbage collection status and historical garbage collection time. The garbage collection status includes whether garbage collection is in progress or not, and the garbage in the storage unit, i.e., data operation information, can be determined from the garbage collection status. The historical garbage collection time includes the time when the storage unit last performed a garbage collection operation.

[0046] In embodiments of the present invention, a storage unit refers to a logical storage unit, which has the functions of storing data and reading / writing data. In some embodiments, S101 can be executed by a control unit connected to the storage unit interface. The control unit can control the storage unit to perform some operations through the interface. For example, sending a garbage collection command controls the storage unit to perform a garbage collection operation. Through the command and the collection of cluster status information, the control unit can statistically determine the garbage collection status of the entire cluster.

[0047] S102. Based on the recycling status, determine at least one first storage unit in the storage unit cluster that is not in the data recycling state.

[0048] In an embodiment of the present invention, at least one first storage unit that is not in a data recycling state is determined in the storage unit cluster based on the recycling status of the collected storage units.

[0049] In an embodiment of the present invention, based on the recycling status of the storage unit, it is determined whether the storage unit is currently undergoing a data recycling operation, and the storage unit that is not undergoing a data recycling operation is identified as the first storage unit.

[0050] S103. When the data recycling status is represented as being in a data recycling state, the storage unit corresponding to the recycling status is determined as the second storage unit.

[0051] In some embodiments of the present invention, the storage unit that is undergoing data reclamation operation is determined as the second storage unit based on the storage unit reclamation status.

[0052] S104. Based on historical reclamation time, determine the target storage unit from at least one first storage unit.

[0053] In an embodiment of the present invention, a target storage unit is determined from at least one storage unit by judging the historical recycling time of the storage unit.

[0054] In embodiments of the present invention, the target storage unit is a storage unit that needs to be reclaimed for data recovery. There may be one or more target storage units.

[0055] S105. Stop the data processing service of the target storage unit, perform data reclamation operation through the target storage unit, and update the reclamation status of the target storage unit.

[0056] In an embodiment of the present invention, after the target storage unit stops data processing services, a data reclamation operation is performed on the target storage unit, and the reclamation status of the target storage unit is updated.

[0057] In an embodiment of the present invention, the data processing service of the target storage unit is suspended, that is, the target storage unit is temporarily removed from the cluster, and then the data reclamation operation is performed on the removed target storage unit, and the reclamation status of the target storage unit is updated, that is, the target storage unit is performing a data reclamation operation.

[0058] S106. Stop the data reclamation operation of the second storage unit and restore the data processing service of the second storage unit.

[0059] In an embodiment of the present invention, the second storage unit stops the data recycling operation and resumes the data processing service of the second storage unit.

[0060] In an embodiment of the present invention, the second storage unit stops the data reclamation operation and is then rejoined to the storage unit cluster for data processing services.

[0061] It is understood that in the embodiments of the present invention, by collecting usage information of storage units within the cluster, and then determining the first and second storage units based on the recycling status of the storage units in the usage information, and determining the target storage unit in the first storage unit based on the historical recycling time, and finally stopping the data processing service of the target storage unit (i.e., removing it from the cluster), a data recycling operation is performed on the target storage unit. On the one hand, removing or adding storage units is a common operation in distributed storage systems and generally does not affect the performance of the cluster. On the other hand, because the storage unit has already been removed from the cluster when data (garbage) recycling is performed, there is no need to consider the performance impact of data recycling on this storage unit. Therefore, during this period, the storage unit can perform full data recycling, that is, occupy as many system resources as possible for data recycling, thus freeing up more storage space.

[0062] In some embodiments of the present invention, S104 can be implemented by S1041 to S1042, which will be described in detail in conjunction with the following steps.

[0063] S1041. The first storage unit with an empty historical reclamation time is selected as the target storage unit; the historical reclamation time represents the time when the last data reclamation operation was performed.

[0064] In some embodiments of the present invention, after determining the first storage unit, the historical reclamation time of the first storage unit is viewed. The historical reclamation time represents the time when the storage unit last performed a data reclamation operation. Therefore, the first storage unit is used as the target storage unit.

[0065] In some embodiments of the present invention, if the historical time of a first storage unit is empty, it indicates that the first storage unit has not undergone garbage collection.

[0066] S1042. If the historical reclamation time of at least one first storage unit is not empty, determine the target storage unit in at least one first storage unit based on at least one of the storage unit load, the amount of data to be reclaimed, and the storage unit usage.

[0067] In some embodiments of the present invention, if at least one of the first storage units has a non-empty historical recycling time, a target storage unit is determined in at least one first storage unit based on at least one of the storage unit load, the amount of data to be recycled, and the storage unit usage.

[0068] In some embodiments of the present invention, if only one first storage unit has an empty historical time, then the first storage unit is directly determined as the target storage unit; if two or more first storage units have an empty historical time, the target storage unit is determined from the two or more storage units based on at least one of the storage unit load, the amount of data to be reclaimed, and the storage unit usage.

[0069] In some embodiments of the present invention, the target storage unit can be determined from two or more storage units based on at least one of the current storage unit load percentage, the percentage of data to be recycled to the total data amount, and the percentage of storage unit usage to the total storage amount.

[0070] It is understood that, in some embodiments of the present invention, a first storage unit with an empty historical reclamation time is determined as the target storage unit. If none of the historical times are empty, the target storage unit is determined from at least one first storage unit based on at least one of the following: storage unit load, amount of data to be reclaimed, and storage unit usage. This allows storage units to occupy system resources in an orderly and reasonable manner for data reclamation.

[0071] In some embodiments of the present invention, such as Figure 2 As shown, Figure 2 A flowchart illustrating a data recovery method provided in an embodiment of the present invention. Figure 2 S1042 can be implemented through S201 to S202, which will be explained in detail in conjunction with the following steps.

[0072] S201. Based on preset weights, at least one of the following is weighted and summed: storage unit load, amount of data to be recycled, and storage unit usage, to obtain a weighted value for each of the at least one first storage units.

[0073] In some embodiments of the present invention, a weighted value for each of the at least one first storage unit is calculated based on at least one of the storage unit load, the amount of data to be recycled, and the storage unit usage, combined with a preset weight.

[0074] In some embodiments of the present invention, when there are two or more first storage units, the weight corresponding to the storage unit load can be set to 0.3, the weight corresponding to the amount of data to be reclaimed can be set to 0.5, and the weight corresponding to the storage unit usage can be set to 0.2. When calculating the weighted value of the first storage unit, only one of the storage unit load, the amount of data to be reclaimed, and the storage unit usage can be used, combined with its corresponding weight, that is, the weights of the other two factors can be temporarily set to 0, to calculate the weighted value of each of the first storage units. For example, the weight corresponding to the amount of data to be reclaimed can be set to 0.5, and the weights corresponding to the storage unit usage and the storage unit load can be set to 0 to calculate the weighted value of the two or more first storage units.

[0075] In some embodiments of the present invention, any two of the following factors—storage unit load, amount of data to be reclaimed, and storage unit usage—can be used, combined with their corresponding weights (i.e., the weights of the remaining factors are set to 0), to calculate the weighted value of each first storage unit. For example, the weight corresponding to the storage unit load can be set to 0.3, the weight corresponding to the amount of data to be reclaimed can be set to 0.5, and the weight corresponding to the storage unit usage can be set to 0 to calculate the weighted value of the storage unit.

[0076] In some embodiments of the present invention, the storage unit load, the amount of data to be recycled, and the storage unit usage can be used simultaneously, combined with corresponding weights, to calculate the weighted value of the storage unit.

[0077] In some embodiments of the present invention, the weighted value of the storage unit can be calculated based on at least one of the current storage unit load percentage, the percentage of data to be recycled to the total data amount, and the percentage of storage unit usage to the total storage amount, combined with corresponding weights.

[0078] S202. Determine the target storage cell from at least one first storage cell based on the weighted value of each first storage cell.

[0079] In some embodiments of the present invention, a target storage cell is determined from at least one first storage cell based on a weighted value calculated for each first storage cell.

[0080] In some embodiments of the present invention, after calculating the weighted value of each first storage unit, the first storage unit with the highest weighted value is selected by sorting and determined as the target storage unit.

[0081] In some embodiments of the present invention, a threshold can be set, and weighted values ​​with a weight value greater than the threshold can be used as candidate values. The candidate values ​​are sorted, the highest weighted value is selected, and the first storage unit corresponding to the highest weighted value is determined as the target storage unit.

[0082] It is understood that, in some embodiments of the present invention, at least one of the following—the storage unit load, the amount of data to be reclaimed, and the storage unit usage—is weighted and summed according to a preset weight to obtain the weighted value of each storage unit in the first storage unit. Based on the weighted value, the final target storage unit is determined, which enables the storage unit to further occupy system resources in an orderly and reasonable manner for data reclamation.

[0083] In some embodiments of the present invention, such as Figure 3 As shown, Figure 3 A flowchart illustrating a data recovery method provided in an embodiment of the present invention. Figure 3 S201 can be implemented through S2011 to S2012, and will be explained in detail in conjunction with the following steps.

[0084] S2011. If the number of first storage units with the highest weighted value in at least one first storage unit is greater than a preset number threshold, the first storage unit with the highest weighted value shall be selected as a candidate storage unit.

[0085] In some embodiments of the present invention, if the number of first storage units with the highest weighted value in at least one first storage unit is greater than a preset number threshold, that is, if multiple first storage units have the same and highest weighted value at the same time, all of these multiple first storage units are regarded as candidate storage units.

[0086] In some embodiments, the preset quantity threshold can be 1, or it can be set according to actual needs. The specific selection depends on the actual situation, and the embodiments of the present invention do not limit it.

[0087] S2012. Determine the target storage unit based on the historical reclamation time of the candidate storage units.

[0088] In some embodiments of the present invention, after obtaining multiple candidate storage units, the target storage unit is determined based on the historical recycling time of the multiple candidate storage units.

[0089] In some embodiments of the present invention, such as Figure 4 As shown, Figure 4 A flowchart illustrating a data recovery method provided in an embodiment of the present invention. Figure 4 S2012 can be implemented through S301 to S302, which will be explained in detail in conjunction with the following steps.

[0090] S301. Calculate the difference between the historical reclamation time and the current time of the candidate memory cell.

[0091] In some embodiments of the present invention, the historical reclamation time of the candidate storage unit is obtained, that is, the time when the candidate storage unit last performed a data reclamation operation, and the difference is calculated using the time when the candidate storage unit last performed a data reclamation operation and the current scheduling time.

[0092] In some embodiments of the present invention, each candidate storage unit corresponds to a difference.

[0093] S302. Based on the difference, determine the target memory cell from the candidate memory cells.

[0094] In some embodiments of the present invention, after calculating the difference corresponding to each candidate storage unit, the differences are sorted, the maximum difference is found, and the candidate storage unit corresponding to the maximum difference is determined as the target storage unit.

[0095] In some embodiments of the present invention, one or more candidate storage units with a difference greater than a preset difference threshold can be determined as target storage units. The number of target storage units can be set according to the actual situation of the storage cluster, such that removing a corresponding number of target storage units will not seriously affect the data service performance of the storage cluster. The specific selection is based on the actual situation, and the embodiments of the present invention do not impose limitations.

[0096] It is understood that, in some embodiments of the present invention, when the weighted values ​​of multiple candidate storage units are the highest and equal, the target storage unit is determined by calculating the difference between the historical reclamation time and the current time of the candidate storage units. This can accurately locate the target storage unit that needs to be reclaimed, thereby improving the reclamation efficiency.

[0097] In some embodiments of the present invention, S106 can be implemented by S1061 to S1063, which will be described in detail in conjunction with the following steps.

[0098] S1061. Send a stop recycling command to the second storage unit through the first preset interface to make the second storage unit stop the data recycling operation.

[0099] In some embodiments of the present invention, the control unit sends a stop recycling command to the second storage unit through a first preset interface, causing the second storage unit to stop the data recycling operation.

[0100] In some embodiments of the present invention, the second storage unit is a storage unit undergoing data reclamation operation.

[0101] S1062. Update the reclamation status of at least one second storage unit to "not in data reclamation state" and update the historical reclamation time of at least one second storage unit.

[0102] In some embodiments of the present invention, the reclamation status of at least one second storage unit is updated to be in a non-data reclamation state, and the historical reclamation time of at least one second storage unit is updated.

[0103] S1063. Restore the data processing service of the second storage unit.

[0104] In some embodiments of the present invention, restoring the data processing service of the second storage unit means rejoining the second storage unit, whose data recycling status has been changed, into the cluster.

[0105] In some embodiments of the present invention, the order of S1062 and S1063 is not limited.

[0106] In some embodiments of the present invention, S102 can be implemented by S1021, as will be explained in detail below.

[0107] S1021. When the recycling status indicates that the data is not in the recycling state, the working status indicates that the data is working normally, and the amount of data to be recycled is greater than the preset data amount threshold, the storage unit is determined as the first storage unit, thereby determining at least one first storage unit; the usage information also includes the working status.

[0108] In some embodiments of the present invention, based on the working status contained in the usage information, storage units that are currently working normally and require data reclamation are identified. Then, the reclamation status of the storage units is checked, and storage units that are not in the data reclamation state are identified. The identified storage units are named as first storage units, thereby identifying at least one first storage unit.

[0109] It is understood that, in some embodiments of the present invention, storage units that are normal and require garbage collection are identified from the storage unit cluster for data reclamation operations, so that system resources can be effectively utilized.

[0110] In some embodiments of the present invention, S105 can be implemented by S1051 to S1052, which will be described in detail in conjunction with the following steps.

[0111] S1051. Send a data recycling instruction to the target storage unit through the second preset interface to stop the data processing service of the target storage unit, and perform a data recycling operation through the target storage unit.

[0112] In some embodiments of the present invention, the control unit sends a data reclamation command to the target storage unit through a preset second interface, causing the target storage unit to stop data processing services and perform data reclamation operations through the target storage unit.

[0113] In some embodiments of the present invention, the target storage unit is temporarily removed from the storage unit cluster by stopping data processing services.

[0114] In some embodiments of the present invention, the first preset interface and the second preset interface may be the same or different.

[0115] S1052. Update the reclamation status of the target storage unit to be in data reclamation status.

[0116] In some embodiments of the present invention, the reclamation status of the target storage unit is updated to be in a data reclamation state, that is, it indicates that the target storage unit is performing a data reclamation operation.

[0117] It is understood that, in some embodiments of the present invention, a data reclamation command is sent to the target storage unit via an interface to stop the data processing service of the target storage unit, and the data reclamation operation is performed by the target storage unit. Removing the target storage unit from the cluster before performing the data reclamation operation allows the target storage unit to utilize its full resources for data reclamation, improving garbage collection efficiency; furthermore, it reduces the impact of the target storage unit's garbage collection operation on the data service performance of the storage cluster, thereby also improving the performance of the storage cluster.

[0118] In some embodiments of the present invention, S401 is also included, as follows.

[0119] S401. Enter the next scheduling cycle after a preset waiting time. In the next scheduling cycle, perform data reclamation processing on the storage unit cluster. Thus, data reclamation scheduling of the storage unit cluster is achieved through at least one scheduling cycle.

[0120] In some embodiments of the present invention, after a preset waiting time, the next scheduling cycle is entered. In the next scheduling cycle, garbage collection is performed on the storage unit cluster again, thereby realizing the data recycling scheduling of the storage unit cluster through at least one scheduling cycle.

[0121] In some embodiments of the present invention, the process begins by entering the next scheduling cycle after a preset waiting time. During the next scheduling cycle, data reclamation is performed on the storage unit cluster, thus achieving data reclamation scheduling of the storage unit cluster through at least one scheduling cycle. It can be seen that the algorithm for determining the target storage unit in this invention is essentially a round-robin algorithm, ensuring that each storage unit completes at least one garbage collection after a certain period. Furthermore, garbage collection scheduling is not affected by time. Generally, the early morning hours are a low point for business access, so the control unit can appropriately increase the number of storage units undergoing garbage collection between 00:00 and 02:00 during scheduling.

[0122] It is understood that, in the embodiments of the present invention, a preset waiting time is used to enter the next scheduling cycle. During the next scheduling cycle, data reclamation processing is performed on the storage unit cluster, thereby achieving data reclamation scheduling of the storage unit cluster through at least one scheduling cycle. This ensures that each storage unit can complete at least one garbage collection after a certain period of time.

[0123] This invention provides a data recycling method applicable to real-world scenarios of garbage collection in storage clusters. The storage cluster can contain multiple storage units, each connected to a control unit via an interface. The method is as follows: Figure 5 As shown.

[0124] S1. The control unit collects information from all storage units.

[0125] S2. Iterate through the information of each storage unit in sequence.

[0126] In S1-S2, the control unit first collects the usage information of all storage units in the entire cluster. This information includes: whether the storage unit is working normally, whether the storage unit needs to be garbage collected, the time when the storage unit last performed a garbage collection operation (empty if there has never been one), and whether the storage unit is currently in a garbage collection state.

[0127] S3. Determine whether the storage unit is in garbage collection state. If yes, execute S8; otherwise, execute S4.

[0128] S4. Determine if the time of the last recycling operation of the storage unit is empty. If yes, proceed to S6; otherwise, proceed to S5.

[0129] In S2-S4, the control unit analyzes the data and selects a storage unit from those that are operating normally and require garbage collection. It temporarily removes the storage unit from the data cluster and updates its status to "garbage collection in progress." Simultaneously, it controls this storage unit to perform a full garbage collection operation. Specifically, if the last garbage collection time for a storage unit is empty, it means that this storage unit has never performed garbage collection, and this storage unit is selected as the storage unit requiring garbage collection, i.e., the first storage unit.

[0130] S5. Determine if the difference between the last garbage collection time of the storage unit and the current time is the largest. If yes, proceed to S6; otherwise, proceed to S3.

[0131] In S5, if all storage units have undergone at least one garbage collection operation, the following weighted values ​​are calculated: the current load percentage of the storage unit and its corresponding weight (e.g., 0.3); the percentage of objects awaiting garbage collection relative to the total number of objects and its corresponding weight (e.g., 0.5); and the percentage of current storage unit usage relative to the total storage capacity and its corresponding weight (e.g., 0.2). The storage unit with the highest weighted value is selected as the target storage unit. If there are multiple units with the same highest weighted value, the storage unit with the largest difference between the time of the last garbage collection operation and the current time is selected as the target storage unit.

[0132] S6. Remove the storage unit from the cluster.

[0133] In S6, the control unit temporarily removes the target storage unit from the data cluster and updates its status information to indicate that garbage collection is in progress.

[0134] S7. The storage unit begins full garbage collection, and then S10 is executed.

[0135] In S6, storage units can utilize all available resources for garbage collection.

[0136] S8, the storage unit stops garbage collection.

[0137] S9. Re-add the storage unit to the cluster, and then execute S10.

[0138] In S9, the control unit analyzes the information about whether the storage unit is in a garbage collection state and finds the storage unit that performed the garbage collection operation in the previous round, designating it as the second storage unit. The control unit then controls the second storage unit to stop the garbage collection operation, re-adds it to the data cluster, updates the information about the time of its last garbage collection operation, cancels its garbage collection state, and resumes its use.

[0139] S10, the time to wait for one scheduling cycle.

[0140] In S10, the control unit waits for one scheduling cycle before returning to S1 to start again.

[0141] It is understood that, in the embodiments of the present invention, by collecting usage information of all storage units within the cluster, and then determining the storage units that need to undergo garbage collection based on the usage information, these storage units are temporarily removed from the cluster, and garbage collection is performed after they are removed. When a storage unit is selected to begin garbage collection and the process ends, that unit is removed from the cluster. At this time, any operation performed on this storage unit will not affect the entire cluster, and garbage collection efficiency is improved.

[0142] This invention provides a data recycling system, comprising: a control unit and a storage unit cluster; the control unit is interconnected with storage units in the storage unit cluster via a preset first interface and a preset second interface; wherein,

[0143] The control unit is configured to: acquire usage information of storage units in the storage unit cluster within the current scheduling cycle; the usage information includes reclamation status and historical reclamation time; based on the reclamation status, determine at least one first storage unit in the storage unit cluster that is not in a data reclamation state; if the reclamation status indicates that the storage unit is in a data reclamation state, determine the storage unit corresponding to the reclamation status as a second storage unit; based on the historical reclamation time, determine a target storage unit from the at least one first storage unit; send a data reclamation instruction to the target storage unit through the preset interface and update the reclamation status of the target storage unit; send a stop reclamation instruction to the second storage unit through the preset interface and restore the data processing service of the second storage unit.

[0144] The storage unit is configured to, upon receiving the data recycling instruction, stop the data processing service and perform a data recycling operation; upon receiving the stop recycling instruction, stop the data recycling operation and start the data processing service accordingly.

[0145] This invention provides a data recycling device, such as... Figure 6 The above, Figure 6 This is a schematic diagram of a data recovery device provided in an embodiment of the present invention. The data recovery device includes: an acquisition unit 601, a determination unit 602, a data recovery unit 603, and a recovery unit 604; wherein,

[0146] The acquisition unit 601 is used to acquire the usage information of the storage units in the storage unit cluster within the current scheduling period; the usage information includes the recycling status and historical recycling time.

[0147] The determining unit 602 is used to determine, based on the recycling status, at least one first storage unit in the storage unit cluster that is not in a data recycling state.

[0148] The determining unit 602 is further configured to, when the recycling state characterization is in a data recycling state, determine the storage unit corresponding to the recycling state as a second storage unit.

[0149] The determining unit 602 is further configured to determine a target storage unit from the at least one first storage unit based on the historical recycling time.

[0150] The data recycling unit 603 is used to stop the data processing service of the target storage unit, perform data recycling operations through the target storage unit, and update the recycling status of the target storage unit.

[0151] The recovery unit 604 is used to stop the data reclamation operation of the second storage unit and restore the data processing service of the second storage unit.

[0152] The determining unit 602 is further configured to select a first storage unit with an empty historical reclamation time as the target storage unit; the historical reclamation time represents the time of the last data reclamation operation; and, if the historical reclamation time of at least one first storage unit is not empty, determine the target storage unit from the at least one first storage unit based on at least one of the storage unit load, the amount of data to be reclaimed, and the storage unit usage. The usage information further includes: storage unit load, amount of data to be reclaimed, and storage unit usage.

[0153] The determining unit 602 is further configured to perform a weighted summation of at least one of the storage unit load, the amount of data to be recycled, and the storage unit usage according to a preset weight, to obtain a weighted value for each of the at least one first storage units; and to determine the target storage unit from the at least one first storage unit according to the weighted value of each first storage unit.

[0154] The determining unit 602 is further configured to: when the number of first storage units with the highest weighted value in the at least one first storage unit is greater than a preset number threshold, select the first storage unit with the highest weighted value as a candidate storage unit; and determine the target storage unit based on the historical recycling time of the candidate storage unit.

[0155] The determining unit 602 is further configured to calculate the difference between the historical reclamation time and the current time of the candidate storage units; and to determine the target storage unit among the candidate storage units based on the difference.

[0156] The recovery unit 604 is configured to send a stop recycling command to the second storage unit through a first preset interface to stop the data recycling operation of the second storage unit; update the recycling status of the at least one second storage unit to be in a non-data recycling state and update the historical recycling time of the at least one second storage unit; and restore the data processing service of the second storage unit.

[0157] The determining unit 602 is further configured to determine the storage unit as a first storage unit when the recycling state indicates that the data is not in a data recycling state, the working state indicates that the data is working normally, and the amount of data to be recycled is greater than a preset data amount threshold, thereby determining the at least one first storage unit.

[0158] The data recycling unit 603 is further configured to send a data recycling instruction to the target storage unit through a second preset interface to stop the data processing service of the target storage unit and perform a data recycling operation through the target storage unit; and update the recycling status of the target storage unit to be in a data recycling state.

[0159] The data recycling unit 603 is also used to enter the next scheduling cycle after a preset waiting time, and to perform data recycling processing on the storage unit cluster through the next scheduling cycle, thereby realizing data recycling scheduling of the storage unit cluster through at least one scheduling cycle.

[0160] Understandably, in the above implementation scheme, by collecting usage information of all storage units within the cluster, and then determining the storage units that require garbage collection based on this information, these units are temporarily removed from the cluster, and garbage collection is performed after removal. On one hand, removing or adding storage units is a common operation in distributed storage systems and generally does not affect cluster performance. On the other hand, because the storage unit has already been removed from the cluster during garbage collection, there is no need to consider the performance impact of garbage collection on this storage unit. Therefore, during this period, the storage unit can undergo full garbage collection, that is, occupy as many system resources as possible for garbage collection, thus freeing up more storage space.

[0161] Based on the methods described in the above embodiments, an embodiment of the present invention provides a structural schematic diagram, as follows: Figure 7 As shown, Figure 7 This application provides a schematic diagram of the structure of a data recycling device, including: a processor 701 and a memory 702; the memory 702 stores one or more programs executable by the processor 701, and when one or more programs are executed, the processor 701 executes a data recycling method corresponding to the previous embodiment.

[0162] This invention provides a computer-readable storage medium storing executable instructions for inducing a processor to execute the data recycling method.

[0163] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0164] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A data retrieval method, characterized in that, include: Within the current scheduling period, obtain the usage information of storage units in the storage unit cluster; the usage information includes recycling status, historical recycling time, storage unit load, amount of data to be recycled, and storage unit usage. Based on the recycling status, at least one first storage unit in the storage unit cluster that is not in a data recycling state is determined; When the recycling state characterizes a data recycling state, the storage unit corresponding to the recycling state is determined as the second storage unit; Based on the historical recycling time, a target storage unit is determined from the at least one first storage unit; Stop the data processing service of the target storage unit, perform a data reclamation operation through the target storage unit, and update the reclamation status of the target storage unit; Stop the data reclamation operation of the second storage unit and restore the data processing service of the second storage unit; The step of determining the target storage unit from the at least one first storage unit based on the historical reclamation time includes: The first storage unit with an empty historical reclamation time is selected as the target storage unit; the historical reclamation time represents the time of the last data reclamation operation. If the historical reclamation time of at least one first storage unit is not empty, according to a preset weight, at least one of the storage unit load, the amount of data to be reclaimed and the storage unit usage is weighted and summed to obtain the weighted value of each first storage unit in the at least one first storage unit. If the number of first storage units with the highest weighted value in the at least one first storage unit is greater than a preset number threshold, the first storage unit with the highest weighted value shall be selected as a candidate storage unit. Calculate the difference between the historical reclamation time and the current time of the candidate memory cell; Based on the difference, the target storage cell is determined from the candidate storage cells.

2. The method according to claim 1, characterized in that, The step of stopping the data reclamation operation of the second storage unit and restoring the data processing service of the second storage unit includes: A stop reclamation command is sent to the second storage unit through the first preset interface, so that the second storage unit stops the data reclamation operation; Update the reclamation status of the at least one second storage unit to "not in data reclamation state" and update the historical reclamation time of the at least one second storage unit; Restore the data processing service of the second storage unit.

3. The method according to claim 1, characterized in that, The usage information also includes the working status, and the step of determining at least one first storage unit in the storage unit cluster that is not in a data reclamation state based on the reclamation status includes: When the recycling state indicates that the data is not in a recycling state, the working state indicates that the data is working normally, and the amount of data to be recycled is greater than a preset data amount threshold, the storage unit is determined as the first storage unit, thereby determining the at least one first storage unit.

4. The method according to claim 1, characterized in that, The step of stopping the data processing service of the target storage unit, performing a data reclamation operation through the target storage unit, and updating the reclamation status of the target storage unit includes: A data recycling instruction is sent to the target storage unit through the second preset interface to stop the data processing service of the target storage unit and to perform a data recycling operation through the target storage unit. Update the recycling status of the target storage unit to be in data recycling status.

5. The method according to claim 1, characterized in that, The method further includes: The next scheduling cycle begins after a preset waiting time. During the next scheduling cycle, data reclamation is performed on the storage unit cluster. Thus, data reclamation scheduling of the storage unit cluster is achieved through at least one scheduling cycle.

6. A data retrieval system, characterized in that, include: A control unit and a storage unit cluster; the control unit is interconnected with the storage units in the storage unit cluster via a preset interface; wherein... The control unit is configured to acquire usage information of storage units in the storage unit cluster within the current scheduling cycle; the usage information includes recycling status, historical recycling time, storage unit load, amount of data to be recycled, and storage unit usage; based on the recycling status, determine at least one first storage unit in the storage unit cluster that is not in a data recycling state; if the recycling status indicates that it is in a data recycling state, determine the storage unit corresponding to the recycling status as a second storage unit; based on the historical recycling time, determine a target storage unit from the at least one first storage unit; send a data recycling instruction to the target storage unit through the preset interface and update the recycling status of the target storage unit; send a stop recycling instruction to the second storage unit through the preset interface and restore the data processing service of the second storage unit; The storage unit is configured to, upon receiving the data recycling instruction, stop the data processing service and perform a data recycling operation; upon receiving the stop recycling instruction, stop the data recycling operation and start the data processing service accordingly. The control unit is further configured to: select a first storage unit with an empty historical recycling time as the target storage unit; the historical recycling time represents the time of the last data recycling operation; when the historical recycling time of at least one first storage unit is not empty, perform a weighted summation of at least one of the storage unit load, the amount of data to be recycled, and the storage unit usage according to a preset weight, to obtain a weighted value for each of the at least one first storage units; when the number of first storage units with the highest weighted value in the at least one first storage unit is greater than a preset number threshold, select the first storage unit with the highest weighted value as a candidate storage unit; calculate the difference between the historical recycling time and the current time of the candidate storage units; and determine the target storage unit from the candidate storage units based on the difference.

7. A data retrieval device, characterized in that, It includes an acquisition unit, a determination unit, a data retrieval unit, and a recovery unit; among which, The acquisition unit is used to acquire the usage information of storage units in the storage unit cluster within the current scheduling period; the usage information includes recycling status, historical recycling time, storage unit load, amount of data to be recycled, and storage unit usage. The determining unit is configured to, based on the reclamation state, determine at least one first storage unit in the storage unit cluster that is not in a data reclamation state; if the reclamation state indicates that the data is in a data reclamation state, determine the storage unit corresponding to the reclamation state as a second storage unit; and determine a target storage unit from the at least one first storage unit based on the historical reclamation time. The data recycling unit is used to stop the data processing service of the target storage unit, perform data recycling operations through the target storage unit, and update the recycling status of the target storage unit. The recovery unit is used to stop the data reclamation operation of the second storage unit and restore the data processing service of the second storage unit; The determining unit is further configured to: select a first storage unit with an empty historical recycling time as the target storage unit; the historical recycling time represents the time of the last data recycling operation; when the historical recycling time of at least one first storage unit is not empty, perform a weighted summation of at least one of the storage unit load, the amount of data to be recycled, and the storage unit usage according to a preset weight, to obtain a weighted value for each of the at least one first storage units; when the number of first storage units with the highest weighted value in the at least one first storage unit is greater than a preset number threshold, select the first storage unit with the highest weighted value as a candidate storage unit; calculate the difference between the historical recycling time and the current time of the candidate storage units; and determine the target storage unit from the candidate storage units based on the difference.

8. A data retrieval device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, It stores executable instructions for causing a processor to execute, thereby implementing the method of any one of claims 1 to 5.

Images