Garbage collection method, device, storage medium and system

Abstract

The application provides a garbage collection method, device, storage medium and system. The method comprises the following steps: determining the first garbage collection efficiency of each user storage service using a distributed storage system at a first time according to a set garbage collection time interval; predicting the second garbage collection efficiency of each user storage service at a second time; and determining the target user storage service that needs to be subjected to garbage collection at the first time according to the change degree of the second garbage collection efficiency of each user storage service compared with the first garbage collection efficiency of each user storage service, the first garbage collection efficiency of each user storage service and the remaining storage space capacity of the distributed storage system. Through the garbage collection strategy, the storage space can be recycled in time, the water level of the remaining storage space is ensured to be safe, and the influence of garbage collection on data read-write service is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, and particularly relates to a garbage collection method, device, storage medium and system. BACKGROUND

[0002] Currently, more and more distributed storage systems are constructed based on Log-Structured read-write flow. Log-Structured is also called Append-only Sequence of Data sometimes, because all write operations will keep adding to the data structure without updating the original value.

[0003] The data writing process of a user storage service (such as a cloud disk) to a distributed storage system is simply that: the user storage service triggers a data writing request carrying data to be written and a logical block address (LBA) corresponding to the data, records the data to be written in a log file in the distributed storage system, and records the mapping relationship between the LBA and the data position of the data to be written in the log file. The reading process is: assuming that the user storage service triggers a data reading request carrying the LBA, the distributed storage system reads the corresponding log file according to the mapping relationship to return the corresponding data.

[0004] Because the data recorded in each log file is finally to be written to each storage node (physical disk) included in the distributed storage system, and a large amount of useless data will be generated in the data read-write process of many user storage services, the useless data will occupy a large amount of storage space, therefore, the distributed storage system will clear the useless data from the log file through a background garbage collection process to finally recover the storage space in the storage node. The garbage collection process is simply that: the valid data in the original log file is read and written into a new log file, the mapping relationship between the LBA and the data position in the log file is updated, and the original log file is deleted to recover the storage space.

[0005] A conventional garbage collection strategy is to preferentially collect user storage services with more useless data. Although the user storage services with more useless data consume less network, CPU and other resources, the user storage services with less useless data cannot be collected, and when a large amount of data is written into the distributed storage system and the remaining storage space is insufficient, the user storage services with less useless data have to be collected to free up more storage space. However, at this time, the user storage services with less useless data will consume more CPU, network, disk and other resources, and will affect the foreground data read and write service. SUMMARY

[0006] Embodiments of the present application provide a garbage collection method, device, storage medium and system, which can collect storage space in time, ensure the water level safety of the remaining storage space, and reduce the impact of garbage collection on data read and write service.

[0007] In a first aspect, embodiments of the present application provide a garbage collection method, which comprises:

[0008] According to a set garbage collection time interval, determining a first garbage collection efficiency of each of a plurality of user storage services using a distributed storage system at a first time;

[0009] Predicting a second garbage collection efficiency of each of the plurality of user storage services at a second time, the second time being later than the first time;

[0010] According to the change degree of the second garbage collection efficiency of each of the plurality of user storage services compared with the first garbage collection efficiency of each of the plurality of user storage services, the first garbage collection efficiency of each of the plurality of user storage services and the remaining capacity of the storage space of the distributed storage system, determining a target user storage service that needs to be collected at the first time.

[0011] In a second aspect, embodiments of the present application provide a garbage collection device, which comprises:

[0012] A determination module is configured to determine a first garbage collection efficiency of each of a plurality of user storage services using a distributed storage system at a first time according to a set garbage collection time interval;

[0013] A prediction module is configured to predict a second garbage collection efficiency of each of the plurality of user storage services at a second time, the second time being later than the first time;

[0014] The scheduling module is configured to determine a target user storage service that needs to perform garbage collection at the first time according to a change degree of the second garbage collection efficiency of each of the user storage services compared with the first garbage collection efficiency of each of the user storage services, the first garbage collection efficiency of each of the user storage services, and a remaining storage space capacity of the distributed storage system.

[0015] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a communication interface. The memory stores executable code. When the executable code is executed by the processor, the processor performs the garbage collection method according to the first aspect.

[0016] In a fourth aspect, a non-transitory machine readable storage medium is provided, which stores executable code. When the executable code is executed by a processor of an electronic device, the processor can at least perform the garbage collection method according to the first aspect.

[0017] In a fifth aspect, a garbage collection system is provided, which includes a distributed storage system and a plurality of user storage services using the distributed storage system. The distributed storage system includes a management and control server.

[0018] The management and control server is configured to determine a first garbage collection efficiency of each of the user storage services at a first time according to a set garbage collection time interval, predict a second garbage collection efficiency of each of the user storage services at a second time, the second time being later than the first time, and determine a target user storage service that needs to perform garbage collection at the first time according to a change degree of the second garbage collection efficiency of each of the user storage services compared with the first garbage collection efficiency of each of the user storage services, the first garbage collection efficiency of each of the user storage services, and a remaining storage space capacity of the distributed storage system.

[0019] In the embodiment of the present application, the data generated by the plurality of user storage services in the distributed storage system is periodically subjected to garbage collection processing according to the set garbage collection time interval. At the current first time, the first garbage collection efficiency corresponding to each of the plurality of user storage services is determined, and the second garbage collection efficiency corresponding to each of the plurality of user storage services at a second time thereafter is predicted, so that the degree of change of the garbage collection efficiency of each user storage service in the future time period can be known. Finally, according to the degree of change of the garbage collection efficiency corresponding to each of the plurality of user storage services, the first garbage collection efficiency currently corresponding to each of the plurality of user storage services, and the remaining storage space capacity of the distributed storage system, the target user storage service that needs to be subjected to garbage collection at the current first time is determined, so as to perform garbage collection processing on the stored data of the target user storage service.

[0020] Therefore, in the embodiment of the present application, the target user storage service that needs to be subjected to garbage collection at the current time is not determined only according to the garbage collection efficiency of each user storage service at the current time, but is determined by comprehensively considering the change of the garbage collection efficiency of each user storage service and the remaining storage space capacity of the distributed storage system. In this way, the current decision result can be adapted to the remaining storage space capacity, so as to ensure that the storage space of the distributed storage system is timely recovered, and the remaining storage space capacity is above the safety level. Moreover, by combining the change of the garbage collection efficiency of each user storage service, the target user storage service that is suitable for being subjected to garbage collection processing at the current time is determined, that is, the time at which each user storage service is subjected to garbage collection processing is determined based on the change of the garbage collection efficiency of each user storage service. In this way, not only the storage space can be timely recovered, but also the influence of the garbage collection process on the data read / write service can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 A schematic diagram of a garbage collection system provided in the embodiment of the present application;

[0023] Figure 2 A flowchart of a garbage collection method provided in the embodiment of the present application;

[0024] Figure 3 A flowchart of a garbage collection method provided in the embodiment of the present application;

[0025] Figure 4 A flow chart of a garbage collection method provided for an embodiment of the present application is shown in FIG. 1.

[0026] Figure 5 A schematic diagram of a garbage collection system provided for an embodiment of the present application is shown in FIG. 2.

[0027] Figure 6 A structural schematic diagram of a garbage collection device provided for an embodiment of the present application is shown in FIG. 3.

[0028] Figure 7 A structural schematic diagram of an electronic device provided for an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application. In addition, the sequence of steps in each of the following method embodiments is only an example, not a strict limitation.

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0031] The garbage collection method provided by the embodiments of the present application can be applied to Figure 1 The garbage collection system shown in FIG. 2, for example, includes a distributed storage system and a plurality of user storage services using the distributed storage system, such as Figure 1 The garbage collection system shown in FIG. 2, for example, includes a distributed storage system and a plurality of user storage services using the distributed storage system, such as Figure 1 The user storage service 1, the user storage service 2 and the user storage service 3 shown in FIG. 1. The distributed storage system includes a management and control server.

[0032] It can be understood that a plurality of storage nodes can be included in the distributed storage system, such as Figure 1 The storage node 1 to the storage node N shown in FIG. 1. In actual application, these storage nodes can be physical storage devices maintained by a cloud vendor.

[0033] The management and control server can be any one of a cluster of physical servers or a cluster of virtual servers running a management and control service program in the cloud. In an embodiment of the present application, the management and control service program is mainly used to manage and control the garbage collection process of each user storage service.

[0034] A user storage service is a cloud storage service provided for a user, such as a cloud disk, block storage, etc. In actual application, a user can deploy and run his own application program using a virtual machine in the cloud, and the data generated by the application program can be stored in a storage node in the distributed storage system through the user storage service of the user.

[0035] Therefore, the user storage service can be simply considered as a logical storage space provided for a user and visible to the user, such as a user obtaining 1TB of logical storage space to store data generated by his own application program. In contrast, the storage space provided by the storage node is a physical storage space.

[0036] As described above, in a Log Structured-based distributed storage system, data is written in a continuously appended manner. Specifically, assuming that a user storage service triggers a data write request, the data write request includes data A and the corresponding logical block address LBA=0, the distributed storage system creates a log file (not limited to a log file, other types of files are also possible) corresponding to the user storage service in response to the data write request, assuming that the log file is named log file, a record of data A and LBA=0 is generated in the log file. Assuming that the user storage service triggers a data write request again afterwards, the data write request includes data B and LBA=0, that is, the user storage service actually wants to modify the data originally written to the address LBA=0 from data A to data B, at this time, the distributed storage system responds to the data write request and generates another record of data B and LBA=0 in the log file, that is, an additional record is appended.

[0037] A size of a log file (such as 1GB) can be preset, and a new log file is created when a log file is full. In fact, the log file is only an intermediate tool, and the content written in each log file is ultimately to be written to a storage node in the distributed storage system for persistent storage. A size of a data block (such as 100MB) can be preset, and a data block can be formed when the amount of data written in a log file reaches the size of a data block, and the data block can be stored in different storage nodes in multiple copies (such as three copies) to achieve redundant backup.

[0038] In the embodiment of the present application, garbage collection is performed for the user storage service. In fact, it can be considered that useless data in the log file corresponding to the user storage service is recycled to finally recycle the storage space in the storage node.

[0039] The useless data is mainly generated due to repeated data update or deletion of the user storage service on the same LBA. For example, in the above example, based on the data write request of the user storage service writing data A on LBA=0, the log file first stores data A and the mapping relationship between data A and LBA=0, and then the user storage service updates data A to data B, so that the log file has one more storage record: data B and the mapping relationship between data B and LBA=0. At this time, the first record is invalid data. If the user storage service subsequently deletes data B, the above two records become invalid data.

[0040] When the management server starts a garbage collection process to perform garbage collection for the user storage service, the garbage collection process reads the valid data in the above log file and writes it into a new log file, and then deletes the original log file to recycle the storage space occupied by each data block corresponding to the log file in the storage node. It can be understood that each data block divided based on the new log file will be stored in different storage nodes.

[0041] The above briefly introduces the origin of useless data, and the purpose of garbage collection is to recycle the useless data generated by each user storage service.

[0042] In the embodiment of the present application, one of the main functions of the management server is to reasonably schedule the garbage collection task to meet the following goals:

[0043] First, the storage space is recycled in time to ensure the safety of the storage space level of the distributed storage system, that is, the remaining capacity of the storage space is above the set safety level. The storage space in this paper refers to the physical storage space provided by the storage node in the distributed storage system. Because in the above distributed storage system, new write is constantly appending data, which will make the occupied storage space constantly grow. Only through garbage collection can the growing storage space usage be reduced. Especially when the storage space occupation of the distributed storage system is high and there are a large number of concurrent data write requests, the efficient and timely garbage collection is challenged. If the storage space is not recycled in time, the storage space of the distributed storage system is full, which will cause the interruption of data read and write service.

[0044] Second, the execution of the garbage collection task has as little impact as possible on the foreground data read-write service. As can be seen from the garbage collection process described above, moving data during the garbage collection process will occupy resources such as central processing units (CPUs), networks, and disks, and these resources are often shared with the foreground data read-write service. Goal 2 and goal 1 will conflict in some scenarios, such as when the remaining capacity of the storage space of the distributed storage system is small and there are a large number of concurrent data write requests, both of which require fast storage space recovery, but faster recovery will use more resources, and the foreground data read-write service also needs more resources, which will cause a conflict.

[0045] Based on this, the garbage collection method provided by the embodiments of the present application mainly provides a more optimal garbage collection scheduling strategy to better meet the above two goals and minimize the conflict probability of the two goals.

[0046] The following describes the garbage collection process of the management server for the storage services of each user in combination with some embodiments.

[0047] Figure 2 A flowchart of a garbage collection method provided by the embodiments of the present application is shown in Figure 2 The method includes the following steps:

[0048] 201. According to the set garbage collection time interval, determine the first garbage collection efficiency of each of the plurality of user storage services at the first time.

[0049] 202. Predict the second garbage collection efficiency of each of the plurality of user storage services at the second time, the second time being later than the first time.

[0050] 203. According to the change degree of the second garbage collection efficiency of each of the plurality of user storage services compared with the first garbage collection efficiency of each of the plurality of user storage services, the first garbage collection efficiency of each of the plurality of user storage services, and the remaining capacity of the storage space of the distributed storage system, determine the target user storage service that needs to be garbage collected at the first time.

[0051] In the embodiments of the present application, a concept of garbage collection efficiency is introduced.

[0052] LS is used to represent the total data amount that has been written in the log files corresponding to the user storage service (it refers to all the created and unreclaimed log files corresponding to the user storage service), US is used to represent the valid data amount (or useful data amount) in the log files corresponding to the user storage service, and GCU is used to represent the garbage collection efficiency of the user storage service, and GCU=(LS-US) / US. Wherein, (LS-US) is the recyclable storage space size, that is, the reclamation benefit. Because the valid data US needs to be read out and written into a new log file, US and the reclamation cost (or resource consumption) are proportional, and (LS-US) / US represents the reclamation benefit / reclamation cost of the garbage collection, which is equivalent to the reclamation efficiency of the garbage collection.

[0053] Therefore, the above-mentioned garbage collection efficiency reflects how much useless data needs to be reclaimed at what cost.

[0054] In the embodiment of the present application, according to the set garbage collection time interval, the management server periodically performs garbage collection processing on the data generated by the multiple user storage services using the distributed storage system. For example, the time interval is set to 5 minutes, 10 minutes, etc.

[0055] Suppose that according to the above-mentioned garbage collection time interval, it is determined that the scheduling of performing garbage collection once at the current first time is needed. Then, it is necessary to determine the first garbage collection efficiency corresponding to each of the multiple user storage services using the distributed storage system at the first time.

[0056] The first garbage collection efficiency corresponding to any one of the user storage services at the first time is represented as GCU(t), t represents the current first time, according to the definition of GCU, it is necessary to determine the total data amount (supposed to be represented as LS1) and the valid data amount (supposed to be represented as US1) in the log files corresponding to the user storage service at the first time t, and GCU(t)=(LS1-US1) / US1.

[0057] Wherein, for any user storage service, at the first time, according to the data amount written in each log file corresponding to the user storage service and not yet subjected to garbage collection processing, the above-mentioned total data amount LS1 can be determined. The LBA contained in each log file is determined, the last corresponding data amount of each LBA is determined, and the above-mentioned valid data amount US1 is obtained after accumulation.

[0058] For example, suppose that a user storage service corresponds to a log file, and the log file contains the following records:

[0059] LBA=0, data A;

[0060] LBA=1, data B;

[0061] LBA = 2, data C;

[0062] LBA = 0, data D;

[0063] LBA = 1, data E.

[0064] For ease of description, it is assumed that the amount of data corresponding to each record is 10MB, so the total amount of data LS1 corresponding to this log file is 50MB. Since there is a repeated writing situation, i.e., a data modification situation, for LBA = 0 and LBA = 1, the actual effective data for these two addresses is LBA = 0, data D; and LBA = 1, data E, so the amount of effective data US1 is 30MB, i.e., including the following three pieces of effective data: LBA = 0, data D; LBA = 2, data C; and LBA = 1, data E.

[0065] After determining the first garbage collection efficiency corresponding to each of the plurality of user storage services at the first time, the second garbage collection efficiency corresponding to each of the plurality of user storage services at the second time is predicted. The second time is later than the first time, such as a time interval of a set time length after the first time. The set time length is represented as T, which can be a minute level or an hour level, such as 10 minutes or 1 hour.

[0066] According to the definition of the GCU, for any user storage service, the second garbage collection efficiency GCU(t+T) corresponding to the user storage service at the second time is actually to predict and determine the total amount of data (assuming to be LS2) and the amount of effective data (assuming to be US2) in the log file corresponding to the user storage service at the second time (t+T), so that GCU(t+T)=(LS2-US2) / US2.

[0067] It is found through actual statistical analysis that the data read and write operations of most user storage services present a strong periodicity. For example, a user storage service has a large number of read and write operations in a certain time period during the day, but has very few read and write operations at night. Based on this, the historical data read and write operations of each user storage service can be statistically analyzed to determine the periodic read and write characteristic information presented by each user storage service.

[0068] For ease of prediction, the statistical time granularity when the periodic read and write characteristic information is counted can be, for example, an hour level, i.e., the periodic read and write characteristic information presented by each user storage service in different time periods of each day is analyzed.

[0069] For example, how much data amount of new data write operation is in a certain time period, how much data amount of data modification behavior is in a certain time period, how much data amount of data deletion operation is in a certain time period, and the like.

[0070] The periodic read-write characteristic information can be counted by using existing neural network models and some statistical analysis methods, which will not be described here.

[0071] For any of the above user storage services, the read-write characteristic information of the user storage service in the time period from the first time t to the second time t+T can be determined based on the periodic read-write characteristic information presented by the user storage service. For example, assuming that the counted periodic read-write characteristic information of the user storage server indicates that there is an average data modification amount of 500MB in the time period from 10:00 to 11:00, and assuming that the time period from the first time t to the second time t+T is exactly the time period from 10:00 to 11:00, the read-write characteristic information of the user storage service in the time period from the first time t to the second time t+T is determined to be 500MB of data modification amount.

[0072] It can be understood that, assuming that the time period from the first time t to the second time t+T is a partial time period between 10:00-11:00, such as 10:00-10:30, for the sake of simplifying the processing, the read-write characteristic information of the user storage service in the time period from the first time t to the second time t+T can also be determined as 250MB of data modification amount according to the time length ratio. When the time period from the first time t to the second time t+T is assumed to be 9:40-10:10, similarly, assuming that the periodic statistics time is 9:00-10:00 and 10:00-11:00, the read-write characteristic information of the user storage service in the time period from the first time t to the second time t+T can be determined according to the time length ratio of each in the corresponding two statistics times.

[0073] Then, according to the read-write characteristic information of the above user storage service in the time period from the first time t to the second time t+T, the total data amount LS2 and the valid data amount US2 in the log file corresponding to the user storage service at the second time t+T are determined.

[0074] For example, if the read-write characteristic information of the user storage service in the time period from the first time t to the second time t+T is that there is a data modification amount of 500 MB, since the data is recorded into the log file in a continuous appending manner, compared with the total data amount LS1 in the log file corresponding to the user storage service at the first time t, the total data amount LS2 corresponding to the second time t+T is LS1+500 MB, but the effective data amount does not change, i.e. US2=US1, because, as described above, the data modification operation is a data modification on the same LBA, and the effective data is the final modified data, and in the case that the data amount of the original data is consistent with the modified data (even if the data amount is slightly different, the difference can be ignored), the data amount does not change.

[0075] For another example, if the read-write characteristic information of the user storage service in the time period from the first time t to the second time t+T is that there is a new data write amount of 300 MB, compared with the total data amount LS1 in the log file corresponding to the user storage service at the first time t, the total data amount LS2 corresponding to the second time t+T is LS1+300 MB, and the effective data amount also increases by the data amount, i.e. US2=US1+300 MB.

[0076] For another example, if the read-write characteristic information of the user storage service in the time period from the first time t to the second time t+T is that there is a data deletion behavior of 400 MB, compared with the total data amount LS1 in the log file corresponding to the user storage service at the first time t, the total data amount LS2 corresponding to the second time t+T is LS1+400 MB, because in fact the data deletion behavior is also implemented by writing null values to some original LBAs. However, the effective data amount decreases by the data amount, i.e. US2=US1-400 MB.

[0077] After obtaining the above LS2 and US2, the second garbage collection efficiency GCU(t+T) corresponding to the user storage service at the second time t+T can be determined based on the definition of GCU.

[0078] Then, according to the change degree of the second garbage collection efficiency GCU(t+T) corresponding to each user storage service compared with the first garbage collection efficiency GCU(t) of each user storage service, the first garbage collection efficiency GCU(t) of each user storage service, and the remaining storage space capacity of the distributed storage system, the target user storage service that needs to perform garbage collection at the first time t is determined, so as to start a garbage collection process to perform garbage collection processing on the stored data of the target user storage service.

[0079] Optionally, the change degree of the second garbage collection efficiency GCU(t+T) of any user storage service compared with the first garbage collection efficiency GCU(t) of the user storage service can be measured by the ratio of GCU(t+T) to GCU(t), and the ratio is denoted as DU, i.e., DU=GCU(t+T) / GCU(t). Optionally, the change degree can also be measured by the difference between GCU(t+T) and GCU(t).

[0080] The change degree of the garbage collection efficiency of any user storage service mainly reflects whether the second garbage collection efficiency of the user storage service at the second time is higher than the first garbage collection efficiency of the user storage service at the first time, so as to determine whether the garbage collection of the user storage service is performed at the current first time or delayed.

[0081] In general, if the second garbage collection efficiency of a user storage service is obviously higher than the first garbage collection efficiency of the user storage service, it means that the garbage collection of the user storage service is delayed to obtain better benefits. On the contrary, if the second garbage collection efficiency of a user storage service is less than or equal to the first garbage collection efficiency of the user storage service, it means that the garbage collection efficiency of the user storage service does not improve at the second time, and the garbage collection efficiency of the user storage service is relatively stable, which belongs to “hard bone”. The user storage service should be preferentially collected to avoid the impact on the read-write service in the foreground when a large amount of read-write tasks are performed in the subsequent distributed storage system, because the garbage collection of these user storage services will occupy more resources.

[0082] The remaining capacity of the storage space of the distributed storage system is mainly used to reflect the urgency of garbage collection. It can be understood that if the remaining capacity of the storage space of the distributed storage system is found to be very small (lower than a set threshold) at the current first time, it means that there is not enough remaining storage space available if a large amount of data is written. At this time, the collection of the storage space is more urgent, and the user storage service that can release more storage space should be preferentially collected. Conversely, if the remaining capacity of the storage space of the distributed storage system is large at the current first time, the collection of the storage space is not so urgent, and the “hard bone” can be preferentially collected.

[0083] The first garbage collection efficiency of each user storage service at the current first time is mainly used to determine the collection sequence of the user storage services after determining which user storage services should be preferentially collected based on the remaining capacity of the storage space of the distributed storage system. Because in a scheduling period, garbage collection is generally not performed on all user storage services, but only on part of the user storage services.

[0084] Therefore, in the embodiment of the present application, the target user storage service that needs to be garbage collected at the current time is determined not only according to the garbage collection efficiency of each user storage service at the current first time, but also according to the change of the garbage collection efficiency of each user storage service and the remaining storage space of the distributed storage system. In this way, the decision result at the current time can be adapted to the remaining storage space, and the storage space of the distributed storage system can be recycled in time, so that the remaining storage space is above the safety level. Moreover, the target user storage service that is suitable for garbage collection at the current time is determined according to the change of the garbage collection efficiency of each user storage service, that is, the time for each user storage service to perform garbage collection is determined according to the change of the garbage collection efficiency of each user storage service. In this way, not only the storage space can be recycled in time, but also the influence of the garbage collection process on the data read / write service can be reduced.

[0085] Figure 3 A flowchart of a garbage collection method provided by the embodiment of the present application is shown in FIG. 3, and the method includes the following steps: Figure 3

[0086] 301. According to the set garbage collection time interval, the first garbage collection efficiency of each user storage service of the distributed storage system at the first time is determined.

[0087] 302. The second garbage collection efficiency of each user storage service at the second time is predicted, and the second time is later than the first time.

[0088] 303. It is determined whether the remaining storage space of the distributed storage system at the first time is lower than the set threshold value. If yes, step 304 is performed, otherwise step 305 is performed.

[0089] 304. According to the sorting result of the first garbage collection efficiency of each user storage service, the target user storage service that needs to be garbage collected at the first time is determined.

[0090] 305. According to the change degree of the second garbage collection efficiency of each user storage service compared with the first garbage collection efficiency, a first user storage service set and a second user storage service set are determined. The change degree of the second garbage collection efficiency of each user storage service in the first user storage service set compared with the first garbage collection efficiency is higher than the change degree of the second garbage collection efficiency of each user storage service in the second user storage service set compared with the first garbage collection efficiency.

[0091] ​306、According to the sorting result of the first garbage collection efficiency corresponding to each user storage service in the second user storage service set, a target user storage service needing garbage collection at the first time is determined.

[0092] An optional implementation of the step 203 is introduced in this embodiment.

[0093] Specifically, according to the change degree of the second garbage collection efficiency corresponding to each user storage service compared with the first garbage collection efficiency corresponding to each user storage service, the plurality of user storage services can be divided into two categories, one category corresponding to the first user storage service set and the other category corresponding to the second user storage service set.

[0094] Suppose that DU=GCU(t+T) / GCU(t) is used as the measurement index of the change degree, and in fact, the same reasoning is applied when the difference between the two is used as the measurement index. User storage services with DU values greater than a set threshold (such as 1, 1.5, etc.) are divided into one category and stored in the first user storage service set, and user storage services with DU values less than or equal to the set threshold are divided into one category and stored in the second user storage service set. Wherein, the DU value greater than the set threshold represents that if the corresponding user storage service is delayed for garbage collection, it can bring an improvement in garbage collection efficiency, on the contrary, the DU value less than or equal to the set threshold represents that if the corresponding user storage service is delayed for garbage collection, it will not bring an improvement in garbage collection efficiency, i.e. the effect of delay is not great.

[0095] When the remaining capacity of the storage space of the distributed storage system is less than a set threshold, it means that the storage space of the distributed storage system is running out and is below the safety level at this time, at this time, the two user storage service sets are uniformly sorted, i.e. without considering the size of the DU value, the GCU(t) corresponding to each user storage service is sorted from high to low, and the user storage service with higher GCU(t) is preferentially recycled. Because the GCU(t) corresponding to a user storage service at the current first time is high, according to the definition of GCU, it means that there is more useless data and less valid data in the log file corresponding to the user storage service, and when recycling, the resources consumed for reading valid data into a new log file are less, and more storage space can be freed. At present, due to the low remaining capacity of the storage space of the distributed storage system, it is urgently needed to recycle more storage space faster to ensure the safety of the storage space level of the distributed storage system, therefore, at this time, the DU value can be not considered first, and the user storage service with high GCU(t) is preferentially selected as the target user storage service for garbage collection.

[0096] In addition, optionally, a number or a range of numbers of user storage services to be recovered when the remaining storage space of the distributed storage system is less than the set threshold can be preset. Based on the setting information, the target user storage services can be selected according to the sorting result of GCU(t) from high to low.

[0097] In a specific implementation process, assuming that the number is set as M1, assuming that the GCU(t) ranked as M1 and the GCU(t) ranked as M1+1 are the same, and the two same GCU(t) correspond to one user storage service in the first user storage service set and the other in the second user storage service set, the user storage service in the second user storage service set can be selected. Of course, this is not limited, and one or both can also be selected randomly.

[0098] When the remaining storage space of the distributed storage system is greater than the set threshold, it means that the storage space of the distributed storage system is still relatively sufficient at this time, and at this time, the user storage services in the second user storage service set are preferentially selected for recovery. Specifically, the user storage services in the second user storage service set are sorted from high to low according to their corresponding GCU(t), and the user storage service with higher GCU(t) is preferentially recovered.

[0099] Similarly, optionally, a number or a range of numbers of user storage services to be recovered when the remaining storage space of the distributed storage system is greater than the set threshold can be preset. Based on the setting information, the target user storage services can be selected according to the sorting result of GCU(t) from high to low. In a specific implementation process, assuming that the number is set as M2, the user storage services in the second user storage service set are sorted from high to low according to their corresponding GCU(t), and the first M2 user storage services are selected as the target user storage services.

[0100] When the water level of the storage space of the distributed storage system is above the safety line, the user storage services with low garbage collection efficiency that will not improve in the future are preferentially recovered, and these “hard bones” are digested in the future, that is, when the remaining storage space of the distributed storage system is greater than the set threshold. When the space water level of the distributed storage system is insufficient, because the first user storage service set, that is, the user storage services with high garbage collection efficiency in the second time, contains a large number of user storage services with high GCU(t), more storage space can be recovered with less resources, and the tense situation when the remaining storage space of the distributed storage system is insufficient and there are a large number of concurrent data write requests can be alleviated.

[0101] Figure 4A flow chart of a garbage collection method provided by an embodiment of the present application is shown in FIG. 3, which includes the following steps: Figure 4

[0102] 401. Determine the first garbage collection efficiency of each of the plurality of user storage services at a first time according to a set garbage collection time interval.

[0103] 402. Predict the second garbage collection efficiency of each of the plurality of user storage services at a second time, which is later than the first time.

[0104] 403. Determine the target number of garbage collection processes to be started according to the load of the distributed storage system at the first time.

[0105] 404. Determine whether the remaining storage space of the distributed storage system at the first time is lower than a set threshold value, if yes, execute step 405, otherwise execute step 406.

[0106] 405. Determine the target user storage services that need to be garbage collected at the first time according to the sorting result of the first garbage collection efficiency of each of the plurality of user storage services and the target number of garbage collection processes, so as to perform garbage collection processing on the target user storage services through the target number of garbage collection processes concurrently.

[0107] 406. Determine the first user storage service set and the second user storage service set according to the change degree of the second garbage collection efficiency of each of the plurality of user storage services compared with the first garbage collection efficiency; wherein the change degree of the second garbage collection efficiency of each user storage service in the first user storage service set compared with the first garbage collection efficiency is higher than the change degree of the second garbage collection efficiency of each user storage service in the second user storage service set compared with the first garbage collection efficiency.

[0108] 407. Determine the target user storage services that need to be garbage collected at the first time according to the sorting result of the first garbage collection efficiency of each of the user storage services in the second user storage service set and the target number of garbage collection processes, so as to perform garbage collection processing on the target user storage services through the target number of garbage collection processes concurrently.

[0109] In this embodiment, the garbage collection scheduling is combined with the load of the distributed storage system. The load refers to the foreground data read-write request, which can be measured by the number of read-write requests, throughput, etc.

[0110] ​In fact, the load of the distributed storage system also has obvious periodic peaks and troughs. In summary, the concurrency of the garbage collection task can be increased during the load trough period to perform larger granularity recovery. Meanwhile, some "hard-to-recover" garbage collection tasks that have low efficiency but will not become more efficient in the future should be preferentially recovered during the trough to avoid being unable to allocate sufficient time and resources for recovery during the load peak period.

[0111] Therefore, the load of the distributed storage system at the current first time can be determined, and the target number of garbage collection processes to be started is determined according to the load.

[0112] Specifically, a plurality of load ranges can be pre-set, and a corresponding relationship between different load ranges and the number of garbage collection processes is set, wherein the corresponding relationship reflects that the higher the load, the lower the number of garbage collection processes.

[0113] Therefore, the target load range to which the load of the distributed storage system at the first time belongs can be determined, and the target number of garbage collection processes corresponding to the target load range is determined based on the above corresponding relationship.

[0114] For example, assume that the load range is divided into a high load range and a low load range by a set threshold.

[0115] If the remaining capacity of the storage space of the distributed storage system at the current first time is greater than the set threshold, and the load is in the higher load range, the concurrency of garbage collection is adjusted to be lower, that is, a smaller number K1 of garbage collection processes is set, and K1 GCU(t) high user storage services in the second user storage service set are selected for garbage collection.

[0116] If the remaining capacity of the storage space of the distributed storage system at the current first time is greater than the set threshold, and the load is in the lower load range, the concurrency of garbage collection is adjusted to be higher, that is, a larger number K2 of garbage collection processes is set, and K2 GCU(t) high user storage services in the second user storage service set are selected for garbage collection. K2>K1.

[0117] If the remaining capacity of the storage space of the distributed storage system at the current first time is less than the set threshold, and the load is in the higher load range, the concurrency of garbage collection is adjusted to be lower, that is, a smaller number K3 of garbage collection processes is set, and all user storage services included in the first user storage service set and the second user storage service set are sorted in descending order of GCU(t), and K3 GCU(t) high user storage services are selected for garbage collection.

[0118] If the remaining storage space of the current first time-distributed storage system is less than a set threshold, and the load thereof is in a lower load interval, the concurrency of the garbage collection is increased, that is, a larger number K4 of garbage collection processes is set, and all user storage services in the first user storage service set and the second user storage service set are sorted in a descending order of GCU(t), and K4 user storage services with high GCU(t) are selected for garbage collection. K4>K3.

[0119] K1 and K3 can be equal or different, and K2 and K4 can be equal or different.

[0120] In the embodiment, the user storage services with garbage collection efficiency that will not be improved are preferentially collected by predicting the garbage collection efficiency of the user storage services at a future time, and the concurrency of the garbage collection process is adjusted according to the load level of the distributed storage system, so that the user storage services with garbage collection efficiency that will not be improved are digested in normal times. When the remaining storage space of the distributed storage system is insufficient, because the user storage services with garbage collection efficiency that will be improved are reserved, the user storage services include many user storage services with high GCU(t), and a large amount of storage space can be collected by consuming a small amount of resources, so that the tense situation in the scenario that the remaining storage space of the distributed storage system is insufficient and there are a large number of concurrent write-in operations is relieved, and the two goals of the garbage collection in the foregoing are better ensured.

[0121] In summary, in the embodiment of the application, the user storage services are divided into two categories of high-delayed-recovery-benefit and low-delayed-recovery-benefit by analyzing the read-write characteristic information of the user storage services and predicting the future garbage collection efficiency, and the number and priority of the user storage services are dynamically adjusted according to the current load level and the remaining storage space of the distributed storage system, so that the following effects can be achieved:

[0122] 1. The garbage collection efficiency of some user storage services is improved by delayed recovery, and the amount of resources consumed in the future garbage collection is reduced.

[0123] 2. The concurrency of the garbage collection process is dynamically adjusted according to the load level of the distributed storage system, and the influence on the foreground data read-write task is reduced.

[0124] 3. When the remaining storage space of the distributed storage system is sufficient, the user storage services with general garbage collection efficiency and garbage collection efficiency that will not be significantly improved at a future time are preferentially collected, so that the storage space occupied by the user storage services is recovered for use at an extreme moment (for example, when there are a large number of concurrent data write-in tasks), and the protection ability of the safe water level of the storage space of the distributed storage system is improved.

[0125] 4. In extreme scenarios where the remaining storage capacity of a distributed storage system is insufficient and there are a large number of concurrent data writing tasks, it can find user storage services with high garbage collection efficiency, consume less resources to reclaim more storage space, improve the protection capability of the distributed storage system's storage space safety level, and reduce the impact on data read and write tasks.

[0126] The above introduces Figure 1 In the system shown, the process of the management server scheduling garbage collection is described. In one optional embodiment, the management server can communicate with each user storage service and each storage node to process data read and write requests triggered by each user storage service and to start the garbage collection process to perform garbage collection on each user storage service. It is also used to query the remaining storage capacity of each storage node to determine the remaining storage capacity of the distributed storage system at different times.

[0127] However, in another alternative embodiment, the distributed storage system may include other functional entities in addition to the aforementioned management server and several storage nodes. Some of the functions provided by the management server can be performed by these functional entities; in other words, the management server can provide data read / write services and garbage collection services by managing these functional entities.

[0128] The following is combined Figure 5 This example illustrates the composition structure of a distributed storage system and the implementation process of the aforementioned garbage collection scheduling method within that system.

[0129] Figure 5 This is a schematic diagram of a waste recycling system provided in an embodiment of the present invention, such as... Figure 5 As shown, the garbage collection system includes: a distributed storage system and multiple user storage services using the distributed storage system, such as... Figure 5 The diagram shows User Storage Service 1, User Storage Service 2, and User Storage Service 3.

[0130] A distributed storage system includes a management server and multiple storage nodes, such as Figure 5 The diagram shows storage node 1 through storage node N.

[0131] In addition, such as Figure 5 As shown, the distributed storage system also includes a metadata server and multiple read / write service processes. These multiple read / write service processes include, for example... Figure 5 The diagram illustrates read / write service process 1, read / write service process 2, and read / write service process 3. These multiple storage nodes are connected to the metadata server.

[0132] In fact, the distributed storage system further comprises a plurality of garbage collection processes, such as garbage collection process 1, garbage collection process 2 and garbage collection process 3 shown in FIG. 1. Figure 5 In fact, the distributed storage system further comprises a plurality of garbage collection processes, such as garbage collection process 1, garbage collection process 2 and garbage collection process 3 shown in FIG. 1.

[0133] Briefly, the read-write service process provides data read-write capability for the user storage service and processes data read-write requests triggered by the user storage service; the garbage collection process is used for garbage collection processing of the log file corresponding to the user storage service; the storage node is used for storing each data block split from the log file; and the metadata server is used for implementing splitting, storage and query management of the data block.

[0134] From the perspective of physical deployment, the garbage collection process and the read-write service process can be located in the storage node. That is, all or part of the storage nodes can start the two processes by default when running. Only the control server sends corresponding tasks to the two processes: sends data read-write requests triggered by the user storage service to the read-write service process, and sends garbage collection tasks for a user storage service to the garbage collection process.

[0135] The distributed storage system composed of the above-mentioned multiple functional entities can be divided into two layers from the functional logic level, as shown in FIG. 1. Figure 5 The bottom layer is a data persistence layer composed of a plurality of storage nodes and metadata servers, which provides an append-only logfile read-write service. Above the data persistence layer is composed of a plurality of read-write service processes, a plurality of garbage collection processes and a control server. The read-write service process can create a logfile and continuously write data in the logfile in an append-only manner, or randomly read data therefrom.

[0136] Based on the composition of the above-mentioned distributed storage system, the data read-write process will be briefly introduced as follows.

[0137] Suppose that the user storage service 1 triggers a data write request, and according to the load balancing algorithm, the data write request is assigned to the read-write service process 1. Suppose that the logfiles created by the read-write service process 1 for the user storage service 1 include logfile A1 and logfile A2, wherein logfile A2 is created after logfile A1 is full of a set data amount. The data write request triggered by the user storage service 1 includes data X to be written and the LBA corresponding to data X. The read-write service process 1 writes data X and its corresponding LBA into logfile A2 based on the data write request, and stores the mapping relationship between the LBA and the data position of data X in logfile A2 in the memory of the device where the read-write service process 1 is located.

[0138] Afterwards, in response to the writing of the data X and its corresponding LBA in the log file 2, the metadata server generates a new data block corresponding to the newly written data when it determines that the newly written data meets a set data block size, stores the data block and its multiple copies in different storage nodes, and stores the correspondence between the data block and its copies and the storage nodes, the user storage service, and the corresponding data location in the log file 2. In this way, the data writing process of the data X is completed.

[0139] Suppose that the user storage service 1 needs to read the data X later, a data reading request containing the LBA corresponding to the data X is triggered, and the read-write service process 1 queries the above mapping relationship between the LBA and the data location in the log file based on the LBA carried in the data reading request to determine the data location of the data X corresponding to the LBA in the log file A2. Afterwards, optionally, the read-write service process 1 can read the data X from the data location in the log file A2 to feed back to the user storage service 1. Alternatively, if the reading from the log file A2 is not successful, the read-write service process 1 can send a query request to the metadata server, which can include the user storage service 1 and the data location. The metadata server queries to determine the storage node where the data block corresponding to the user storage service 1 and the data location is located, feeds back the identifier of the storage node to the read-write service process 1, so that the read-write service process 1 reads the data X from the storage node.

[0140] In the Figure 5 , suppose that the log file corresponding to the user storage service 1 includes the log file A1 and the log file A2, the log file corresponding to the user storage service 2 includes the log file B1, the log file B2 and the log file B3, and the log file corresponding to the user storage service 3 includes the log file C1 and the log file C2.

[0141] Suppose that according to the set garbage collection time interval, it is determined that the first time needs to be garbage collected once, the management and control server can query the total data amount and the valid data amount of the log file corresponding to each user storage service from each read-write service process, and can also query the load of each read-write service process to determine the load of the distributed storage system, and query the metadata server to determine the remaining capacity of the storage space of the current distributed storage system to determine the target user storage service that needs to be garbage collected according to the garbage collection method introduced in the foregoing embodiments.

[0142] Specifically, the metadata server determines the storage space remaining capacity of the distributed storage system according to the storage space remaining capacities of the plurality of storage nodes in the distributed storage system at the first time, and sends the storage space remaining capacity of the distributed storage system to the management and control server. The plurality of read-write service processes determine the total data amount and the valid data amount in the log files corresponding to the plurality of user storage services respectively at the first time, and send the total data amount and the valid data amount in the log files corresponding to the plurality of user storage services respectively to the management and control server, so that the management and control server determines the first garbage collection efficiency of the plurality of user storage services respectively at the first time.

[0143] The plurality of read-write service processes send their respective loads at the first time to the management and control server, and the management and control server determines the load of the distributed storage system at the first time according to the loads of the plurality of read-write service processes at the first time, so that the target number of garbage collection processes to be started can be determined according to the load of the distributed storage system at the first time, and the target number of garbage collection processes are started to perform garbage collection processing on the target user storage services.

[0144] Suppose it is determined that the user storage service 1, the user storage service 2 and the user storage service 3 need to be sequentially subjected to garbage collection processing at present, then the management and control server can send corresponding garbage collection tasks to the three garbage collection processes as shown in the flowchart of FIG. 8. Figure 5 For example, the garbage collection task for the user storage service 1 is sent to the garbage collection process 1, the garbage collection task for the user storage service 2 is sent to the garbage collection process 2, and the garbage collection task for the user storage service 3 is sent to the garbage collection process 3. In the garbage collection task, the identifier of the corresponding user storage service and the identifier of the log file corresponding to the user storage service can be included.

[0145] Taking the user storage service 1 as an example, the garbage collection task received by the garbage collection process 1 includes the identifiers of the log file A1 and the log file A2. The garbage collection process 1 reads the valid data in the log file A1 and the log file A2 to write into a new log file A12, and updates the LBA of the valid data and the data position in the log file A12 to the read-write service process 1 corresponding to the user storage service 1, and then deletes the log file A1 and the log file A2 to recover the storage space.

[0146] It can be understood that in the process of deleting the log file A1 and the log file A2, the garbage collection process 1 needs to query the metadata server to determine the storage nodes corresponding to the data blocks corresponding to the two log files, so as to delete the data blocks from the corresponding storage nodes. Similarly, for the newly generated log file A12, the metadata server determines the data blocks corresponding to the log file A12, which are stored in different storage nodes in the form of multiple copies.

[0147] The garbage collection processes of the user storage service 2 and the user storage service 3 are the same. In the user storage service 2, the valid data in the log files B1, B2 and B3 is read out and written into a new log file B13, and the log files B1, B2 and B3 are deleted by the garbage collection process 2. Figure 6 The garbage collection processes of the user storage service 2 and the user storage service 3 are the same. In the user storage service 2, the valid data in the log files B1, B2 and B3 is read out and written into a new log file B13, and the log files B1, B2 and B3 are deleted by the garbage collection process 2.

[0148] The garbage collection apparatuses of one or more embodiments of the present application will be described in detail below. Those skilled in the art can understand that these apparatuses can be configured by using commercially available hardware components through the steps taught by the present solution.

[0149] Figure 6 A structure diagram of a garbage collection apparatus provided for an embodiment of the present application is shown in the figure. The apparatus is applied to a management server. As shown in the figure, the apparatus comprises a determination module 11, a prediction module 12 and a scheduling module 13. Figure 6

[0150] The determination module 11 is configured to determine, according to a set garbage collection time interval, first garbage collection efficiencies respectively corresponding to a plurality of user storage services using a distributed storage system at a first time.

[0151] The prediction module 12 is configured to predict second garbage collection efficiencies respectively corresponding to the plurality of user storage services at a second time, the second time being later than the first time.

[0152] The scheduling module 13 is configured to determine, according to a change degree of the second garbage collection efficiencies respectively corresponding to the plurality of user storage services compared with the first garbage collection efficiencies respectively corresponding to the plurality of user storage services, the first garbage collection efficiencies respectively corresponding to the plurality of user storage services and a remaining storage space capacity of the distributed storage system, a target user storage service that needs to be garbage collected at the first time.

[0153] Optionally, the scheduling module 13 is specifically configured to: if the remaining storage space capacity of the distributed storage system is lower than a set threshold, determine, according to a sorting result of the first garbage collection efficiencies respectively corresponding to the plurality of user storage services, the target user storage service that needs to be garbage collected at the first time.

[0154] ​Optionally, the scheduling module 13 is specifically configured to: if the remaining capacity of the storage space of the distributed storage system is higher than a set threshold, determine a first user storage service set and a second user storage service set according to the change degree of the second garbage collection efficiency of each of the user storage services compared with the first garbage collection efficiency; wherein the change degree of the second garbage collection efficiency of each of the user storage services in the first user storage service set compared with the first garbage collection efficiency is higher than the change degree of the second garbage collection efficiency of each of the user storage services in the second user storage service set compared with the first garbage collection efficiency; and determine the target user storage service requiring garbage collection at the first time according to the sorting result of the first garbage collection efficiency of each of the user storage services in the second user storage service set.

[0155] Optionally, the scheduling module 13 is further configured to: determine the target number of garbage collection processes to be started according to the load of the distributed storage system at the first time; and determine the target user storage service requiring garbage collection at the first time according to the target number of garbage collection processes, so as to perform garbage collection processing on the target user storage service through the target number of garbage collection processes concurrently.

[0156] Optionally, the scheduling module 13 is specifically configured to: determine a target load range to which the load of the distributed storage system at the first time belongs in a set of multiple load ranges; and determine the target number according to a correspondence between different load ranges and the number of garbage collection processes and the target load range; wherein the correspondence reflects that the higher the load, the lower the number of garbage collection processes.

[0157] Optionally, the determining module 11 is specifically configured to: for any user storage service in the multiple user storage services, determine the total data amount and the valid data amount in the log file corresponding to the any user storage service at the first time; determine the useless data amount in the log file corresponding to the any user storage service at the first time according to the total data amount and the valid data amount in the log file corresponding to the any user storage service at the first time; and determine the first garbage collection efficiency of the any user storage service at the first time as the ratio of the useless data amount to the valid data amount of the any user storage service at the first time.

[0158] Optionally, the prediction module 12 is specifically configured to: acquire periodic read-write characteristic information presented by the any user storage service; determine read-write characteristic information of the any user storage service in a time period from the first time to the second time according to the periodic read-write characteristic information; determine a total data amount and an effective data amount in the log file corresponding to the any user storage service at the second time according to the read-write characteristic information of the any user storage service in the time period from the first time to the second time; and determine the second garbage collection efficiency of the any user storage service at the second time according to the total data amount and the effective data amount in the log file corresponding to the any user storage service at the second time.

[0159] Figure 6 The apparatus can perform the steps performed by the server in the foregoing embodiments, and the detailed execution process and technical effects are described in the foregoing embodiments, which will not be described here.

[0160] In one possible design, the above Figure 7 The garbage collection apparatus can be implemented as an electronic device. As shown in ​ The electronic device can include a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code. When the executable code is executed by the processor 21, the processor 21 can at least implement the garbage collection method performed by the client in the foregoing embodiments.

[0161] In addition, the embodiment of the present application provides a non-transitory machine readable storage medium, and the non-transitory machine readable storage medium stores executable code. When the executable code is executed by the processor of the electronic device, the processor can at least implement the garbage collection method provided in the foregoing embodiments.

[0162] The apparatus embodiments described above are merely schematic, and units described as separate components can or can not be physically separated. Part or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0163] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of the necessary general hardware platform, and of course can also be implemented by means of the combination of hardware and software. Based on such understanding, the above technical solutions can be embodied in the form of computer products, and the present application can be in the form of computer program products implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0164] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A garbage collection method characterized by, The method comprises: determining respective first garbage collection efficiencies of a plurality of user storage services using a distributed storage system at a first time according to a set garbage collection time interval; predicting respective second garbage collection efficiencies of the plurality of user storage services at a second time later than the first time; determining target user storage services requiring garbage collection at the first time according to changes in the respective second garbage collection efficiencies of the plurality of user storage services compared with the respective first garbage collection efficiencies, the respective first garbage collection efficiencies of the plurality of user storage services, and a remaining storage space capacity of the distributed storage system.

2. The method of claim 1, wherein, The determining of the target user storage services requiring garbage collection at the first time according to the changes in the respective second garbage collection efficiencies of the plurality of user storage services compared with the respective first garbage collection efficiencies, the respective first garbage collection efficiencies of the plurality of user storage services, and the remaining storage space capacity of the distributed storage system comprises: if the remaining storage space capacity of the distributed storage system is lower than a set threshold, determining the target user storage services requiring garbage collection at the first time according to a sorting result of the respective first garbage collection efficiencies of the plurality of user storage services.

3. The method of claim 1, wherein, The determining of the target user storage services requiring garbage collection at the first time according to the changes in the respective second garbage collection efficiencies of the plurality of user storage services compared with the respective first garbage collection efficiencies, the respective first garbage collection efficiencies of the plurality of user storage services, and the remaining storage space capacity of the distributed storage system comprises: if the remaining storage space capacity of the distributed storage system is higher than a set threshold, determining a first user storage service set and a second user storage service set according to the changes in the respective second garbage collection efficiencies of the plurality of user storage services compared with the respective first garbage collection efficiencies; wherein the change in the respective second garbage collection efficiency of each user storage service in the first user storage service set compared with the respective first garbage collection efficiency is higher than the change in the respective second garbage collection efficiency of each user storage service in the second user storage service set compared with the respective first garbage collection efficiency; determining the target user storage services requiring garbage collection at the first time according to a sorting result of the respective first garbage collection efficiencies of the user storage services in the second user storage service set.

4. The method according to claim 2 or 3, characterized in that, The determining of the target user storage services requiring garbage collection at the first time comprises: determining a target number of garbage collection processes to be started according to a load of the distributed storage system at the first time; determining the target user storage services requiring garbage collection at the first time according to the target number of garbage collection processes, so as to perform garbage collection processing on the target user storage services concurrently through the target number of garbage collection processes.

5. The method of claim 4, wherein, The target number of garbage collection processes to be started is determined according to the load of the distributed storage system at the first time, including: determining a target load range to which the load of the distributed storage system at the first time belongs, in a plurality of set load ranges; determining the target number according to a correspondence relationship between different load ranges and the number of garbage collection processes and the target load range, wherein the correspondence relationship reflects that the higher the load, the lower the number of garbage collection processes.

6. The method according to any one of claims 1 to 5, characterized in that, The first garbage collection efficiency of each of the plurality of user storage services of the distributed storage system at the first time is determined, including: for any one of the plurality of user storage services, determining the total data amount and the valid data amount in the log file corresponding to the any one user storage service at the first time; determining the amount of useless data corresponding to the any one user storage service at the first time according to the total data amount and the valid data amount in the log file corresponding to the any one user storage service at the first time; determining the ratio of the amount of useless data to the valid data of the any one user storage service at the first time as the first garbage collection efficiency of the any one user storage service at the first time.

7. The method of claim 6, wherein, The second garbage collection efficiency of each of the plurality of user storage services at the second time is predicted, including: obtaining periodic read-write characteristic information exhibited by the any one user storage service; determining the read-write characteristic information of the any one user storage service in a time period from the first time to the second time according to the periodic read-write characteristic information; determining the total data amount and the valid data amount in the log file corresponding to the any one user storage service at the second time according to the read-write characteristic information of the any one user storage service in the time period from the first time to the second time; determining the second garbage collection efficiency of the any one user storage service at the second time according to the total data amount and the valid data amount in the log file corresponding to the any one user storage service at the second time.

8. An electronic device, comprising: including: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the garbage collection method according to any one of claims 1 to 7.

9. A non-transitory machine-readable storage medium, comprising: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by the processor of the electronic device, the processor executes the garbage collection method according to any one of claims 1 to 7.

10. A garbage collection system, characterized by, including: a distributed storage system and a plurality of user storage services using the distributed storage system; wherein the distributed storage system comprises a control server; the control server is configured to execute the garbage collection method according to any one of claims 1 to 7.

11. The system of claim 10, wherein, The distributed storage system further comprises a plurality of read-write service processes, a metadata server, and a plurality of storage nodes connected to the metadata server; The metadata server is configured to determine the remaining storage space of the distributed storage system according to the remaining storage space of the plurality of storage nodes at the first time, and send the remaining storage space of the distributed storage system to the management server. The plurality of read-write service processes are configured to determine the total data amount and the valid data amount in the log files corresponding to the plurality of user storage services respectively at the first time, and send the total data amount and the valid data amount in the log files corresponding to the plurality of user storage services respectively to the management server, so that the management server determines the first garbage collection efficiency corresponding to the plurality of user storage services respectively at the first time.

12. The system of claim 11, wherein, The plurality of read-write service processes are further configured to send the respective loads at the first time to the management server. The management server is further configured to determine the load of the distributed storage system at the first time according to the loads of the plurality of read-write service processes at the first time, determine the target number of garbage collection processes to be started according to the load of the distributed storage system at the first time, and start the target number of garbage collection processes to perform garbage collection processing on the target user storage services.

Images