Garbage collection method, device and storage medium for partition storage device

Abstract

The disclosed embodiment provides a garbage collection method, device and storage medium for a partition storage device, which determines valid data segments in a source partition to be recycled in the partition storage device according to metadata of a file object cached in memory; reads each valid data segment from the source partition, writes the read valid data segment into a cache pool corresponding to the corresponding file object in memory, and aggregates the valid data segments in the same cache pool into a valid aggregated data segment; determines a target cache pool to be written into the partition storage device from all cache pools, writes the valid aggregated data segments in the target cache pool into a target partition in the partition storage device based on a sequential write mechanism, and recycles the storage space of the source partition after all valid data segments in the source partition have been read. By reducing the number of metadata queries and data writes during the garbage collection process, impact on the partition storage device is avoided, and normal read and write performance is guaranteed.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular to a garbage collection method, device and storage medium for a partitioned storage device. Background Art

[0002] After years of development, solid state disks (SSDs) have been widely used in various storage centers. Traditional SSDs are provided to the host in the form of block devices, but it is difficult for the host to exert the true performance of the SSD, and the data is stored randomly, and it is impossible to distinguish between hot and cold data. At the same time, garbage collection is also required to achieve effective data migration and data erasure, but garbage collection inside the SSD will occupy a large amount of channel bandwidth, causing a significant decrease in SSD performance. At the same time, the host's latency cannot be guaranteed, and garbage collection further affects the life of the SSD. Due to these shortcomings of traditional SSDs, and the host is closer to the data and knows more data details, the ZNS (Zoned Storage) protocol and ZNS devices have emerged.

[0003] In a ZNS device, the entire device is divided into partitions (Zones) that can only be written sequentially. When performing garbage collection on a ZNS device, it is necessary to migrate valid data in the partition to other partitions and then erase the partition so that the partition can be used again.

[0004] However, when there are many data entries stored in the partition, garbage collection will have an impact on the ZNS device, thereby affecting the normal read and write performance of the ZNS device. Summary of the invention

[0005] The embodiments of the present disclosure provide a method, device and storage medium for garbage collection of a partitioned storage device, so as to avoid impact on the partitioned storage device during garbage collection and ensure normal read and write performance of the partitioned storage device.

[0006] In a first aspect, an embodiment of the present disclosure provides a garbage collection method for a partitioned storage device, comprising:

[0007] Determining valid data segments in a source partition to be reclaimed in a partition storage device according to file object metadata cached in the memory, wherein the file object corresponding to the file object metadata is stored in the partition storage device;

[0008] Reading each valid data segment from the source partition, writing the read valid data segment into a cache pool corresponding to the file object in the memory according to the file object to which the valid data segment belongs, and aggregating the valid data segments in the same cache pool into a valid aggregated data segment;

[0009] Determine a target cache pool to be written to the partition storage device from all cache pools, write valid aggregated data segments in the target cache pool to the target partition in the partition storage device based on a sequential write mechanism, and reclaim the storage space of the source partition after all valid data segments in the source partition have been read.

[0010] In a second aspect, an embodiment of the present disclosure provides a garbage collection device for a partitioned storage device, including:

[0011] A valid data determination unit, configured to determine a valid data segment in a source partition to be recovered in a partition storage device according to file object metadata cached in the memory, wherein the file object corresponding to the file object metadata is stored in the partition storage device;

[0012] a reading unit, configured to read each valid data segment from the source partition, write the read valid data segment into a cache pool corresponding to the file object in the memory according to the file object to which the valid data segment belongs, and aggregate the valid data segments in the same cache pool into one valid aggregated data segment;

[0013] A writing unit is used to determine a target cache pool to be written to a partition storage device from all cache pools, write valid aggregated data segments in the target cache pool into a target partition in the partition storage device based on a sequential writing mechanism, and reclaim the storage space of the source partition after all valid data segments in the source partition have been read.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0015] The memory stores computer-executable instructions;

[0016] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the garbage collection method for the partitioned storage device as described in the first aspect and various possible designs of the first aspect.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, a garbage collection method for a partitioned storage device as described in the first aspect and various possible designs of the first aspect is implemented.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer execution instructions. When a processor executes the computer execution instructions, it implements the garbage collection method of the partitioned storage device as described in the first aspect and various possible designs of the first aspect.

[0019] The garbage collection method, device and storage medium of a partitioned storage device provided by the embodiments of the present disclosure determine valid data segments in a source partition to be recycled in the partitioned storage device according to metadata of file objects cached in a memory, wherein the file objects corresponding to the file object metadata are stored in the partitioned storage device; read each valid data segment from the source partition, and write the read valid data segment into a cache pool corresponding to the file object in the memory according to the file object to which the valid data segment belongs, and aggregate the valid data segments in the same cache pool into a valid aggregated data segment; determine a target cache pool to be written into the partitioned storage device from all cache pools, write the valid aggregated data segments in the target cache pool into a target partition in the partitioned storage device based on a sequential write mechanism, and recycle the storage space of the source partition after all valid data segments in the source partition have been read. In this embodiment, by batch querying the valid data segments in the source partition from the file object metadata cached in the memory, the metadata query can be reduced; and after reading each valid data segment in the source partition, the data segments are aggregated in the cache pool of the memory according to the file objects to which they belong and then written to the target partition, which can effectively reduce the number of data writes, avoid impact on the partition storage device, ensure the normal read and write performance of the partition storage device, and the aggregation of valid data belonging to the same file object can provide a larger bandwidth and reasonable delay for subsequent reading. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 A scenario example diagram of a garbage collection method for a partitioned storage device provided in an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a garbage collection method for a partitioned storage device according to an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of the organization of the middle block and the metadata in the memory of a partition storage device provided by an embodiment of the present disclosure;

[0024] Figure 4A schematic flow chart of a garbage collection method for a partitioned storage device provided by another embodiment of the present disclosure;

[0025] Figure 5 A structural block diagram of a garbage collection device for a partition storage device provided in an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] In a ZNS device, the entire device is divided into partitions (Zones) that can only be written sequentially. When performing garbage collection on a ZNS device, it is necessary to migrate valid data in the partition to other partitions and then erase the partition (the Zone Reset command can be used to reset the write pointer to the partition start address) so that the partition can be used again.

[0029] However, when there are many data items stored in the partition, garbage collection will have an impact on the ZNS device, thereby affecting the normal read and write performance of the ZNS device. The specific process may include:

[0030] a) Read the reverse index information of each data segment in the source partition;

[0031] b) Traverse the reverse index information of each data segment in turn and query the metadata in the memory to determine whether the data segment is a valid data segment. If it is invalid data, skip it, otherwise execute c);

[0032] c) Read the valid data segment and write it into the target partition;

[0033] d) Repeat the above steps until all data segments in the source partition are traversed, and then erase the source partition, thereby turning it into an empty partition that can be used again.

[0034] However, as the capacity of SSD NAND continues to grow, the size of a partition of the ZNS device is also getting larger and larger, and the number of data entries stored in the partition is also increasing. Whether it is in the process of determining whether it is a valid data segment to find the reverse index information, or in the process of relocating each valid data segment, a large number of query requests or read and write requests will be generated, increasing the pressure on the ZNS device and affecting the normal reading and writing of the ZNS device (front-end reading and writing for users).

[0035] In order to solve at least one of the above technical problems, the present disclosure provides a garbage collection method for a partitioned storage device, which determines the valid data segments in the source partition to be recycled in the partitioned storage device according to the file object metadata cached in the memory, wherein the file object corresponding to the file object metadata is stored in the partitioned storage device; reads each valid data segment from the source partition, and writes the read valid data segment into a cache pool corresponding to the file object in the memory according to the file object to which the valid data segment belongs, and aggregates the valid data segments in the same cache pool into a valid aggregated data segment; determines the target cache pool to be written into the partitioned storage device from all cache pools, writes the valid aggregated data segments in the target cache pool into the target partition in the partitioned storage device based on a sequential write mechanism, and recycles the storage space of the source partition after all valid data segments in the source partition have been read. In this embodiment, by batch querying the valid data segments in the source partition from the file object metadata cached in the memory, the metadata query can be reduced; and after reading each valid data segment in the source partition, the data segments are aggregated in the cache pool of the memory according to the file objects to which they belong and then written to the target partition, which can effectively reduce the number of data writes (IOPS) and avoid impact on the partition storage device, thereby ensuring the normal read and write performance of the partition storage device, and the aggregation of valid data belonging to the same file object can provide a larger bandwidth and reasonable delay for subsequent reading.

[0036] The garbage collection method for partition storage devices disclosed in the present invention is applicable to the following scenarios: Figure 1 As shown, valid data segments in a source partition to be reclaimed in a partition storage device are determined based on metadata of file objects cached in memory; each valid data segment is read from the source partition, and based on the file object to which the valid data segment belongs, the read valid data segment is written into a cache pool in memory corresponding to the file object to which it belongs, and the valid data segments in the same cache pool are aggregated into a valid aggregated data segment; a target cache pool to be written into the partition storage device is determined from all cache pools, and the valid aggregated data segments in the target cache pool are written into the target partition in the partition storage device based on a sequential write mechanism, and the storage space of the source partition is reclaimed after all valid data segments in the source partition have been read.

[0037] It should be noted that the data involved in this disclosure (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0038] The garbage collection method of the partition storage device disclosed in the present invention will be described in detail below in conjunction with specific embodiments.

[0039] refer to Figure 2 , Figure 2 A flow chart of a garbage collection method for a partitioned storage device provided in an embodiment of the present disclosure. The method of this embodiment can be applied in a terminal device or a server. The garbage collection method for a partitioned storage device includes:

[0040] S201. Determine valid data segments in a source partition to be reclaimed in a partition storage device according to file object metadata cached in a memory, wherein a file object corresponding to the file object metadata is stored in the partition storage device.

[0041] In this embodiment, when garbage collection is required for a partitioned storage (ZNS) device, the partition that needs garbage collection is first determined as the source partition. In addition, a target partition to which valid data segments in the source partition are to be relocated can be selected. The target partition can be an empty partition. After the valid data in the source partition is subsequently migrated to the target partition, the source partition can be cleared to reclaim the storage space of the source partition and realize garbage collection.

[0042] Considering the identification of valid data in the source partition when performing garbage collection in the traditional partition storage (ZNS) device, it is usually to read the reverse index information of each data segment (data entry) in the source partition, wherein each data segment is a part of the data of one or more file objects, and traverse the reverse index information of each data segment in turn, query the metadata of the file object cached in the memory, if the matching metadata is queried for the reverse index information of any data segment, then the data segment is determined to be a valid data segment, otherwise it is an invalid data segment (garbage data), when the number of data segments in the source partition is large (possibly hundreds of thousands or millions), traversing the reverse index information of each data segment to query the metadata will result in too large a query volume (amount of queries per second QPS), which will cause an impact on the metadata, increase the pressure of the partition storage device, and affect the read and write performance. In order to solve the above technical problems, in this embodiment, it is not necessary to query the corresponding metadata in the memory for the reverse index information of each data segment of the source partition to determine whether it is a valid data segment, but directly determine the valid data segments in the source partition in batches according to the file object metadata cached in the memory, thereby reducing the query volume, avoiding the impact on the metadata, and avoiding the increase in the pressure of the partition storage device during the query of the valid data segment.

[0043] Specifically, metadata of each file object is cached in memory, such as Figure 3 As shown in, that is, Inode (index node) information, where Inode is a data structure in the file system, used to store the metadata of the file object. The Inode information includes BaseMeta (basic metadata) and InodeSpace (Inode space), where BaseMeta records the basic attributes of Inode, such as inodeid (Inode identifier), ctime (metadata modification time), atime (last access time), mtime (last modification time), length (file length), physical_length (the actual space occupied by the file on the disk), etc.; and InodeSpace is a key-value pair (key-value) structure, which mainly records the location of a logical range of an Inode on the partition storage device, that is, the mapping relationship between each data segment of the file object and the identifier of the partition in the partition storage device.

[0044] Further, according to the identification of the source partition, the data segment corresponding to the identification of the source partition in each data segment of the file object can be queried from the file object metadata cached in the memory as the valid data segment in the source partition. For example, as shown in the figure, for any file object metadata in the memory, different block data blocks are recorded in the InodeSpace, each block data block includes multiple data segments, and the multiple data segments may be stored in one or more partitions, and the identification of the partition where the multiple data segments are located is recorded through key-value pairs. It is only necessary to query the key-value pair according to the identification of the source partition. As long as the identification of the source partition is included in the query Value, the corresponding data block can be determined to be valid data, and all valid data can be queried based on one query. It should be noted that if different file object metadata, that is, different Inode information, are cached in the memory, the data segment corresponding to the identification of the source partition can be queried in the different file object metadata, so that the valid data segments belonging to different file objects in the source partition can be finally determined.

[0045] S202. Read each valid data segment from the source partition, write the read valid data segment into a cache pool corresponding to the file object in the memory according to the file object to which the valid data segment belongs, and aggregate the valid data segments in the same cache pool into a valid aggregated data segment.

[0046] In this embodiment, after determining all valid data segments in the source partition, each valid data segment needs to be migrated to the target partition, that is, each valid data segment is read from the source partition and then written to the target partition. However, considering that the valid data segments may not be continuous in the source partition and each valid data segment is large or small, directly writing each valid data segment to the target partition will generate a large number of IOPS (input / output operations per second), which has a large impact on the partition storage device. In order to solve the above problem, in this embodiment, a cache pool can be created in the memory, and each file object creates a cache pool. The capacity of each cache pool can be set according to actual needs. Further, the read valid data can be stored in the corresponding cache pool according to the file object to which it belongs, and the valid data segments in the same cache pool are aggregated into a whole segment of valid data, which is recorded as a valid aggregated data segment. Then, when writing to the target partition later, only a whole segment of valid data aggregated data segment needs to be written to the target partition, which reduces IOPS and reduces the impact on the partition storage device. And the valid data segments of the same file object are made as continuous as possible, providing a larger bandwidth and reasonable delay for subsequent reading.

[0047] Optionally, when reading each valid data segment from the source partition, if each valid data segment is directly read, a large number of IOPS will also be generated, which has a great impact on the partition storage device. Therefore, the following improvements are made to the reading process in this embodiment:

[0048] If the data volume of any valid data segment in the source partition exceeds a first data volume threshold, directly reading the any valid data segment from the source partition; or

[0049] According to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segment between the adjacent valid data segments, at least two adjacent valid data segments and the invalid data segment between the at least two valid data segments are determined as an aggregate data segment, and the aggregate data segment is read from the source partition at one time.

[0050] In this embodiment, the data volume of any valid data segment in the source partition can be determined. If it is determined that the data volume of any valid data segment in the source partition exceeds the first data volume threshold, it can be determined that the valid data segment is a large block of data segments, and the valid data segment can be directly read from the source partition. Figure 1 The valid data segment marked as 5 in the source partition; for small valid data blocks, adjacent valid data blocks can be aggregated, and a small number of invalid data blocks can be allowed to be mixed in, that is, a large data segment is read in a greedy manner, for example Figure 1 The large data segments marked as 1 and 4 in the source partition include valid data segments and a small number of invalid data segments (white). Specifically, according to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segments between the adjacent valid data segments, it can be determined which adjacent data segments (including valid data segments and invalid data segments) can be read as a large data segment (represented as an aggregated data segment here, but in fact the data segments included therein are originally connected and there is no real aggregation process).

[0051] Optionally, when determining the aggregated data segment, some conditions need to be met to make the benefit of reading the aggregated data segment greater than reading the valid data segment individually. For example, the data volume of the invalid data segment between two adjacent valid data segments cannot be too large (e.g., it needs to be less than the second data volume threshold), and the total number of invalid data segments in the aggregated data segment cannot account for too large a proportion (e.g., less than the first preset proportion). In addition, the total data volume of the aggregated data segment can also be limited.

[0052] In specific implementation, when it is necessary to determine the aggregated data segment, it can be achieved through traversal. After a valid data segment (for example, a relatively small valid data segment, whose data volume is less than the fourth data volume threshold) is added to the aggregated data segment, it can be determined whether the data volume of the adjacent invalid data segment after the valid data segment is less than the second data volume threshold, and whether the proportion of the total data volume of the invalid data segment is less than the first preset ratio when the adjacent invalid data segment and the subsequent adjacent valid data segment are added to the aggregated data segment. If both conditions are less than, the aggregated data segment can be added, and then the subsequent data segments can be traversed.

[0053] During the subsequent traversal process, after any first valid data segment is added to the aggregate data segment, if it is determined that the data amount of the first adjacent invalid data segment after the first valid data segment is less than the second data amount threshold, and when the first adjacent invalid data segment and the first adjacent valid data segment after the first valid data segment are added to the aggregate data segment, the proportion of the total data amount of the invalid data segments is less than the first preset ratio, then the first adjacent invalid data segment and the first adjacent valid data segment after the first valid data segment are added to the aggregate data segment.

[0054] And as Figure 1 The valid data marked as 2 and 3 are read separately because they cannot meet the above conditions.

[0055] It should be noted that various data volume thresholds and preset ratios in this embodiment can be set according to actual needs and are not limited here.

[0056] Furthermore, in the case of a one-time reading of aggregated data segments from a source partition, since there are valid data segments and invalid data segments in the aggregated data segments, it is necessary to obtain the valid data segments included in the aggregated data segments, and then write them into the cache pool corresponding to the corresponding file object in the memory respectively, so as to realize aggregation during reading and splitting before writing to the cache pool, thereby ensuring that the invalid data segments are screened out at this time.

[0057] In addition, when reading the valid data segment, the consistency of the data can also be verified, for example, the check code (such as CRC or MD5 code) of the valid data segment is calculated and compared with the check code stored in the metadata. If the comparison is consistent, it means that the valid data segment is consistent with the memory.

[0058] S203: Determine a target cache pool to be written into the partition storage device from all cache pools, and write valid aggregated data segments in the target cache pool into a target partition in the partition storage device based on a sequential write mechanism.

[0059] In this embodiment, considering that the storage space of the cache pool in the memory is not infinite, it is necessary to write the valid aggregated data segments in the cache pool to the target partition after meeting specific conditions, so as to realize the migration of the valid data segments from the source partition through the cache pool to the target partition, while also ensuring that the cache pool can release cache space so that there is space to continue caching the valid data blocks read from the source partition.

[0060] Among them, when determining the target cache pool to be written to the partition storage device, it can be determined based on preset rules. Specifically, when it is determined that the amount of data in any cache pool exceeds the third data amount threshold (for example, 128Kb), it means that the valid aggregated data segment in the cache pool is large enough, and the cache pool can be determined as the target cache pool to be written to the partition storage device; or, if it is determined that the total amount of data in all cache pools exceeds the second preset proportion of the total capacity of all cache pools (for example, 80%), all cache pools are determined as the target cache pools to be written to the partition storage device, ensuring that the target cache pool can release cache space to ensure that there is space to store the valid data blocks read subsequently.

[0061] In addition, since sequential writing is required in each partition of the partitioned storage device, the valid aggregate data segments to be written to the target partition are written to the target partition based on a sequential writing mechanism. Specifically, the current pointer position in the target partition can be determined, and the valid aggregate data segments are written starting from the pointer position.

[0062] S204: Reclaim the storage space of the source partition after all valid data segments in the source partition have been read.

[0063] In this embodiment, the storage space of the source partition needs to be reclaimed in the end, and the timing of the recycling can be any time after all valid data segments in the source partition have been read, of course, it can be after all valid data segments have been written to the target partition, or it can be any other time, and finally the garbage collection of the source partition is completed.

[0064] The garbage collection method for a partitioned storage device of the present embodiment determines valid data segments in a source partition to be recycled in the partitioned storage device based on metadata of file objects cached in a memory, wherein the file objects corresponding to the file object metadata are stored in the partitioned storage device; reads each valid data segment from the source partition, and writes the read valid data segment into a cache pool corresponding to the file object in the memory based on the file object to which the valid data segment belongs, and aggregates the valid data segments in the same cache pool into a valid aggregated data segment; determines a target cache pool to be written into the partitioned storage device from all cache pools, writes the valid aggregated data segments in the target cache pool into a target partition in the partitioned storage device based on a sequential write mechanism, and recycles the storage space of the source partition after all valid data segments in the source partition have been read. In this embodiment, by batch querying the valid data segments in the source partition from the file object metadata cached in the memory, the metadata query can be reduced; and after reading each valid data segment in the source partition, the data segments are aggregated in the cache pool of the memory according to the file objects to which they belong and then written to the target partition, which can effectively reduce the number of data writes (IOPS) and avoid impact on the partition storage device, thereby ensuring the normal read and write performance of the partition storage device, and the aggregation of valid data belonging to the same file object can provide a larger bandwidth and reasonable delay for subsequent reading.

[0065] Based on any of the above embodiments, the partition storage device in this embodiment can be used as follows: Figure 3 As shown in the data layout, each partition of the partition storage device includes multiple minimum write units (sectors, whose size is generally 4Kb) and a zone footer, wherein each minimum write unit may include a data portion and a tail subunit (sector footer), and the tail subunit (sector footer) is used to store the internal metadata of the minimum write unit, such as the file object to which the minimum write unit belongs, and the position information of the minimum write unit in the file object to which it belongs (for example, the sector of the file object to which it belongs, etc.); and the file object to which it belongs is used to store the reverse index information of the current partition, such as which range of which file object the current partition belongs to, and where the record is in the current partition, which can be used for data segment relocation during garbage collection.

[0066] Based on the above embodiment, when writing the valid aggregated data segments in the target cache pool into the target partition in the partition storage device based on the sequential writing mechanism in S203, specifically as follows Figure 4 As shown, it may include:

[0067] S2031, determining a target data volume to be written to a target partition in the valid aggregate data segments in the target cache pool according to the data volume of the minimum write unit of the partition storage device, the data volume of the metadata inside the minimum write unit, and the data volume of the valid aggregate data segments in the target cache pool;

[0068] S2032. Obtain data segments of the target data volume from valid aggregated data segments in the target cache pool as data segments to be written, and write the data segments to be written into the target partition in the partition storage device based on a sequential writing mechanism.

[0069] In this embodiment, since the partition storage device writes in the minimum write unit (4Kb), and the metadata inside each minimum write unit also occupies the control of the minimum write unit, for example, occupies 16 bytes; therefore, the data that can actually be written by a minimum write unit is 4Kb-16Byte. On this basis, if the data segment to be written is still written in a multiple of the data amount of the minimum write unit, some free space will inevitably be generated in the last small write unit, resulting in a waste of storage space. Therefore, in order to maximize the use of storage space and avoid waste of storage space, in this embodiment, when writing the data segment to be written to the target partition, the complete minimum write unit is occupied as much as possible, so that there is no remaining space in each occupied minimum write unit. In order to achieve this purpose, in this embodiment, according to the data amount of the minimum write unit, the data amount of the metadata inside the minimum write unit, and the data amount of the effective aggregated data segment in the target cache pool, the target data amount to be written to the target partition in the effective aggregated data segment in the target cache pool can be determined, and then the data segment of the target data amount is obtained from the effective aggregated data segment as the data segment to be written, and the data segment to be written is written to the target partition.

[0070] Optionally, when determining the target data volume, the target number of required minimum write units can be determined based on the data volume of valid aggregate data segments in the target cache pool and the data volume of minimum write units. For example, the data volume of valid aggregate data segments can be divided by the data volume of minimum write units and rounded to obtain the target number of minimum write units. Of course, the target number can also be determined by other algorithms. Further, a first product of the data volume of the minimum write unit and the target number, and a second product of the data volume of the metadata inside the minimum write unit and the target number are determined, and the difference between the first product and the second product is determined as the target data volume. For example, assuming that the target number of minimum write units is 10, the data volume of the minimum write unit is 4Kb, and the data volume of the metadata inside the minimum write unit is 16Byte, then the target data volume is 4Kb×10-16Byte×10=40Kb-160Byte. In this way, after the data segments to be written of the target data volume are written to the target partition, they only occupy the complete 10 minimum write units, and there is no remaining space in each occupied minimum write unit.

[0071] Of course, determining the target data volume to be written to the target partition in the valid aggregate data segments in the target cache pool based on the data volume of the minimum write unit, the data volume of the metadata inside the minimum write unit, and the data volume of the valid aggregate data segments in the target cache pool is not limited to the above method, and other methods can also be used, which are not limited in this embodiment.

[0072] Furthermore, when writing valid aggregate data segments (data segments to be written) into the target partition, the valid aggregate data segments can be sequentially written into each minimum write unit of the target partition, and internal metadata can be recorded in each minimum write unit. The internal metadata is rounded to record information including but not limited to the file object to which the current minimum write unit belongs and its location in the file object, thereby completing the writing.

[0073] In addition, in another optional embodiment, a processing method is provided when a user writes a data segment of a file object to any partition. Specifically, when used to write any current data segment to the partition, it is determined whether the current data segment and the adjacent previous data segment belong to the same file object. If they belong to the same file object, the metadata of the current data segment and the previous data segment are merged. For example, the current data segment and the previous data segment are taken as a whole data segment. In this way, only one metadata is required, especially the starting address, end address, data segment length, file object identifier, etc., which only need to be recorded once to reduce the amount of metadata occupied, thereby achieving full memory metadata and improving read latency.

[0074] Corresponding to the garbage collection method of the partition storage device in the above embodiment, Figure 5A block diagram of a garbage collection device for a partition storage device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 5 The garbage collection device 500 of the partition storage device includes: a valid data determination unit 501, a reading unit 502 and a writing unit 503.

[0075] The valid data determination unit 501 is used to determine the valid data segment in the source partition to be recovered in the partition storage device according to the file object metadata cached in the memory, wherein the file object corresponding to the file object metadata is stored in the partition storage device;

[0076] The reading unit 502 is used to read each valid data segment from the source partition, write the read valid data segment into a cache pool corresponding to the file object in the memory according to the file object to which the valid data segment belongs, and aggregate the valid data segments in the same cache pool into a valid aggregated data segment;

[0077] The writing unit 503 is used to determine the target cache pool to be written to the partition storage device from all cache pools, write the valid aggregated data segments in the target cache pool into the target partition in the partition storage device based on a sequential writing mechanism, and reclaim the storage space of the source partition after all valid data segments in the source partition have been read.

[0078] In one or more embodiments of the present disclosure, the file object metadata includes a mapping relationship between each data segment of the file object and an identifier of the partition in which the data segment is located in the partition storage device;

[0079] Accordingly, when determining the valid data segments in the source partition to be reclaimed in the partition storage device according to the file object metadata cached in the memory, the valid data determining unit 501 is used to:

[0080] According to the identifier of the source partition, a data segment corresponding to the identifier of the source partition in each data segment of the file object is queried from the file object metadata cached in the memory as a valid data segment in the source partition.

[0081] In one or more embodiments of the present disclosure, when reading each valid data segment from the source partition, the reading unit 502 is used to:

[0082] If the data volume of any valid data segment in the source partition exceeds a first data volume threshold, directly reading the any valid data segment from the source partition; or

[0083] According to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segment between the adjacent valid data segments, at least two adjacent valid data segments and the invalid data segment between the at least two valid data segments are determined as an aggregate data segment, and the aggregate data segment is read from the source partition at one time.

[0084] In one or more embodiments of the present disclosure, when determining at least two adjacent valid data segments and the invalid data segment between the at least two valid data segments as one aggregated data segment according to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segment between the adjacent valid data segments, the reading unit 502 is configured to:

[0085] After any first valid data segment is added to the aggregate data segment, if it is determined that the data amount of the first adjacent invalid data segment after the first valid data segment is less than the second data amount threshold, and when the first adjacent invalid data segment and the first adjacent valid data segment after the first valid data segment are added to the aggregate data segment, the proportion of the total data amount of the invalid data segments is less than the first preset ratio, then the first adjacent invalid data segment and the first adjacent valid data segment after the first valid data segment are added to the aggregate data segment.

[0086] In one or more embodiments of the present disclosure, when writing the read valid data segment into the cache pool corresponding to the corresponding file object in the memory, the reading unit 502 is used to:

[0087] In the case of reading the aggregated data segment from the source partition at one time, valid data segments are obtained from the aggregated data segment and are written into cache pools corresponding to the corresponding file objects in the memory respectively.

[0088] In one or more embodiments of the present disclosure, when determining the target cache pool to be written to the partition storage device from all cache pools, the writing unit 503 is used to:

[0089] If it is determined that the amount of data in any buffer pool exceeds a third data amount threshold, the any buffer pool is determined as a target buffer pool to be written into the partition storage device; or

[0090] If it is determined that the total amount of data in all the buffer pools exceeds the second preset ratio of the total capacity of all the buffer pools, all the buffer pools are determined as target buffer pools to be written into the partition storage device.

[0091] In one or more embodiments of the present disclosure, when the writing unit 503 writes the valid aggregated data segments in the target cache pool into the target partition in the partition storage device based on the sequential writing mechanism, it is configured to:

[0092] Determine the target data volume to be written to the target partition in the valid aggregate data segments in the target cache pool according to the data volume of the minimum write unit 503 of the partition storage device, the data volume of the metadata inside the minimum write unit 503, and the data volume of the valid aggregate data segments in the target cache pool;

[0093] The data segments of the target data volume are obtained from the valid aggregated data segments in the target cache pool as the data segments to be written, and the data segments to be written are written into the target partition in the partition storage device based on a sequential writing mechanism.

[0094] In one or more embodiments of the present disclosure, when determining the target data amount to be written to the target partition in the valid aggregate data segments in the target cache pool according to the data amount of the minimum write unit 503 of the partition storage device, the data amount of the metadata inside the minimum write unit 503, and the data amount of the valid aggregate data segments in the target cache pool, the write unit 503 is used to:

[0095] Determine the target number of required minimum write units 503 according to the data volume of the valid aggregated data segments in the target cache pool and the data volume of the minimum write unit 503;

[0096] Determine a first product of the data volume of the minimum write unit 503 and the target number, and a second product of the data volume of the metadata inside the minimum write unit 503 and the target number, and determine the difference between the first product and the second product as the target data volume, so that there is no remaining space in each minimum write unit 503 occupied after the data segment to be written of the target data volume is written to the target partition.

[0097] In one or more embodiments of the present disclosure, when the writing unit 503 writes the valid aggregated data segments in the target cache pool into the target partition in the partition storage device based on the sequential writing mechanism, it is configured to:

[0098] The valid aggregate data segments are sequentially written into each minimum write unit 503 of the target partition, and internal metadata is recorded in each minimum write unit 503, wherein the internal metadata is used to record the file object to which the current minimum write unit 503 belongs and its location information in the file object.

[0099] In one or more embodiments of the present disclosure, the writing unit 503 is further configured to:

[0100] When writing any current data segment to any partition of the partition storage device, determine whether the current data segment and the adjacent previous data segment in the partition belong to the same file object. If they belong to the same file object, merge the metadata of the current data segment with that of the previous data segment.

[0101] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0102] refer to Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure, and the electronic device 600 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0103] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0104] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0105] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0106] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0107] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0108] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0109] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0110] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0112] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] In a first aspect, according to one or more embodiments of the present disclosure, a garbage collection method for a partitioned storage device is provided, comprising:

[0115] Determining valid data segments in a source partition to be reclaimed in a partition storage device according to file object metadata cached in the memory, wherein the file object corresponding to the file object metadata is stored in the partition storage device;

[0116] Reading each valid data segment from the source partition, writing the read valid data segment into a cache pool corresponding to the file object in the memory according to the file object to which the valid data segment belongs, and aggregating the valid data segments in the same cache pool into a valid aggregated data segment;

[0117] Determine a target cache pool to be written to the partition storage device from all cache pools, write valid aggregated data segments in the target cache pool to the target partition in the partition storage device based on a sequential write mechanism, and reclaim the storage space of the source partition after all valid data segments in the source partition have been read.

[0118] According to one or more embodiments of the present disclosure, the file object metadata includes a mapping relationship between each data segment of the file object and an identifier of the partition in which the data segment is located in the partition storage device;

[0119] Correspondingly, determining the valid data segments in the source partition to be recycled in the partition storage device according to the file object metadata cached in the memory includes:

[0120] According to the identifier of the source partition, a data segment corresponding to the identifier of the source partition in each data segment of the file object is queried from the file object metadata cached in the memory as a valid data segment in the source partition.

[0121] According to one or more embodiments of the present disclosure, reading each valid data segment from the source partition includes:

[0122] If the data volume of any valid data segment in the source partition exceeds a first data volume threshold, directly reading the any valid data segment from the source partition; or

[0123] According to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segment between the adjacent valid data segments, at least two adjacent valid data segments and the invalid data segment between the at least two valid data segments are determined as an aggregate data segment, and the aggregate data segment is read from the source partition at one time.

[0124] According to one or more embodiments of the present disclosure, determining at least two adjacent valid data segments and the invalid data segment between the at least two valid data segments as an aggregate data segment according to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segment between the adjacent valid data segments includes:

[0125] After any first valid data segment is added to the aggregate data segment, if it is determined that the data amount of the first adjacent invalid data segment after the first valid data segment is less than the second data amount threshold, and when the first adjacent invalid data segment and the first adjacent valid data segment after the first valid data segment are added to the aggregate data segment, the proportion of the total data amount of the invalid data segments is less than the first preset ratio, then the first adjacent invalid data segment and the first adjacent valid data segment after the first valid data segment are added to the aggregate data segment.

[0126] According to one or more embodiments of the present disclosure, writing the read valid data segment into a cache pool corresponding to the corresponding file object in the memory includes:

[0127] In the case of reading the aggregated data segment from the source partition at one time, valid data segments are obtained from the aggregated data segment and are written into cache pools corresponding to the corresponding file objects in the memory respectively.

[0128] According to one or more embodiments of the present disclosure, determining a target cache pool to be written to a partition storage device from all cache pools includes:

[0129] If it is determined that the amount of data in any buffer pool exceeds a third data amount threshold, the any buffer pool is determined as a target buffer pool to be written into the partition storage device; or

[0130] If it is determined that the total amount of data in all the buffer pools exceeds the second preset ratio of the total capacity of all the buffer pools, all the buffer pools are determined as target buffer pools to be written into the partition storage device.

[0131] According to one or more embodiments of the present disclosure, writing the valid aggregated data segments in the target cache pool into the target partition in the partition storage device based on a sequential write mechanism includes:

[0132] Determine the target data volume to be written to the target partition in the valid aggregate data segments in the target cache pool according to the data volume of the minimum write unit of the partition storage device, the data volume of the metadata inside the minimum write unit, and the data volume of the valid aggregate data segments in the target cache pool;

[0133] The data segments of the target data volume are obtained from the valid aggregated data segments in the target cache pool as the data segments to be written, and the data segments to be written are written into the target partition in the partition storage device based on a sequential writing mechanism.

[0134] According to one or more embodiments of the present disclosure, determining the target data amount to be written to the target partition in the valid aggregate data segments in the target cache pool according to the data amount of the minimum write unit of the partition storage device, the data amount of metadata inside the minimum write unit, and the data amount of the valid aggregate data segments in the target cache pool includes:

[0135] Determine the target number of required minimum write units according to the data amount of the valid aggregated data segments in the target cache pool and the data amount of the minimum write unit;

[0136] Determine a first product of the data amount of the minimum write unit and the target number, and a second product of the data amount of the metadata inside the minimum write unit and the target number, and determine the difference between the first product and the second product as the target data amount, so that there is no remaining space in each minimum write unit occupied after the data segment to be written of the target data amount is written to the target partition.

[0137] According to one or more embodiments of the present disclosure, writing the valid aggregated data segments in the target cache pool into the target partition in the partition storage device based on a sequential write mechanism includes:

[0138] The valid aggregate data segments are sequentially written into each minimum write unit of the target partition, and internal metadata is recorded in each minimum write unit, where the internal metadata is used to record the file object to which the current minimum write unit belongs and its location information in the file object.

[0139] According to one or more embodiments of the present disclosure, the method further includes:

[0140] When writing any current data segment to any partition of the partition storage device, determine whether the current data segment and the adjacent previous data segment in the partition belong to the same file object. If they belong to the same file object, merge the metadata of the current data segment with that of the previous data segment.

[0141] In a second aspect, according to one or more embodiments of the present disclosure, a garbage collection device for a partitioned storage device is provided, comprising:

[0142] A valid data determination unit, configured to determine a valid data segment in a source partition to be recovered in a partition storage device according to file object metadata cached in the memory, wherein the file object corresponding to the file object metadata is stored in the partition storage device;

[0143] a reading unit, configured to read each valid data segment from the source partition, write the read valid data segment into a cache pool corresponding to the file object in the memory according to the file object to which the valid data segment belongs, and aggregate the valid data segments in the same cache pool into one valid aggregated data segment;

[0144] A writing unit is used to determine a target cache pool to be written to a partition storage device from all cache pools, write valid aggregated data segments in the target cache pool into a target partition in the partition storage device based on a sequential writing mechanism, and reclaim the storage space of the source partition after all valid data segments in the source partition have been read.

[0145] According to one or more embodiments of the present disclosure, the file object metadata includes a mapping relationship between each data segment of the file object and an identifier of the partition in which the data segment is located in the partition storage device;

[0146] Accordingly, when determining the valid data segments in the source partition to be reclaimed in the partition storage device according to the file object metadata cached in the memory, the valid data determination unit is used to:

[0147] According to the identifier of the source partition, a data segment corresponding to the identifier of the source partition in each data segment of the file object is queried from the file object metadata cached in the memory as a valid data segment in the source partition.

[0148] According to one or more embodiments of the present disclosure, when reading each valid data segment from the source partition, the reading unit is used to:

[0149] If the data volume of any valid data segment in the source partition exceeds a first data volume threshold, directly reading the any valid data segment from the source partition; or

[0150] According to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segment between the adjacent valid data segments, at least two adjacent valid data segments and the invalid data segment between the at least two valid data segments are determined as an aggregate data segment, and the aggregate data segment is read from the source partition at one time.

[0151] According to one or more embodiments of the present disclosure, when determining at least two adjacent valid data segments and the invalid data segment between the at least two valid data segments as one aggregated data segment according to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segment between the adjacent valid data segments, the reading unit is configured to:

[0152] After any first valid data segment is added to the aggregate data segment, if it is determined that the data amount of the first adjacent invalid data segment after the first valid data segment is less than the second data amount threshold, and when the first adjacent invalid data segment and the first adjacent valid data segment after the first valid data segment are added to the aggregate data segment, the proportion of the total data amount of the invalid data segments is less than the first preset ratio, then the first adjacent invalid data segment and the first adjacent valid data segment after the first valid data segment are added to the aggregate data segment.

[0153] According to one or more embodiments of the present disclosure, when the reading unit writes the read valid data segment into the cache pool corresponding to the corresponding file object in the memory, it is used to:

[0154] In the case of reading the aggregated data segment from the source partition at one time, valid data segments are obtained from the aggregated data segment and are written into cache pools corresponding to the corresponding file objects in the memory respectively.

[0155] According to one or more embodiments of the present disclosure, when determining a target cache pool to be written to a partition storage device from all cache pools, the writing unit is configured to:

[0156] If it is determined that the amount of data in any buffer pool exceeds a third data amount threshold, the any buffer pool is determined as a target buffer pool to be written into the partition storage device; or

[0157] If it is determined that the total amount of data in all the buffer pools exceeds the second preset ratio of the total capacity of all the buffer pools, all the buffer pools are determined as target buffer pools to be written into the partition storage device.

[0158] According to one or more embodiments of the present disclosure, when the writing unit writes the valid aggregate data segments in the target cache pool into the target partition in the partition storage device based on a sequential writing mechanism, it is configured to:

[0159] Determine the target data volume to be written to the target partition in the valid aggregate data segments in the target cache pool according to the data volume of the minimum write unit of the partition storage device, the data volume of the metadata inside the minimum write unit, and the data volume of the valid aggregate data segments in the target cache pool;

[0160] The data segments of the target data volume are obtained from the valid aggregated data segments in the target cache pool as the data segments to be written, and the data segments to be written are written into the target partition in the partition storage device based on a sequential writing mechanism.

[0161] According to one or more embodiments of the present disclosure, when determining the target data amount to be written to the target partition in the valid aggregate data segments in the target cache pool according to the data amount of the minimum write unit of the partition storage device, the data amount of the metadata inside the minimum write unit, and the data amount of the valid aggregate data segments in the target cache pool, the write unit is used to:

[0162] Determine the target number of required minimum write units according to the data volume of the valid aggregated data segments in the target cache pool and the data volume of the minimum write unit;

[0163] Determine a first product of the data amount of the minimum write unit and the target number, and a second product of the data amount of the metadata inside the minimum write unit and the target number, and determine the difference between the first product and the second product as the target data amount, so that there is no remaining space in each minimum write unit occupied after the data segment to be written of the target data amount is written to the target partition.

[0164] According to one or more embodiments of the present disclosure, when the writing unit writes the valid aggregate data segments in the target cache pool into the target partition in the partition storage device based on a sequential writing mechanism, it is configured to:

[0165] The valid aggregate data segments are sequentially written into each minimum write unit of the target partition, and internal metadata is recorded in each minimum write unit, where the internal metadata is used to record the file object to which the current minimum write unit belongs and its location information in the file object.

[0166] According to one or more embodiments of the present disclosure, the writing unit is further used for:

[0167] When writing any current data segment to any partition of the partition storage device, determine whether the current data segment and the adjacent previous data segment in the partition belong to the same file object. If they belong to the same file object, merge the metadata of the current data segment with that of the previous data segment.

[0168] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0169] The memory stores computer-executable instructions;

[0170] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the garbage collection method for the partitioned storage device as described in the first aspect and various possible designs of the first aspect.

[0171] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the garbage collection method for the partitioned storage device as described in the first aspect and various possible designs of the first aspect is implemented.

[0172] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the garbage collection method for the partitioned storage device as described in the first aspect and various possible designs of the first aspect is implemented.

[0173] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0174] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0175] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A garbage collection method for a partitioned storage device, characterized in that: include: Determining valid data segments in a source partition to be reclaimed in a partition storage device according to file object metadata cached in the memory, wherein the file object corresponding to the file object metadata is stored in the partition storage device; Reading each valid data segment from the source partition, writing the read valid data segment into a cache pool corresponding to the file object in the memory according to the file object to which the valid data segment belongs, and aggregating the valid data segments in the same cache pool into a valid aggregated data segment; Determine a target cache pool to be written into the partition storage device from all cache pools, write valid aggregated data segments in the target cache pool into the target partition in the partition storage device based on a sequential write mechanism, and reclaim the storage space of the source partition after all valid data segments in the source partition have been read; The reading each valid data segment from the source partition comprises: If the data volume of any valid data segment in the source partition exceeds a first data volume threshold, directly reading the any valid data segment from the source partition; or According to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segment between the adjacent valid data segments, at least two adjacent valid data segments and the invalid data segment between the at least two valid data segments are determined as an aggregate data segment, and the aggregate data segment is read from the source partition at one time.

2. The method according to claim 1, characterized in that The file object metadata includes a mapping relationship between each data segment of the file object and an identifier of the partition in which the data segment is located in the partition storage device; Correspondingly, determining the valid data segments in the source partition to be recycled in the partition storage device according to the file object metadata cached in the memory includes: According to the identifier of the source partition, a data segment corresponding to the identifier of the source partition in each data segment of the file object is queried from the file object metadata cached in the memory as a valid data segment in the source partition.

3. The method according to claim 1, characterized in that The step of determining, according to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segment between the adjacent valid data segments, at least two adjacent valid data segments and the invalid data segment between the at least two valid data segments as an aggregated data segment comprises: After any first valid data segment is added to the aggregate data segment, if it is determined that the data amount of the first adjacent invalid data segment after the first valid data segment is less than the second data amount threshold, and when the first adjacent invalid data segment and the first adjacent valid data segment after the first valid data segment are added to the aggregate data segment, the proportion of the total data amount of the invalid data segments is less than the first preset ratio, then the first adjacent invalid data segment and the first adjacent valid data segment after the first valid data segment are added to the aggregate data segment.

4. The method according to claim 1, characterized in that: The step of writing the read valid data segment into a cache pool corresponding to the file object in the memory includes: In the case of reading the aggregated data segment from the source partition at one time, valid data segments are obtained from the aggregated data segment and are written into cache pools corresponding to the corresponding file objects in the memory respectively.

5. The method according to claim 1, characterized in that The step of determining a target cache pool to be written into the partition storage device from all cache pools includes: If it is determined that the amount of data in any buffer pool exceeds a third data amount threshold, the any buffer pool is determined as a target buffer pool to be written into the partition storage device; or If it is determined that the total amount of data in all the buffer pools exceeds the second preset ratio of the total capacity of all the buffer pools, all the buffer pools are determined as target buffer pools to be written into the partition storage device.

6. The method according to claim 1, characterized in that The step of writing the valid aggregated data segments in the target cache pool into the target partition in the partition storage device based on a sequential write mechanism includes: Determine the target data volume to be written to the target partition in the valid aggregate data segments in the target cache pool according to the data volume of the minimum write unit of the partition storage device, the data volume of the metadata inside the minimum write unit, and the data volume of the valid aggregate data segments in the target cache pool; The data segments of the target data volume are obtained from the valid aggregated data segments in the target cache pool as the data segments to be written, and the data segments to be written are written into the target partition in the partition storage device based on a sequential writing mechanism.

7. The method according to claim 6, characterized in that The step of determining the target data amount to be written to the target partition in the valid aggregate data segments in the target cache pool according to the data amount of the minimum write unit of the partition storage device, the data amount of metadata inside the minimum write unit, and the data amount of the valid aggregate data segments in the target cache pool comprises: Determine the target number of required minimum write units according to the data volume of the valid aggregated data segments in the target cache pool and the data volume of the minimum write unit; Determine a first product of the data amount of the minimum write unit and the target number, and a second product of the data amount of the metadata inside the minimum write unit and the target number, and determine the difference between the first product and the second product as the target data amount, so that there is no remaining space in each minimum write unit occupied after the data segment to be written of the target data amount is written to the target partition.

8. The method according to any one of claims 1 to 7, characterized in that: The step of writing the valid aggregated data segments in the target cache pool into the target partition in the partition storage device based on a sequential write mechanism includes: The valid aggregate data segments are sequentially written into each minimum write unit of the target partition, and internal metadata is recorded in each minimum write unit, where the internal metadata is used to record the file object to which the current minimum write unit belongs and its location information in the file object.

9. The method according to claim 1, characterized in that: The method further comprises: When writing any current data segment to any partition of the partition storage device, determine whether the current data segment and the adjacent previous data segment in the partition belong to the same file object. If they belong to the same file object, merge the metadata of the current data segment with that of the previous data segment.

10. A garbage collection device for a partitioned storage device, characterized in that: include: A valid data determination unit, configured to determine a valid data segment in a source partition to be recovered in a partition storage device according to file object metadata cached in the memory, wherein the file object corresponding to the file object metadata is stored in the partition storage device; a reading unit, configured to read each valid data segment from the source partition, write the read valid data segment into a cache pool corresponding to the file object in the memory according to the file object to which the valid data segment belongs, and aggregate the valid data segments in the same cache pool into one valid aggregated data segment; A writing unit, used to determine a target cache pool to be written into the partition storage device from all cache pools, write valid aggregated data segments in the target cache pool into a target partition in the partition storage device based on a sequential writing mechanism, and reclaim the storage space of the source partition after all valid data segments in the source partition have been read; When reading each valid data segment from the source partition, the reading unit is used to: If the data volume of any valid data segment in the source partition exceeds a first data volume threshold, directly reading the any valid data segment from the source partition; or According to the data amount of each adjacent valid data segment in the source partition and the data amount of the invalid data segment between the adjacent valid data segments, at least two adjacent valid data segments and the invalid data segment between the at least two valid data segments are determined as an aggregate data segment, and the aggregate data segment is read from the source partition at one time.

11. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

Images