Data storage garbage collection method, data writing method, device and data storage system

Abstract

The application relates to the technical field of data storage, and discloses a storage data garbage collection method, a write data method, a device and a storage system. The garbage collection method comprises the following steps: in the case that garbage collection has been started, if an overwrite operation of a target storage address of a target disk space is received, newly written first data is stored in a second chunk; if a first storage address in the first chunk is associated with the target storage address, a first version flag of the first data is set, a second storage address of the first data in the second chunk is associated with the target storage address; the first valid data in the first chunk and the corresponding version flag are migrated to a third chunk, and a third storage address of the first valid data in the third chunk is associated with the storage address on the target disk space; and the data stored in the first chunk is deleted. The application can improve the read-write performance of the data storage system.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, specifically to a method for garbage collection of stored data, a method for writing data, an apparatus, and a storage system. Background Technology

[0002] With the rapid development of information technology and the improvement of storage hardware technology, the amount of data generated and used by humankind has increased exponentially since entering the information society. Whether it's massive amounts of short videos, ubiquitous sensors and cameras, hundreds of millions of online orders, or map data with meter-level accuracy, vast amounts of data are constantly being generated. This data is crucial to people, and its storage and computation determine the efficiency and development of various industries. To better store and compute this data, more flexible and efficient data storage, processing, and analysis tools need to be designed. The arrival of the big data era will inevitably bring about improvements or fundamental changes to technologies and applications such as cloud computing, data warehousing, and data mining.

[0003] However, big data presents a significant challenge to data write performance, requiring the storage of massive amounts of continuously generated data such as videos, images, and logs. Therefore, modern storage systems typically store data using an append-only approach at the engine layer. This approach eliminates address lookups during writes, instead continuously appending data as a log, thus improving write efficiency. However, append-only writes also encounter problems. To achieve overwrite semantics, each overwrite generates redundant data for the appender. Garbage collection of this overwritten data has become a new concern for storage systems.

[0004] Like solid-state drives (SSDs), garbage collection can cause a significant performance drop, largely because users are writing to the same areas while data is being migrated, requiring locking to ensure the order of write transactions. Extending this to distributed storage systems, the scope of these locks becomes even larger, resulting in a more substantial impact on performance. Summary of the Invention

[0005] In view of this, the present invention provides a data storage garbage collection method, a data writing method, an apparatus, and a storage system to solve the problem that data storage systems experience a significant performance degradation when performing garbage collection.

[0006] In a first aspect, the present invention provides a method for garbage collection of stored data, applied to a data storage system, the method comprising:

[0007] If garbage collection for the first chunk has been initiated, and an overwrite write operation is received for the target storage address on the target disk space, the newly written first data will be stored in the second chunk.

[0008] If the first storage address in the first chunk is associated with the target storage address, then the version flag of the first data is set to the first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address;

[0009] The first valid data and its corresponding version flag in the first chunk are moved to the third chunk, and the third storage address of the first valid data in the third chunk is associated with the storage address on the target disk space; the first valid data is the data stored in the first chunk and whose storage address in the first chunk is associated with the storage address on the target disk space.

[0010] Delete the data stored in the first chunk.

[0011] In an optional implementation, after migrating the first valid data and the corresponding version flag from the first chunk to the third chunk, and associating the storage address of the first valid data in the third chunk with the storage address on the target disk space, the method further includes:

[0012] Receive a data read request for a target storage address on the target disk space;

[0013] Based on the received data read request, obtain the chunk storage address associated with the target storage address, the chunk storage address including the second storage address in the second chunk and the third storage address in the third chunk;

[0014] Read the first version flag corresponding to the data stored at the second storage address and the second version flag corresponding to the data stored at the third storage address, respectively.

[0015] If the version indicated by the first version flag is newer than the version indicated by the second version flag, the data stored in the third storage address is read.

[0016] In one alternative implementation, the data reading and writing and garbage collection of the data storage system are separated into two independent processes.

[0017] In one optional implementation, the data storage system is a distributed data storage system, wherein the data read / write service and the garbage collection service of the data storage system are deployed on different nodes.

[0018] In one alternative implementation, garbage collection of stored data further includes:

[0019] Obtain the real-time load pressure of the target I / O corresponding to the garbage collection service, as well as the upper limit of the load pressure of the target I / O;

[0020] The QoS setting of the garbage collection service is determined based on the difference between the real-time load pressure of the target I / O and the upper limit of the load pressure.

[0021] In one alternative implementation, after storing the newly written first data in the second chunk, the method further includes:

[0022] Receive a write operation targeting a fourth storage address on the target disk;

[0023] The new data corresponding to the write operation is stored in the second chunk, and the storage address of the new data in the second chunk is associated with the fourth storage address on the target disk.

[0024] In an optional implementation, after migrating the first valid data and the corresponding version flag from the first chunk to the third chunk, and associating the third storage address of the first valid data in the third chunk with the storage address on the target disk space, the method further includes:

[0025] The second valid data and corresponding version flag in the second chunk, and the target data and corresponding version flag in the third chunk are moved to the fourth chunk;

[0026] Delete the data in the second chunk and the data in the third chunk;

[0027] The second valid data in the second chunk is data stored in the second chunk whose storage address in the second chunk is associated with a storage address on the target disk space; the target data is data stored in the fourth chunk whose disk storage address is different from the second valid data.

[0028] Secondly, the present invention provides a method for overwriting data, applied to a data storage system, the method comprising:

[0029] If garbage collection for the first chunk has been initiated, and an overwrite write operation is received for the target storage address on the target disk space, the newly written first data will be stored in the second chunk.

[0030] If the first storage address in the first chunk is associated with the target storage address, then the version flag of the first data is set to the first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address.

[0031] Thirdly, the present invention provides a data storage garbage collection device for use in a data storage system, the device comprising:

[0032] The operation receiving module is used to, when garbage collection for the first chunk has been started, if it receives an overwrite write operation for the target storage address on the target disk space, store the newly written first data in the second chunk.

[0033] The storage module is configured to, if a first storage address in the first chunk is associated with the target storage address, set the version flag of the first data to a first version flag, and associate the second storage address of the first data in the second chunk with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address;

[0034] The first data migration module is used to migrate the first valid data and the corresponding version flag in the first chunk to the third chunk, and associate the third storage address of the first valid data in the third chunk with the storage address on the target disk space; the first valid data is the data stored in the first chunk and whose storage address in the first chunk is associated with the storage address on the target disk space.

[0035] The first data deletion module is used to delete the data stored in the first chunk.

[0036] Fourthly, the present invention provides a data storage system, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the data storage garbage collection method of the first aspect or any corresponding embodiment described above, or to perform the overwrite data method of the second aspect or any corresponding embodiment described above.

[0037] Fifthly, the present invention provides a computer-readable storage medium storing computer instructions, which are used to cause a computer to perform the storage data garbage collection method of the first aspect or any corresponding embodiment described above, or to perform the overwrite data method of the second aspect or any corresponding embodiment described above.

[0038] In a sixth aspect, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute the storage data garbage collection method of the first aspect or any corresponding embodiment described above, or to execute the overwrite data method of the second aspect or any corresponding embodiment described above.

[0039] The storage data garbage collection method, data writing method, apparatus, and storage system provided in this invention, for overwrite writes performed during garbage collection, determine a new version flag based on the version flag of the old data as the version flag of the newly written data. Therefore, even if the association between the target storage address on the target disk space and the storage address in the chunk where the newly written data is located is established first during garbage collection, and then the association between the target storage address on the target disk space and the storage address of the chunk after data migration (i.e., the third chunk) is established, subsequent data reading operations will determine the latest data to be read based on the latest version flag. In other words, this invention provides a lock-free multi-version garbage collection scheme. By isolating data versions, garbage collection and user write operations can be performed in parallel, without waiting for garbage collection to be completed before performing write operations. This avoids the problem of the first valid data migrated during garbage collection overwriting the newly written data. The parallel execution of garbage collection and user write operations can greatly reduce the write performance loss of the data storage system caused by garbage collection and improve the write performance of the data storage system. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the LSM tree structure.

[0042] Figure 2 This is a schematic diagram of the waste recycling process;

[0043] Figure 3 This is a diagram illustrating transaction consistency.

[0044] Figure 4 This is a diagram of a locked waste recycling process.

[0045] Figure 5 This is a schematic flowchart of a data storage garbage collection method according to an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of a lockless, high-performance waste recycling process according to an embodiment of the present invention;

[0047] Figure 7 This is a schematic diagram of version cleanup for multi-version garbage collection according to an embodiment of the present invention;

[0048] Figure 8 This is a schematic diagram of the PID feedback QoS for waste recycling according to an embodiment of the present invention;

[0049] Figure 9 This is a structural block diagram of a data storage garbage collection device according to an embodiment of the present invention;

[0050] Figure 10 This is a structural block diagram of an overlay writing data device according to an embodiment of the present invention;

[0051] Figure 11 This is a schematic diagram of the hardware structure of the data storage system according to an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] In related technologies, the Log-Structured Merge Tree (LSM) storage system, which separates metadata from data, is a typical append-write storage structure, such as... Figure 1As shown, a chunk is the smallest unit of data storage. C0 and C1 trees are B+tree structures that can sort the index, increasing read performance.

[0054] like Figure 1 As shown, the process of writing data in the Append storage engine is as follows:

[0055] (1) First write Chunk to persist the data.

[0056] (2) Writing index data into the C0 tree data structure makes indexing easier.

[0057] (3) When the C0 tree is large enough, merge it into the C1 tree.

[0058] Overwrite operations generate a large amount of garbage data in the chunk, which requires garbage collection (GC) to relocate.

[0059] The process of garbage data recycling in related technologies is as follows:

[0060] like Figure 2 As shown, the Chunk stores three data entries with addresses 0-4K, 4-8K, and 16-20K on the disk space. Then, the data at address 0-4K is overwritten, the data at address 4-8K is overwritten, and the data at address 0-4K is overwritten again. During the overwrite process, the storage address on the disk space needs to be associated with the storage address of the latest data in the Chunk (the association between the storage address on the disk space and the storage address in the Chunk is...). Figure 2 The index shown is used in the garbage collection process. During garbage collection, valid data stored in a Chunk (whose storage address in the Chunk is still associated with its storage address on disk) needs to be moved to a new GCChunk. The index is then modified to point to the new GCChunk, meaning the storage address of the valid data on disk is now associated with its storage address in the new GCChunk. However, as shown in the image... Figure 3As shown, since both user overwrites and garbage collection modify the index, transactions are involved. For example, for a storage address 0-4K on disk space, an overwrite operation will break the association between storage address 0-4K and the storage address of the old data (data before the overwrite) in the Chunk, and establish an association between it and the storage address of the new data (data after the overwrite) in the Chunk. Garbage collection, on the other hand, establishes an association between storage address 0-4K on disk space and the storage address in GCChunk. Therefore, if the association (i.e., the index) is not locked, the final association with storage address 0-4K on disk space might be with the storage address in GCChunk (the storage address of the migrated data), rather than the storage address of the newly written data in the Chunk. This means there is a possibility that GCChunk data could overwrite the user's newly written data. Locking the index during garbage collection can avoid this problem.

[0061] like Figure 4 As shown, the garbage collection process for locking the index is as follows:

[0062] 1. After garbage collection starts, for the chunk of garbage to be collected, first lock the indexes related to the storage address in the chunk, that is, lock the association relationship related to the storage address in the chunk, to prevent the disk space storage address associated with the storage address in the chunk from being overwritten;

[0063] 2. Move the valid data in the Chunk of garbage to be recycled to GCChunk.

[0064] 3. After the migration is successful, move the index from its original position to its new position and then release the index. In other words, switch the storage address of the chunk associated with the disk space storage address corresponding to the valid data from the storage address of the chunk to be garbage collected to the storage address of the valid data storage location in the GCChunk.

[0065] 4. Delete the data in the Chunk containing the garbage to be recycled.

[0066] According to an embodiment of the present invention, a method for garbage collection of stored data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of executable computer instructions. Furthermore, although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0067] This embodiment provides a method for garbage collection of stored data, which can be used in a data storage system. This data storage system can be a centralized storage system, a distributed storage system, or other types of storage systems. Figure 5 This is a flowchart of a storage data garbage collection method according to an embodiment of the present invention, such as... Figure 5 As shown, the process includes the following steps:

[0068] Step S501, for the first chunk (i.e. Figure 6 If garbage collection in Chunk1 is already enabled, and an overwrite write operation is received targeting a specific storage address on the target disk space, then the newly written first data is stored in the second chunk (i.e., ...). Figure 6 In Chunk2, the second chunk is a different chunk from the first. Overwrite operations are at the disk level; at the storage engine level (i.e., within a chunk), data is stored in an append-only manner.

[0069] Specifically, the target disk space could be, for example, a cloud hard drive.

[0070] In step S502, if the first storage address in the first chunk is associated with the target storage address, then the version flag of the first data is set to a first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address. The second data and the first data are the data before and after being overwritten at the target storage address on the target disk space, respectively. Step S501 can be executed simultaneously with step S502 or combined into one step.

[0071] In other words, if the garbage collection process for the first chunk is started, and the user needs to overwrite data at a target storage address on the target disk space, and the chunk storage address currently associated with the target storage address is a storage address in the first chunk (i.e., the data previously written to the target storage address is stored in the first chunk), then the newly written data will be stored in another chunk (i.e., the second chunk) outside the first chunk. Moreover, a new version flag (first version flag) will be determined based on the version flag of the old data (second version flag) as the version flag of the newly written data. For example, if the version flag of the old data is V1, then V2 can be used as the version flag of the newly written data.

[0072] Step S503: Move the first valid data and the corresponding version flag from the first chunk to the third chunk (i.e., ... Figure 6 In GCChunk1), the first valid data is stored in the first chunk, and its third storage address in the third chunk is associated with its storage address on the target disk space. The first valid data is data stored in the first chunk, and its storage address in the first chunk is associated with its storage address on the target disk space. The first valid data in the first chunk includes not only the second data mentioned above but also other data.

[0073] Step S504: Delete the data stored in the first chunk.

[0074] The storage data garbage collection method provided in this embodiment, for overwrite writes performed during garbage collection, determines a new version flag based on the version flag of the old data as the version flag of the newly written data. Therefore, even if the association between the target storage address on the target disk space and the storage address in the chunk where the newly written data is located is established first during garbage collection, and then the association between the target storage address on the target disk space and the storage address of the chunk after data migration (i.e., the third chunk) is established, subsequent data reading operations will determine the latest data to be read based on the latest version flag. In other words, this embodiment of the invention provides a lock-free multi-version garbage collection scheme. By isolating data versions, garbage collection and user write operations can be performed in parallel, without waiting for garbage collection to be completed before performing write operations. This avoids the problem of effective data being migrated during garbage collection overwriting newly written data. The parallel execution of garbage collection and user write operations can greatly reduce the write performance loss of the data storage system caused by garbage collection and improve the write performance of the data storage system.

[0075] In some optional implementations, after step S503, that is, after migrating the first valid data and the corresponding version flag in the first chunk to the third chunk and associating the storage address of the first valid data in the third chunk with the storage address on the target disk space, the method further includes:

[0076] Receive a data read request for a target storage address on the target disk space;

[0077] Based on the received data read request, obtain the chunk storage address associated with the target storage address, the chunk storage address including the second storage address in the second chunk and the third storage address in the third chunk;

[0078] Read the first version flag corresponding to the data stored at the second storage address and the second version flag corresponding to the data stored at the third storage address, respectively.

[0079] If the version indicated by the first version flag is newer than the version indicated by the second version flag, the data stored in the third storage address is read.

[0080] This invention provides a data reading method for a data storage system that applies a lock-free multi-version garbage collection scheme. When reading data at a target storage address in a target disk space, if the target storage address is associated with two different chunk storage addresses, it is necessary to determine which of the two different chunk storage addresses is the latest version of the data and read the latest version of the data.

[0081] In some optional implementations, after storing the newly written first data in the second chunk, the method further includes:

[0082] Receive a write operation targeting a fourth storage address on the target disk; the fourth storage address may or may not be the target storage address. The write operation may or may not be an overwrite write operation.

[0083] The new data corresponding to the write operation is stored in the second chunk, and the storage address of the new data in the second chunk is associated with the fourth storage address on the target disk.

[0084] like Figure 7 As shown, after the last garbage collection was completed, the first chunk (i.e. Figure 6 The first valid data in Chunk1 has been moved to GCChunk1, and the data overwritten during garbage collection is stored in the second chunk (i.e., Figure 6 and Figure 7 After Chunk2 is created, data written to the target disk space by write operations (including overwrites and new writes) will also be stored in Chunk2. For subsequent garbage collection processes for Chunk2, please refer to the section above regarding the first chunk (i.e., ...). Figure 6 During the garbage collection process of Chunk1, GCChunk2 can serve as the destination for migrating valid data from Chunk2. Data overwritten after the garbage collection process starts can be stored in the previously cleared Chunk1. Figure 7 (Not shown).

[0085] In another alternative implementation, such as Figure 7As shown, after migrating the first valid data and its corresponding version flag from the first chunk to the third chunk, and associating the third storage address of the first valid data in the third chunk with the storage address on the target disk space, the method further includes:

[0086] The second valid data and corresponding version flag in the second chunk, and the target data and corresponding version flag in the third chunk are moved to the fourth chunk (i.e., ...). Figure 7 In GCChunk2); the fourth chunk can be the first chunk that has already been cleared, or it can be any other chunk;

[0087] Delete the data in the second chunk and the data in the third chunk;

[0088] The second valid data in the second chunk is data stored in the second chunk whose storage address in the second chunk is associated with the storage address on the target disk space; the target data is data stored in the fourth chunk whose disk storage address (i.e., the storage address in the target disk space) is different from the second valid data.

[0089] The above process can be the next garbage collection process after the garbage collection of the first chunk is completed. In this garbage collection process, not only is garbage data collected for the second chunk, but also for the third chunk (i.e., Figure 7 Garbage collection is performed in GCChunk1 of the fourth chunk. The disk storage address associated with the chunk storage address of the target data in the fourth chunk is different from that of the second valid data, indicating that the target data has not been overwritten and is still the latest data at the corresponding disk storage address. Even if its version is lower than the second valid data, it is still valid data and needs to be moved to the fourth chunk (i.e., GCChunk1) as valid data. Figure 7 In GCChunk2).

[0090] In this embodiment of the invention, redundant data caused by versioning needs to be cleaned up during the garbage collection process. For example... Figure 7 As shown, the data in 0-4K and 4-8K have two versions. The lower version data and version flag need to be cleaned up during recycling to completely clean up the garbage data in the storage system.

[0091] The version used in this embodiment of the invention is a data version, which differs from the hierarchical versioning concept of LSM trees in related technologies. LSM tree compaction (the process of merging and cleaning data in an LSM tree) involves a 7-level tree structure, with merging and garbage collection occurring between levels, and concurrent data writes require locking. In contrast, in this embodiment of the invention, garbage collection is performed without locks. Data versioning allows for fine-grained garbage collection, resulting in a smaller amount of data involved and less impact on the data storage system, thereby improving the performance of the data storage system.

[0092] In some optional implementations, the data reading and writing and garbage collection of the data storage system are separated into two independent processes.

[0093] In this embodiment of the invention, since the entire garbage collection process is lock-free, the read / write and garbage collection processes can be separated into two processes, thereby reducing the pressure on the read / write processes and improving read / write performance.

[0094] In some optional implementations, the data storage system is a distributed data storage system, and the data read / write service and garbage collection service of the data storage system are deployed on different nodes.

[0095] In this embodiment of the invention, since garbage collection does not require locks, the user read / write and garbage collection processes can be completely isolated. Therefore, for distributed data storage systems, read / write and garbage collection services can be deployed on different nodes, making it more flexible and more suitable for distributed cloud storage scenarios.

[0096] In addition, since it does not rely on read / write services, the scalability of the garbage collection service is enhanced, and the number of garbage collection services can be flexibly expanded according to the specific needs of the service.

[0097] In some optional implementations, the data storage garbage collection method further includes:

[0098] Obtain the real-time load pressure of the target I / O (input / output) corresponding to the garbage collection service, as well as the upper limit of the load pressure of the target I / O;

[0099] The QoS setting of the garbage collection service is determined based on the difference between the real-time load pressure of the target I / O and the upper limit of the load pressure.

[0100] like Figure 8As shown, since garbage collection involves data migration, it puts I / O pressure on the backend storage cluster. Therefore, a feedback mechanism is needed to notify the garbage collection service to control the throughput. When the garbage collection service reads data, it obtains the current service load (IOPS) using the same I / O. The garbage collection service (i.e., gcserver) adjusts its QoS (Quality of Service) based on the IOPS value. QoS is a service designed using the token bucket algorithm. The total IOPS of the service is set to IOPS0 (i.e., the upper limit of load pressure), and the QoS settings are adjusted through a PID negative feedback mechanism between the current real-time IOPS and IOPS0. The token bucket algorithm generates tokens based on the bucket size, and the QoS service controls data migration by acquiring tokens, resulting in a more balanced load on the underlying storage service.

[0101] This invention achieves intelligent control of garbage collection speed by acquiring the underlying IOPS performance and adjusting QoS according to the PID negative feedback mechanism, thereby reducing the average load of the data storage system and making data storage system read and write operations smoother.

[0102] According to an embodiment of the present invention, an embodiment of a method for overwriting data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of executable computer instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0103] This embodiment provides a method for overwriting data, which can be used in a data storage system. The data storage system can be a centralized storage system, a distributed storage system, or other types of storage systems. The process includes the following steps:

[0104] Step 1: If garbage collection for the first chunk has been started, and an overwrite write operation is received for the target storage address on the target disk space, the newly written first data will be stored in the second chunk.

[0105] Step 2: If the first storage address in the first chunk is associated with the target storage address, then set the version flag of the first data to the first version flag, and associate the second storage address of the first data in the second chunk with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address.

[0106] This embodiment also provides a data storage garbage collection device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0107] This embodiment provides a data storage garbage collection device, such as... Figure 9 As shown, it includes:

[0108] The operation receiving module 901 is used to, when garbage collection for the first chunk has been started, if an overwrite operation for the target storage address on the target disk space is received, store the newly written first data in the second chunk.

[0109] Storage module 902 is configured to, if a first storage address in the first chunk is associated with the target storage address, set the version flag of the first data to a first version flag, and associate the second storage address of the first data in the second chunk with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address;

[0110] The first data migration module 903 is used to migrate the first valid data and the corresponding version flag in the first chunk to the third chunk, and associate the third storage address of the first valid data in the third chunk with the storage address on the target disk space; the first valid data is the data stored in the first chunk and whose storage address in the first chunk is associated with the storage address on the target disk space.

[0111] The first data deletion module 904 is used to delete the data stored in the first chunk.

[0112] In some alternative implementations, the data storage garbage collection device further includes:

[0113] A read request receiving module is used to receive data read requests for a target storage address on the target disk space.

[0114] The storage address acquisition module is used to acquire the chunk storage address associated with the target storage address based on the received data read request, wherein the chunk storage address includes the second storage address in the second chunk and the third storage address in the third chunk;

[0115] The version flag reading module is used to read the first version flag corresponding to the data stored at the second storage address and the second version flag corresponding to the data stored at the third storage address, respectively.

[0116] The data reading module is used to read the data stored in the third storage address when the version indicated by the first version flag is newer than the version indicated by the second version flag.

[0117] In some alternative implementations, the data storage garbage collection device further includes:

[0118] The load pressure acquisition module is used to acquire the real-time load pressure of the target I / O corresponding to the garbage collection service, as well as the upper limit of the load pressure of the target I / O;

[0119] The parameter setting determination module is used to determine the QoS setting of the garbage collection service based on the difference between the real-time load pressure of the target I / O and the upper limit of the load pressure.

[0120] In some alternative implementations, the data storage garbage collection device further includes:

[0121] A write operation receiving module is used to receive write operations targeting a fourth storage address on the target disk;

[0122] The write data storage module is used to store the new data corresponding to the write operation in the second chunk, and associate the storage address of the new data in the second chunk with the fourth storage address on the target disk.

[0123] In some alternative implementations, the data storage garbage collection device further includes:

[0124] The second data migration module is used to migrate the second valid data and the corresponding version flag in the second chunk, the target data of the third chunk and the corresponding version flag to the fourth chunk;

[0125] The second data deletion module is used to delete data in the second chunk and data in the third chunk;

[0126] The second valid data in the second chunk is data stored in the second chunk whose storage address in the second chunk is associated with a storage address on the target disk space; the target data is data stored in the fourth chunk whose disk storage address is different from the second valid data.

[0127] Further functional descriptions of the above modules are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0128] In this embodiment, the data storage garbage collection device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0129] This embodiment also provides an overwrite data writing device for implementing the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0130] This embodiment provides an overwrite data writing device, such as... Figure 10 As shown, it includes:

[0131] The overwrite operation receiving module 1001 is used to, when garbage collection for the first chunk has been started, receive an overwrite operation for the target storage address on the target disk space, and then store the newly written first data in the second chunk.

[0132] The write data storage module 1002 is used to set the version flag of the first data to a first version flag if the first storage address in the first chunk is associated with the target storage address, and to associate the second storage address of the first data in the second chunk with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address.

[0133] In this embodiment, the overwrite data device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0134] This invention also provides a data storage system having the above-described features. Figure 9 The device shown and / or having the above Figure 10 The aforementioned device.

[0135] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a data storage system provided in an optional embodiment of the present invention, such as... Figure 11 As shown, the data storage system includes one or more processors 10, a memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the data storage system, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Figure 11 Take a processor 10 as an example.

[0136] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0137] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0138] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the data storage system. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the data storage system via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0139] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0140] The data storage system also includes a communication interface 30 for communicating with other devices or communication networks.

[0141] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0142] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0143] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for garbage collection of stored data, characterized in that, Applied to a data storage system, the method includes: If garbage collection for the first chunk has been initiated, and an overwrite write operation is received for the target storage address on the target disk space, the newly written first data will be stored in the second chunk. If the first storage address in the first chunk is associated with the target storage address, then the version flag of the first data is set to the first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address; the second data and the first data are respectively the data before and after overwriting the target storage address on the target disk space; The first valid data and its corresponding version flag in the first chunk are moved to the third chunk, and the third storage address of the first valid data in the third chunk is associated with the storage address on the target disk space; the first valid data is the data stored in the first chunk and whose storage address in the first chunk is associated with the storage address on the target disk space. Delete the data stored in the first chunk.

2. The method according to claim 1, characterized in that, After migrating the first valid data and its corresponding version flag from the first chunk to the third chunk, and associating the storage address of the first valid data in the third chunk with the storage address on the target disk space, the process further includes: Receive a data read request for a target storage address on the target disk space; Based on the received data read request, obtain the chunk storage address associated with the target storage address, the chunk storage address including the second storage address in the second chunk and the third storage address in the third chunk; Read the first version flag corresponding to the data stored at the second storage address and the second version flag corresponding to the data stored at the third storage address, respectively. If the version indicated by the first version flag is newer than the version indicated by the second version flag, the data stored in the third storage address is read.

3. The method according to claim 1, characterized in that, The data storage system separates data reading and writing and garbage collection into two independent processes.

4. The method according to claim 1, characterized in that, The data storage system is a distributed data storage system, in which data read / write services and garbage collection services are deployed on different nodes.

5. The method according to claim 1, characterized in that, Also includes: Obtain the real-time load pressure of the target I / O corresponding to the garbage collection service, as well as the upper limit of the load pressure of the target I / O; The QoS setting of the garbage collection service is determined based on the difference between the real-time load pressure of the target I / O and the upper limit of the load pressure.

6. The method according to claim 1, characterized in that, After migrating the first valid data and its corresponding version flag from the first chunk to the third chunk, and associating the third storage address of the first valid data in the third chunk with the storage address on the target disk space, the process further includes: The second valid data and corresponding version flag in the second chunk, and the target data and corresponding version flag in the third chunk are moved to the fourth chunk; Delete the data in the second chunk and the data in the third chunk; The second valid data in the second chunk is data stored in the second chunk whose storage address in the second chunk is associated with a storage address on the target disk space; the target data is data stored in the fourth chunk whose disk storage address is different from the second valid data.

7. A method for overwriting data, characterized in that, Applied to a data storage system, the method includes: If garbage collection for the first chunk has been initiated, and an overwrite write operation is received for the target storage address on the target disk space, the newly written first data will be stored in the second chunk. If the first storage address in the first chunk is associated with the target storage address, then the version flag of the first data is set to the first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address; the second data and the first data are respectively the data before and after overwriting the target storage address on the target disk space.

8. A data storage garbage recycling device, characterized in that, The device, used in a data storage system, includes: The operation receiving module is used to, when garbage collection for the first chunk has been started, if it receives an overwrite write operation for the target storage address on the target disk space, store the newly written first data in the second chunk. The storage module is configured to, if a first storage address in the first chunk is associated with the target storage address, set the version flag of the first data to a first version flag, and associate the second storage address of the first data in the second chunk with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address; the second data and the first data are respectively the data before and after being overwritten at the target storage address on the target disk space; The first data migration module is used to migrate the first valid data and the corresponding version flag in the first chunk to the third chunk, and associate the third storage address of the first valid data in the third chunk with the storage address on the target disk space; the first valid data is the data stored in the first chunk and whose storage address in the first chunk is associated with the storage address on the target disk space. The first data deletion module is used to delete the data stored in the first chunk.

9. A data storage system, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 7.

Images