Storage data garbage collection method, data writing method and device and storage system

Abstract

The invention relates to the technical field of data storage, and discloses a storage data garbage collection method, a data writing method, a data writing device and a storage system.The garbage collection method comprises the steps that under the condition that garbage collection is started, if overwrite operation for a target storage address of a target disk space is received, the overwrite operation is started; if yes, storing the newly written first data in the second chunk; if the first storage address in the first chunk is associated with the target storage address, setting a first version mark for the first data, and associating a second storage address of the first data in the second chunk with the target storage address; moving the first valid data in the first chunk and the corresponding version mark to a third chunk, and associating a third storage address of the first valid data in the third chunk with the storage address on the target disk space; and deleting the data stored in the first chunk. The read-write performance of the data storage system can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a storage data garbage recovery method, a data writing method, a device and a storage system. Background Art

[0002] With the rapid development of information technology and the improvement of storage hardware technology, the data generated and used by humans after entering the information society has experienced exponential growth. Whether it is a large number of short videos, or ubiquitous sensors and cameras, whether it is hundreds of millions of customer online orders, or map data with an accuracy of meters, a large amount of data is generated all the time. These data are vital to people, and the storage and calculation of these data also determine the efficiency and development of all walks of life. In order to better store and calculate these data, it is necessary to design more flexible and efficient data storage, processing and analysis tools. The advent of the big data era will also inevitably bring about the improvement or fundamental change of technologies and applications such as cloud computing, data warehouses, and data mining.

[0003] However, big data is a huge test for data writing performance. It needs to store a large amount of videos, pictures, logs and other continuous data. Therefore, modern storage systems store data in the engine layer in the append mode. This mode no longer addresses when writing, but continuously appends in the form of logs, thereby improving writing efficiency. However, append writing also encounters problems. In order to achieve overwrite writing semantics, each overwrite write will generate redundant data for append. Garbage collection of these overwritten data has become a new concern for storage systems.

[0004] Like solid state drives (SSDs), garbage collection can cause a significant drop in performance, largely because users are writing to the same area while relocating data, and locks are needed to ensure the order of write transactions. Extending the system to a distributed storage system will increase the scope of this lock and have a greater impact on performance. Summary of the invention

[0005] In view of this, the present invention provides a storage data garbage collection method, a data writing method, an apparatus and a storage system to solve the problem that the performance of the data storage system is greatly reduced when performing garbage collection.

[0006] In a first aspect, the present invention provides a method for storing data garbage collection, which is applied to a data storage system, and the method comprises:

[0007] When garbage collection for the first chunk has been started, if an overwrite write operation for a target storage address on the target disk space is received, the newly written first data is stored in the second chunk;

[0008] If the first storage address in the first chunk is associated with the target storage address, the version flag of the first data is set to the first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address;

[0009] relocating the first valid data and the corresponding version mark in the first chunk to the third chunk, and associating the third storage address of the first valid data in the third chunk with the storage address on the target disk space; the first valid data is the data stored in the first chunk, and the storage address in the first chunk is associated with the storage address on the target disk space;

[0010] Delete the data stored in the first chunk.

[0011] In an optional implementation, after the first valid data and the corresponding version flag in the first chunk are moved to the third chunk, and the storage address of the first valid data in the third chunk is associated with the storage address on the target disk space, the method further includes:

[0012] receiving a data read request for a target storage address on the target disk space;

[0013] Based on the received data read request, obtaining a chunk storage address associated with the target storage address, the chunk storage address including a second storage address in the second chunk and a third storage address in the third chunk;

[0014] respectively reading the first version flag corresponding to the data stored at the second storage address and the second version flag corresponding to the data stored at the third storage address;

[0015] When the version indicated by the first version flag is newer than the version indicated by the second version flag, the data stored in the third storage address is read.

[0016] In an optional implementation, data reading and writing and garbage collection of the data storage system are separated into two independent processes.

[0017] In an optional implementation, the data storage system is a distributed data storage system, and the data reading and writing service and the garbage collection service of the data storage system are deployed on different nodes.

[0018] In an optional implementation, storage data garbage collection further includes:

[0019] Obtaining the real-time load pressure of the target I / O corresponding to the garbage collection service and the upper limit value of the load pressure of the target I / O;

[0020] The setting of the QOS of the garbage collection service is determined based on the difference between the real-time load pressure of the target I / O and the load pressure upper limit.

[0021] In an optional implementation, after storing the newly written first data in the second chunk, the method further includes:

[0022] receiving a write operation for a fourth storage address on the target disk;

[0023] The new data corresponding to the write operation is stored in the second chunk, and the storage address of the new data in the second chunk is associated with the fourth storage address on the target disk.

[0024] In an optional implementation, after the first valid data and the corresponding version flag in the first chunk are moved to the third chunk, and the third storage address of the first valid data in the third chunk is associated with the storage address on the target disk space, the method further includes:

[0025] relocating the second valid data and the corresponding version flag in the second chunk, and the target data and the corresponding version flag in the third chunk to the fourth chunk;

[0026] Deleting the data in the second chunk and the data in the third chunk;

[0027] Among them, the second valid data in the second chunk is data stored in the second chunk, and the storage address in the second chunk is associated with the storage address on the target disk space; the target data is data stored in the fourth chunk, and the disk storage address associated with the chunk storage address is different from the second valid data.

[0028] In a second aspect, the present invention provides a method for overwriting data, which is applied to a data storage system, and the method comprises:

[0029] When garbage collection for the first chunk has been started, if an overwrite write operation for a target storage address on the target disk space is received, the newly written first data is stored in the second chunk;

[0030] If the first storage address in the first chunk is associated with the target storage address, the version flag of the first data is set to the first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address.

[0031] In a third aspect, the present invention provides a storage data garbage recovery device, which is applied to a data storage system, and the device comprises:

[0032] an operation receiving module, configured to store the newly written first data in the second chunk if an overwrite write operation to a target storage address on a target disk space is received when garbage collection for the first chunk has been started;

[0033] a storage module, configured to, if the first storage address in the first chunk is associated with the target storage address, set the version flag of the first data to the first version flag, and associate the second storage address of the first data in the second chunk with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address;

[0034] a first data relocation module, configured to relocate the first valid data and the corresponding version mark in the first chunk to the third chunk, and associate the third storage address of the first valid data in the third chunk with the storage address on the target disk space; the first valid data is data stored in the first chunk, and the storage address in the first chunk is associated with the storage address on the target disk space;

[0035] The first data deleting module is used to delete the data stored in the first chunk.

[0036] In a fourth aspect, the present invention provides a data storage system, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the storage data garbage collection method of the above-mentioned first aspect or any corresponding embodiment thereof, or executes the overwriting data method of the above-mentioned second aspect or any corresponding embodiment thereof by executing the computer instructions.

[0037] In a fifth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the storage data garbage collection method of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the overwriting data method of the above-mentioned second aspect or any corresponding embodiment thereof.

[0038] In a sixth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the storage data garbage collection method of the above-mentioned first aspect or any corresponding embodiment, or to execute the overwriting data method of the above-mentioned second aspect or any corresponding embodiment.

[0039] The storage data garbage collection method, data writing method, device and storage system provided by the embodiments of the present invention, for overwriting performed during the garbage collection process, will determine a new version mark as the version mark of the newly written data based on the version mark of the old data. Therefore, even if the association relationship between the target storage address on the target disk space and the storage address in the chunk where the newly written data is located is first established during the garbage collection process, and then the association relationship between the target storage address on the target disk space and the storage address of the chunk after the data is relocated (i.e., the third chunk) is established, when subsequent data reading and other operations are performed, the latest data will be determined to be read based on the latest version mark. In other words, the embodiments of the present invention provide a lock-free multi-version garbage collection scheme, which can enable garbage collection and user write operations to be performed in parallel through isolation between data versions, and there is no need to wait for the garbage collection to be completed before performing the write operation, thereby avoiding the problem of the first valid data relocated during the garbage collection process overwriting the newly written data, and the garbage collection and user write operations are performed in parallel, which can greatly reduce the write performance loss of the data storage system caused by garbage collection, and improve the write performance of the data storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a schematic diagram of the structure of the LSM tree

[0042] Figure 2 is a schematic diagram of the garbage collection process;

[0043] Figure 3 It is a diagram of transaction consistency;

[0044] Figure 4 This is a diagram of the locked garbage collection process.

[0045] Figure 5 is a flow chart of a method for garbage collection of stored data according to an embodiment of the present invention;

[0046] Figure 6 is a schematic diagram of a lock-free high-performance garbage collection process according to an embodiment of the present invention;

[0047] Figure 7 is a schematic diagram of version cleaning of multi-version garbage collection according to an embodiment of the present invention;

[0048] Figure 8 is a schematic diagram of garbage collection PID feedback QOS according to an embodiment of the present invention;

[0049] Fig. 9 is a structural block diagram of a storage data garbage collection device according to an embodiment of the present invention;

[0050] Fig.10 is a structural block diagram of a device for overwriting data according to an embodiment of the present invention;

[0051] Fig.11 It is a schematic diagram of the hardware structure of the data storage system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0053] In the Append storage engine in the related art, the Log-Structured Merge Trees (LSM tree) storage system in which metadata and data are separated is a typical storage structure written by Append, such as Figure 1As shown, chunk is the smallest data storage unit, and C0 and C1 trees are B+tree structures, which can sort indexes and improve read performance.

[0054] like Figure 1 As shown in the figure, the process of writing data in the Append storage engine is:

[0055] (1) Write Chunk first to complete data persistence.

[0056] (2) Write the index data into the data structure of C0 tree to facilitate indexing.

[0057] (3) When the C0 tree is large enough, it is merged into the C1 tree.

[0058] Overwriting will generate a large amount of garbage data in the Chunk, which requires garbage collection (GC) to move.

[0059] The process of garbage data recovery in the related art is:

[0060] like Figure 2 As shown in the figure, three pieces of data with addresses 0-4K, 4-8K and 16-20K on the disk space are stored in the Chunk. Then the data with addresses 0-4K on the disk space is overwritten, the data with addresses 4-8K is overwritten, and the data with addresses 0-4K is overwritten again. When overwriting, it is necessary to associate the storage address on the disk space with the storage address of the latest data in the Chunk (the association relationship between the storage address on the disk space and the storage address in the Chunk is Figure 2 In the garbage collection process, the valid data stored in the Chunk (its storage address in the Chunk is still associated with the storage address on the disk space) needs to be moved to the new GCChunk, and the index needs to be modified to point to the new GCChunk, that is, the storage address corresponding to the valid data on the disk space is associated with the storage address of the data in the new GCChunk. Figure 3As shown, since both user overwrite and garbage collection will modify the index, transactional issues are involved. For example, for a storage address 0-4K on the disk space, the overwrite operation will remove the association between the storage address 0-4K and the storage address where the old data (data before the overwrite) is located in the Chunk, and establish an association with the storage address where the new data (data after the overwrite) is located in the Chunk. Garbage collection will establish an association between the storage address 0-4K on the disk space and the storage address in the GCChunk. Therefore, if the association (i.e., index) is not locked, then the storage address 0-4K on the disk space may eventually be associated with the storage address in the GCChunk (the storage address of the relocated data), rather than the storage address of the newly written data in the Chunk, that is, there may be a problem of GCChunk data overwriting the user's newly written data. This problem can be avoided by locking the index during the garbage collection process.

[0061] like Figure 4 As shown in the figure, the garbage collection process for locking the index is:

[0062] 1. After garbage collection is started, for the chunk to be collected, first lock the index related to the storage address in the chunk, that is, lock the association relationship related to the storage address in the chunk, to prevent the disk space storage address associated with the storage address in the chunk from being overwritten;

[0063] 2. Move the valid data in the chunk to be recycled to the GC chunk.

[0064] 3. After the relocation is successful, the index is moved from the position before the relocation to the position after the relocation and then released. That is, the Chunk storage address associated with the disk space storage address corresponding to the valid data is switched from the storage in the Chunk to be recycled to the storage address of the valid data storage location in the GCChunk.

[0065] 4. Delete the data in the chunk to be recycled.

[0066] According to an embodiment of the present invention, an embodiment of a storage data garbage collection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of executable computer instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0067] In this embodiment, a storage data garbage collection method is provided, which can be used in a data storage system. The data storage system can be a centralized storage system, a distributed storage system, or other types of storage systems. Figure 5 is a flow chart of a method for garbage collection of storage data according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0068] Step S501, for the first chunk (i.e. Figure 6 In the case where the garbage collection of Chunk1 in the target disk space has been started, if an overwrite write operation is received for the target storage address on the target disk space, the newly written first data is stored in the second chunk (i.e. Figure 6 The second chunk is different from the first chunk. Overwriting is done at the disk level, and data is stored in the storage engine level (i.e., in the chunk) in the append mode.

[0069] Specifically, the target disk space may be, for example, a cloud hard disk.

[0070] Step S502: if the first storage address in the first chunk is associated with the target storage address, the version flag of the first data is set to the first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address. The second data and the first data are the data before and after the target storage address on the target disk space is overwritten. Step S501 can be performed simultaneously with step S502 or combined into one step.

[0071] In other words, when the garbage collection process for the first chunk is started, if the user needs to overwrite data at the target storage address on the target disk space, and the chunk storage address currently associated with the target storage address is a storage address in the first chunk (i.e., the data previously written to the target storage address is stored in the first chunk), then the newly written data will be stored in another chunk (i.e., the second chunk) other than the first chunk, and a new version flag (the first version flag) will be determined based on the version flag of the old data (the second version flag) as the version flag of the newly written data. For example, if the version flag of the old data is V1, then V2 can be used as the version flag of the newly written data.

[0072] Step S503: Move the first valid data and the corresponding version mark in the first chunk to the third chunk (i.e. Figure 6 The first valid data is stored in the first chunk and the storage address in the first chunk is associated with the storage address on the target disk space. The first valid data in the first chunk includes not only the second data but also other data.

[0073] Step S504: delete the data stored in the first chunk.

[0074] The storage data garbage collection method provided in this embodiment will determine a new version mark as the version mark of the newly written data based on the version mark of the old data for the overwrite performed during the garbage collection process. Therefore, even if the association relationship between the target storage address on the target disk space and the storage address in the chunk where the newly written data is located is first established during the garbage collection process, and then the association relationship between the target storage address on the target disk space and the storage address of the chunk after the data is relocated (i.e., the third chunk) is established, when subsequent data reading and other operations are performed, the latest data will be determined to be read based on the latest version mark. In other words, the embodiment of the present invention provides a lock-free multi-version garbage collection scheme, which can enable garbage collection and user write operations to be performed in parallel through isolation between data versions, and there is no need to wait for the garbage collection to be completed before performing the write operation, thereby avoiding the problem of the relocated valid data overwriting the newly written data during the garbage collection process. The garbage collection and user write operations are performed in parallel, which can greatly reduce the write performance loss of the data storage system caused by garbage collection and improve the write performance of the data storage system.

[0075] In some optional specific implementations, after step S503, i.e., after the first valid data and the corresponding version mark in the first chunk are moved to the third chunk, and the storage address of the first valid data in the third chunk is associated with the storage address on the target disk space, the method further includes:

[0076] receiving a data read request for a target storage address on the target disk space;

[0077] Based on the received data read request, obtaining a chunk storage address associated with the target storage address, the chunk storage address including a second storage address in the second chunk and a third storage address in the third chunk;

[0078] respectively reading the first version flag corresponding to the data stored at the second storage address and the second version flag corresponding to the data stored at the third storage address;

[0079] When the version indicated by the first version flag is newer than the version indicated by the second version flag, the data stored in the third storage address is read.

[0080] An embodiment of the present invention provides a data reading method for a data storage system that applies a lock-free multi-version garbage collection solution. When reading data at a target storage address in a target disk space, if the target storage address is associated with two different chunk storage addresses, it is necessary to determine which version of the data stored in the two different chunk storage addresses is the latest version, and read the latest version of the data.

[0081] In some optional specific implementations, after storing the newly written first data in the second chunk, the method further includes:

[0082] receiving a write operation for a fourth storage address on the target disk; the fourth storage address may be a target storage address or may not be a target storage address. The write operation may be an overwrite write operation or may not be an overwrite write operation;

[0083] The new data corresponding to the write operation is stored in the second chunk, and the storage address of the new data in the second chunk is associated with the fourth storage address on the target disk.

[0084] like Figure 7 As shown, after the last garbage collection is completed, the first chunk (i.e. Figure 6 The first valid data in Chunk1 in has been moved to GCChunk1, and the data overwritten during the garbage collection process is stored in the second chunk (i.e. Figure 6 and Figure 7 After that, the data written by the data write operation (including overwrite and new write) to the target disk space will also be stored in Chunk2. For the subsequent garbage collection process for Chunk2, please refer to the above for the first chunk (i.e. Figure 6 In the garbage collection process of Chunk1 in the GC, GCChunk2 can be used as the relocation destination of the valid data in Chunk2. After the garbage collection process is started, the overwritten data can be stored in the previously cleared Chunk1 ( Figure 7 not shown).

[0085] In another optional embodiment, if Figure 7As shown, after the first valid data and the corresponding version mark in the first chunk are moved to the third chunk, and the third storage address of the first valid data in the third chunk is associated with the storage address on the target disk space, the method further includes:

[0086] The second valid data and the corresponding version mark in the second chunk, the target data and the corresponding version mark in the third chunk are moved to the fourth chunk (i.e. Figure 7 The fourth chunk may be the cleared first chunk or another chunk;

[0087] Deleting the data in the second chunk and the data in the third chunk;

[0088] Among them, the second valid data in the second chunk is data stored in the second chunk, and the storage address in the second chunk is associated with the storage address on the target disk space; the target data is data stored in the fourth chunk, and the disk storage address (that is, the storage address in the fourth chunk) associated with the chunk storage address is different from the second valid data.

[0089] The above process may be the next garbage collection process after the garbage collection of the first chunk is completed. In this garbage collection process, not only the garbage data of the second chunk is collected, but also the garbage data of the third chunk (i.e. Figure 7 The disk storage address associated with the chunk storage address of the target data in the fourth chunk is different from the second valid data, indicating that the target data has not been overwritten and is still the latest data on the corresponding disk storage address. Even if its version is lower than the second valid data, it is still valid data and needs to be moved to the fourth chunk (i.e. Figure 7 In GCChunk2).

[0090] In the embodiment of the present invention, it is necessary to clean up the redundant data caused by the version in the process of garbage collection. Figure 7 As shown, there are two versions of data in the range of 0-4K and 4-8K. The lower version data and version flags need to be cleaned up during recycling, thereby completely cleaning up the garbage data in the storage system.

[0091] The version in the embodiment of the present invention is a data version, which is different from the hierarchical version concept of the LSM tree in the related art. The merging (Compaction, which is the process used to merge and clean up data in the LSM tree) of the LSM tree is to set a 7-layer tree structure, merge and recycle between layers, and concurrent data writing requires locking. In the embodiment of the present invention, garbage collection is performed under lock-free conditions, and the data version allows garbage collection to be performed at a fine granularity, so that the amount of data involved is small, and the impact on the data storage system is small, thereby improving the performance of the data storage system.

[0092] In some optional specific implementations, data reading and writing and garbage collection of the data storage system are separated into two independent processes.

[0093] In the embodiment of the present invention, since the garbage collection is completely lock-free, the reading and writing and garbage collection processes can be separated into two processes, thereby reducing the pressure of the reading and writing processes and improving the reading and writing performance.

[0094] In some optional specific implementations, the data storage system is a distributed data storage system, and the data reading and writing service and the garbage collection service of the data storage system are deployed on different nodes.

[0095] In the embodiment of the present invention, since garbage collection does not require locks, the user reading and writing and garbage collection processes can be completely isolated. Therefore, for a distributed data storage system, the reading and writing and garbage collection services can be deployed on different nodes, which is more flexible and more suitable for distributed scenarios of cloud storage.

[0096] In addition, since it does not rely on read-write services, the scalability of the garbage collection service is enhanced, and the number of garbage collection services can be flexibly expanded according to the specific circumstances of the service.

[0097] In some optional specific implementations, the storage data garbage collection method further includes:

[0098] Obtaining the real-time load pressure of the target I / O (input / output) corresponding to the garbage collection service and the upper limit value of the load pressure of the target I / O;

[0099] The setting of the QOS of the garbage collection service is determined based on the difference between the real-time load pressure of the target I / O and the load pressure upper limit.

[0100] like Figure 8As shown in the figure, since the garbage collection service involves data migration, it will cause I / O pressure on the backend storage cluster, so a feedback mechanism is needed to notify the garbage collection service to control the throughput of garbage collection. When the garbage collection service reads data, the same I / O obtains the load pressure (IOPS) of the current service. The garbage collection service (i.e., gcserver) adjusts the QOS (Quality of Service) of the garbage collection service according to the value of IOPS. QOS is a service designed with a token bucket algorithm. Set the total IOPS of the service = IOPS0 (i.e., the upper limit of the load pressure), and adjust the settings of the QOS service through the PID negative feedback mechanism between the current real-time IOPS and IOPS0. The token bucket algorithm generates tokens according to the size of the bucket, and the QOS service controls the migration of data by obtaining tokens. This makes the load of the underlying storage service more balanced.

[0101] The embodiment of the present invention obtains the underlying IOPS performance and adjusts the QOS according to the PID negative feedback mechanism to intelligently control the speed of garbage collection, thereby reducing the average load of the data storage system and making the data storage system read and write more smoothly.

[0102] According to an embodiment of the present invention, an embodiment of a method for overwriting data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of executable computer instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0103] In this embodiment, a method for overwriting data is provided, which can be used in a data storage system. The data storage system can be a centralized storage system, a distributed storage system, or other types of storage systems. The process includes the following steps:

[0104] Step 1: When garbage collection for the first chunk has been started, if an overwrite write operation for a target storage address on the target disk space is received, the newly written first data is stored in the second chunk;

[0105] Step 2: If the first storage address in the first chunk is associated with the target storage address, the version flag of the first data is set to the first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address.

[0106] In this embodiment, a storage data garbage collection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0107] This embodiment provides a storage data garbage collection device, such as Fig. 9 As shown, including:

[0108] An operation receiving module 901 is configured to store the newly written first data in the second chunk if an overwrite write operation to a target storage address on a target disk space is received when garbage collection for the first chunk has been started;

[0109] The storage module 902 is configured to set the version flag of the first data to the first version flag if the first storage address in the first chunk is associated with the target storage address, and associate the second storage address of the first data in the second chunk with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address;

[0110] The first data relocation module 903 is used to relocate the first valid data and the corresponding version mark in the first chunk to the third chunk, and associate the third storage address of the first valid data in the third chunk with the storage address on the target disk space; the first valid data is the data stored in the first chunk, and the storage address in the first chunk is associated with the storage address on the target disk space;

[0111] The first data deleting module 904 is used to delete the data stored in the first chunk.

[0112] In some optional implementations, the storage data garbage collection device further includes:

[0113] A read request receiving module, used for receiving a data read request for a target storage address on the target disk space;

[0114] a storage address acquisition module, configured to acquire, based on the received data read request, a chunk storage address associated with the target storage address, wherein the chunk storage address includes a second storage address in the second chunk and a third storage address in the third chunk;

[0115] a version mark reading module, used to read the first version mark corresponding to the data stored in the second storage address and the second version mark corresponding to the data stored in the third storage address respectively;

[0116] The data reading module is used to read the data stored in the third storage address when the version indicated by the first version mark is newer than the version indicated by the second version mark.

[0117] In some optional implementations, the storage data garbage collection device further includes:

[0118] A load pressure acquisition module, used to acquire the real-time load pressure of the target I / O corresponding to the garbage collection service, and the load pressure upper limit value of the target I / O;

[0119] The parameter setting determination module is used to determine the QOS setting of the garbage collection service based on the difference between the real-time load pressure of the target I / O and the load pressure upper limit value.

[0120] In some optional implementations, the storage data garbage collection device further includes:

[0121] A write operation receiving module, used for receiving a write operation for a fourth storage address on the target disk;

[0122] A write data storage module is used to store new data corresponding to the write operation in the second chunk, and associate the storage address of the new data in the second chunk with the fourth storage address on the target disk.

[0123] In some optional implementations, the storage data garbage collection device further includes:

[0124] a second data relocation module, configured to relocate the second valid data and the corresponding version flag in the second chunk, and the target data and the corresponding version flag in the third chunk to the fourth chunk;

[0125] A second data deletion module, used to delete the data in the second chunk and the data in the third chunk;

[0126] Among them, the second valid data in the second chunk is data stored in the second chunk, and the storage address in the second chunk is associated with the storage address on the target disk space; the target data is data stored in the fourth chunk, and the disk storage address associated with the chunk storage address is different from the second valid data.

[0127] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0128] The storage data garbage collection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0129] In this embodiment, a device for overwriting data is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0130] This embodiment provides a device for overwriting data, such as Fig.10 As shown, including:

[0131] The overwrite operation receiving module 1001 is used for storing the newly written first data in the second chunk if an overwrite operation for a target storage address on a target disk space is received when garbage collection for the first chunk has been started;

[0132] The overwrite data storage module 1002 is used to set the version flag of the first data to the first version flag if the first storage address in the first chunk is associated with the target storage address, and associate the second storage address of the first data in the second chunk with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address.

[0133] The data overwriting device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0134] The embodiment of the present invention also provides a data storage system having the above Fig. 9 The device shown and / or having the above Fig.10 The device described.

[0135] See also Fig.11 , Fig.11 is a schematic diagram of a data storage system provided by an optional embodiment of the present invention, such as Fig.11 As shown, the data storage system includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the data storage system, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Fig.11 A processor 10 is taken as an example.

[0136] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0137] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0138] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the data storage system, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the data storage system via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0140] The data storage system further includes a communication interface 30 for the data storage system to communicate with other devices or a communication network.

[0141] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0142] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0143] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for storing data garbage collection, characterized in that: Applied to a data storage system, the method comprises: When garbage collection for the first chunk has been started, if an overwrite write operation for a target storage address on the target disk space is received, the newly written first data is stored in the second chunk; If the first storage address in the first chunk is associated with the target storage address, the version flag of the first data is set to the first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address; relocating the first valid data and the corresponding version mark in the first chunk to the third chunk, and associating the third storage address of the first valid data in the third chunk with the storage address on the target disk space; the first valid data is the data stored in the first chunk, and the storage address in the first chunk is associated with the storage address on the target disk space; Delete the data stored in the first chunk.

2. The method according to claim 1, characterized in that After the first valid data and the corresponding version mark in the first chunk are moved to the third chunk, and the storage address of the first valid data in the third chunk is associated with the storage address on the target disk space, the method further includes: receiving a data read request for a target storage address on the target disk space; Based on the received data read request, obtaining a chunk storage address associated with the target storage address, the chunk storage address including a second storage address in the second chunk and a third storage address in the third chunk; respectively reading the first version flag corresponding to the data stored at the second storage address and the second version flag corresponding to the data stored at the third storage address; When the version indicated by the first version flag is newer than the version indicated by the second version flag, the data stored in the third storage address is read.

3. The method according to claim 1, characterized in that The data reading and writing and garbage collection of the data storage system are separated into two independent processes.

4. The method according to claim 1, characterized in that: The data storage system is a distributed data storage system, and the data reading and writing service and the garbage collection service of the data storage system are deployed on different nodes.

5. The method according to claim 1, characterized in that Also includes: Obtaining the real-time load pressure of the target I / O corresponding to the garbage collection service and the upper limit value of the load pressure of the target I / O; The setting of the QOS of the garbage collection service is determined based on the difference between the real-time load pressure of the target I / O and the load pressure upper limit.

6. The method according to claim 1, characterized in that After the first valid data and the corresponding version mark in the first chunk are moved to the third chunk, and the third storage address of the first valid data in the third chunk is associated with the storage address on the target disk space, the method further includes: relocating the second valid data and the corresponding version flag in the second chunk, and the target data and the corresponding version flag in the third chunk to the fourth chunk; Deleting the data in the second chunk and the data in the third chunk; Among them, the second valid data in the second chunk is data stored in the second chunk, and the storage address in the second chunk is associated with the storage address on the target disk space; the target data is data stored in the fourth chunk, and the disk storage address associated with the chunk storage address is different from the second valid data.

7. A method for overwriting data, characterized in that: Applied to a data storage system, the method comprises: When garbage collection for the first chunk has been started, if an overwrite write operation for a target storage address on the target disk space is received, the newly written first data is stored in the second chunk; If the first storage address in the first chunk is associated with the target storage address, the version flag of the first data is set to the first version flag, and the second storage address of the first data in the second chunk is associated with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address.

8. A storage data garbage collection device, characterized in that: Applied to a data storage system, the device comprises: an operation receiving module, configured to store the newly written first data in the second chunk if an overwrite write operation to a target storage address on a target disk space is received when garbage collection for the first chunk has been started; a storage module, configured to, if the first storage address in the first chunk is associated with the target storage address, set the version flag of the first data to the first version flag, and associate the second storage address of the first data in the second chunk with the target storage address; the version indicated by the first version flag is newer than the version indicated by the second version flag, and the second version flag is the version flag of the second data stored in the first storage address; a first data relocation module, configured to relocate the first valid data and the corresponding version mark in the first chunk to the third chunk, and associate the third storage address of the first valid data in the third chunk with the storage address on the target disk space; the first valid data is data stored in the first chunk, and the storage address in the first chunk is associated with the storage address on the target disk space; The first data deleting module is used to delete the data stored in the first chunk.

9. A data storage system, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images