Method and system for large-value data storage space management

Abstract

The invention relates to the field of computer technology storage, in particular to a method and system for managing a large-value data storage space, and the method comprises the following steps: managing the state of a data storage block in a memory; realizing persistence of a data storage block state change event in a hard disk; the memory state change of the data storage block and the persistence process of the change event in the hard disk are separately realized. The method has the beneficial effects that the distributor is introduced, and the process separation of memory management storage space and distribution / recovery event persistence is realized through the use of the bitmap and the reference count table, so that the optimal management of defragmentation and space release is realized. Through atomization distribution, writing and metadata persistence methods, the method is suitable for a high-reliability storage scene. In the write-in stage, when data write-in fails, it is ensured that stock metadata and a data storage block are not affected; in the deletion stage, it is ensured that deletion operation does not block the reading request; and in the data adding stage, the operation is more efficient.

Description

Technical Field

[0001] The present invention relates to the field of computer technology storage, and particularly to a method and system for managing large-value data storage space. Background Art

[0002] With the explosive growth of scenarios such as the Internet of Things, real-time logs, and multimedia metadata, modern storage systems face the demand for efficient management of massive large-value data (BLOB, Binary Large Object).

[0003] As a high-performance embedded storage engine, RocksDB based on the LSM-Tree architecture performs excellently in small key-value scenarios, but there are significant bottlenecks in processing large values, such as write amplification and space waste. An improved approach is to adopt a BLOB system, which is a system for storing large-capacity binary data files for storing and managing a large amount of data. By storing large values in separate blob files, dedicated files introduced by storage are stored separately from SST files. However, in the prior art, when allocating and deleting data blocks, single-phase commit or loosely coupled metadata management is adopted, which has the following defects: 1. The allocator directly modifies the persistent metadata. If the data write fails, it causes the metadata to be inconsistent with the real data; 2. Intermediate state risk: While modifying the memory, the persistent metadata is also modified. If the persistence fails, the residual invalid data block positions occupy space and are difficult to recycle; 3. The deletion operation directly clears the memory reference, resulting in the sudden invalidation of the data storage block being read, causing a data access error; 4. Using Copy-on-write for data append increases memory copying and reduces operation efficiency. Summary of the Invention

[0004] The present invention aims to provide a method and system for managing large-value data storage space to reduce storage fragmentation and improve space utilization by separating the process of modifying the allocation / recovery state of data storage blocks in memory from the persistence of allocation / recovery events.

[0005] The present invention achieves the above object through the following solution: A method for managing large-value data storage space includes the following steps:

[0006] Managing the state of data storage blocks in memory;

[0007] Implementing the persistence of data storage block state change events in the hard disk;

[0008] The change of the memory state of the data storage block is separated from the persistence process of the change event in the hard disk.

[0009] Preferably, the state in the memory of the data storage block is managed by an allocator.

[0010] Preferably, the persistence of the data storage block state change event is implemented in the metadata storage in the hard disk; changes to the metadata involved in the process of changing the data storage block also need to be persisted in the metadata storage.

[0011] Preferably, the process of jointly persisting the data storage block state change event and the metadata change adopts atomic update.

[0012] Preferably, the allocator includes a bitmap, and one bit in the bitmap corresponds to a data storage block, indicating whether the corresponding data storage block is occupied.

[0013] Preferably, the allocator further includes a reference count table, which is used to record how many ongoing IO operations there are on the occupied data storage block.

[0014] Preferably, the allocator manages and allocates data storage bits in the memory, and is responsible for modifying the memory state of the data storage block; the data storage bit is a logical location identifier dynamically generated by the allocator in the memory, and is used to record the starting position of data writing.

[0015] Preferably, the method for managing the large-value data storage space includes writing data, specifically including the following steps:

[0016] (101) In the memory, allocate a new data storage bit and point it to the corresponding hard disk space;

[0017] (102) Write data to the hard disk space pointed to by the allocated data storage bit;

[0018] (103) Generate corresponding metadata and persist it together with the event of allocating the data storage bit to the metadata storage.

[0019] Preferably, the method for managing the large-value data storage space includes deleting data, specifically including the following steps:

[0020] (201) Quickly load all allocation information in the memory;

[0021] (202) Persist the deletion mark in the metadata storage;

[0022] (203) Determine whether the read and write operations of the data storage block are completed. If completed, proceed to the next step (204). If not completed, delay the recycling and wait for the read and write to complete;

[0023] (204) Modify the data storage block allocation in the memory and release the corresponding data storage block back to the resource pool.

[0024] Preferably, the method for managing the large-value data storage space includes appending data, which specifically includes the following steps:

[0025] (301) Modify the reference count table in the memory;

[0026] (302) Directly append data to the corresponding hard disk space;

[0027] (303) Generate corresponding metadata and persist it into the metadata storage.

[0028] A system for managing the large-value data storage space includes: a distributor and a metadata storage; the distributor is located in the memory and is used to implement the modification of the memory state of the data storage block; the metadata storage is located on the hard disk and is used to implement the metadata storage of the data storage block and the persistence of the data storage block state change event; the modification of the memory state of the data storage block and the persistence process of the data storage block state change event are separated and implemented.

[0029] The beneficial effects of the present invention are as follows: By introducing a distributor and using a bitmap and a reference count table, the process separation of the memory management storage space and the persistence of the allocation / recycling event is realized, so as to realize the optimized management of defragmentation and space release, reduce storage fragmentation, improve space utilization rate and recycling efficiency. Through the atomic allocation, writing and metadata persistence methods, it is applicable to high-reliability storage scenarios such as object storage and log systems. In the allocation stage, it is executed purely in memory. When the data writing fails or is interrupted, the existing metadata and data storage blocks are not affected. The metadata update and the data storage block allocation information are atomically committed only after the data is successfully written, ensuring that the existing data is not contaminated and there is no intermediate state residue in case of power failure or system failure during the writing process. In the data storage block deletion stage, through the reference counting and delayed recycling mechanism, it is ensured that the deletion operation does not block the read request and zero conflicts are achieved. In the data appending stage, the copy of the original data in the memory is reduced, and the appending data is directly operated behind the original data in the hard disk space, making the appending operation more efficient. At the same time, the appending process is not affected by the read operation and does not affect the original data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flowchart of a method for managing the large-value data storage space provided by an embodiment of the present invention;

[0031] Figure 2 is a flowchart of another method for managing the large-value data storage space provided by an embodiment of the present invention;

[0032] Figure 3It is a schematic flowchart of another method for managing large-value data storage space provided by an embodiment of the present invention;

[0033] Figure 4 It is a schematic structural diagram of a system for managing large-value data storage space provided by an embodiment of the present invention;

[0034] Figure 5 It is a schematic structural diagram of the allocator provided by an embodiment of the present invention. Detailed implementation manners

[0035] The present invention will be further described below in conjunction with specific implementation examples, but the protection scope of the present invention is not limited thereto:

[0036] A method for managing large-value data storage space includes:

[0037] Managing the state in the memory of the data storage block;

[0038] Implementing the persistence of the data storage block state change event in the hard disk.

[0039] The change of the memory state of the data storage block (chunk) is separated from the process of persisting the data storage block state change event in the hard disk.

[0040] The state in the memory of the data storage block is managed through an allocator. The persistence of the data storage block state change event is implemented in the metadata storage (MetaStore) in the hard disk. The change of metadata involved in the process of data storage block change also needs to be persisted in the metadata storage.

[0041] The process of persisting the data storage block state change event and the metadata change together adopts atomic update.

[0042] The persistence refers to the process of storing data from volatile memory to non-volatile media (such as hard disks, SSDs, or databases), ensuring that the data can be retained for a long time and recovered after the system power-off, crash, or restart, and guaranteeing the reliability and consistency of the data. The data storage block usually refers to a continuous space in the storage system for storing data. The size of the data storage block can be set according to actual needs, such as 64KB, 512KB, 1MB, 4MB, etc.

[0043] In an example embodiment of the present invention, a method for managing the writing of large-value data into the storage space is provided, as Figure 1 shown, and specifically includes the following steps:

[0044] S101: In the memory, allocate a new data storage position (chunk_pos) and point it to the corresponding hard disk space.

[0045] The data storage location refers to the logical location identifier dynamically generated by an Allocator in the memory of the storage system, which points to a newly allocated data storage block on the disk and is used to accurately record the starting position of data writing, such as file ID and offset. The Allocator manages and allocates the data storage locations in the memory and is responsible for modifying the memory state of allocating / reclaiming data storage space.

[0046] The management of data storage locations is implemented through a bitmap and a reference count table in the Allocator's memory. The bitmap is usually used to track the occupancy status of data storage blocks. The storage space on the hard disk is divided into data storage blocks of a fixed size, and each data storage block corresponds to a bit in the bitmap. Each bit represents whether a data storage block is occupied. For example, 1 indicates that the block is occupied, and 0 indicates that the block is free and available for allocation. Assuming that each data storage block is 1M in size, 1 bit can record whether 1M of hard disk space is used. 128K of memory space can manage 1TB of hard disk space. Therefore, using a bitmap can greatly save memory space. The Allocator finds free data storage blocks by scanning the bitmap, marks them as used, and when reclaiming, marks them as free again. This mechanism reduces fragmentation. Compared with the linked list structure, the bitmap can quickly locate continuous free areas without traversing the linked list structure. Through corresponding memory instructions, continuous free space can be found faster, especially when dealing with a large number of small blocks, the efficiency is higher.

[0047] In this embodiment, the allocation result of the bitmap is also temporarily stored in the memory and temporarily stored in the AllocatorLog without being immediately persisted to avoid causing metadata pollution.

[0048] The Allocator also includes a reference count table for recording how many ongoing IO operations there are for the occupied data storage blocks.

[0049] S102: Write data to the hard disk space pointed to by the allocated data storage location.

[0050] In a specific embodiment, the DataWriter is used to write data to the disk location corresponding to the data storage location, and Direct I / O or O_DSYNC is used to ensure that the data is written to the disk.

[0051] Since this space is independent, a power failure at this time causing the write operation to fail does not affect any other existing data. If the write fails due to reasons such as disk damage, the chunk_pos is discarded and marked as invalid at the same time, and no metadata update is triggered.

[0052] S103: Generate corresponding metadata and persist it together with the event of allocating data storage bits into the metadata storage.

[0053] In a specific embodiment, it is necessary to persist the event of allocating data storage bits and the metadata of the corresponding generated data storage block into the metadata storage. The metadata includes, for example, the location of the data storage block, data length, hash value, version information, etc. The metadata storage refers to a specific independent area on the hard disk for storing metadata information and bitmap. That is to say, it is necessary to persist the information that the data storage bit is marked as allocated, that is, the change of the data storage bit reflected in the bitmap in memory, into the metadata storage. At the same time, the metadata changes caused by the allocation / reclamation operation also need to be persisted into the metadata storage.

[0054] In an embodiment, the process of jointly persisting the data storage bit allocation event and metadata change in S103 can rely on the WriteBatch of RocksDB to form an atomic update. The WriteBatch is a mechanism of RocksDB for batch write operations. It allows multiple write operations, such as insert, update, delete, to package the batch write operations into a logical unit and atomically apply them to the database through a single commit operation, thus ensuring that all operations are either all successful or all failed, guaranteeing data consistency.

[0055] In this embodiment, if the operation is successful, the allocator updates the memory state to "committed" and releases the temporary records in the allocator log. If the operation fails (such as RocksDB crashing), roll back the memory allocation state through the allocator log to ensure that the data storage bits can be reallocated. The entire write process is either successful or failed, and there is no intermediate state.

[0056] In another embodiment of the present invention, a method for deleting data management in a large-value data storage space is provided, as Figure 2 shown, specifically including the following steps:

[0057] S201: Quickly load all allocation information in memory.

[0058] In a specific implementation, the allocator quickly loads the bitmap from the metadata storage on the hard disk during startup, which includes the allocation information of the data storage bits.

[0059] S202: Persist the deletion mark in the metadata storage.

[0060] In a specific embodiment, after a deletion request is issued, the metadata store can be atomically updated through RocksDB's WriteBatch. Specifically, it is necessary to persist the target chuck deletion flag corresponding to the bitmap in the metadata store and modify the corresponding metadata at the same time. It should be noted that only the hard disk metadata store is modified at this stage, and the memory state or physical data is not involved. If this stage fails (such as a crash), it will roll back after restart, and the bitmap and metadata will remain in their original states.

[0061] Therefore, when a data storage block deletion event is written to disk, as long as there is a reference to the data storage block in memory, the data in the data storage block is still readable.

[0062] S203: Determine whether the read / write operation of the data storage block has ended. If it has ended, proceed to the next step S204. If it has not ended, delay the recycling and wait for the read / write to complete.

[0063] In a specific embodiment, the method for determining whether the read / write operation of the data storage block has ended is to determine whether the reference count in the reference count table is 0. The reference count of the data storage block is the number of I / O operations for reading and writing the current data block. A reference count of 0 indicates that the current read / write operation on the data block has ended. If the reference count is not 0, it means that the current data block is still being read and written, and the recycling is delayed and waiting for the read / write to complete.

[0064] In a specific embodiment, a reference Arc to a location voucher is assigned to the data storage bit <chunkpos>Permissions can achieve conflict-free reading and writing of data storage blocks. The read operation of the data storage block first obtains a reference Arc to the chunk <chunkpos>, this chunk is marked as in use by reference. Before the read operation is completed, as long as the reference is not released, concurrent write or delete operations can still be read normally.

[0065] S204: Modify the allocation of data storage blocks in memory and release the corresponding data storage blocks back to the resource pool.

[0066] In one embodiment, after the read and write are completed, modify the allocation of bitmap data storage blocks in memory and release the deleted data blocks.

[0067] Specifically, after the reference count of the data storage block waits until it is 0, the allocator marks the allocation status of the data storage block in the memory bitmap as 0 and recycles it to the resource pool, indicating that the data block is free and can be reallocated.

[0068] In another embodiment of the present invention, a method for managing the addition of data to a large-value data storage space is provided, as Figure 3 shown, specifically including the following steps:

[0069] S301: Modify the reference count table in memory.

[0070] The reference count table in the allocator's memory records how many ongoing IO operations there are for the occupied data storage blocks. In a specific embodiment, first obtain the reference Arc of the data storage location voucher <chunkpos>Permissions are obtained, and the reference count table corresponding to the data storage bit in memory is modified to increase the reference count. This represents the current IO operation on the data storage block.

[0071] S302: Directly append data to the corresponding hard disk space.

[0072] In a specific embodiment, after obtaining the reference of the data storage bit, an append operation is directly performed behind the original data in the hard disk space.

[0073] S303: Generate corresponding metadata and persist it to the metadata storage.

[0074] In a specific embodiment, it is necessary to persist the metadata corresponding to the updated data storage block to the metadata storage. The metadata includes, for example, the location, data length, hash value, version information, etc. of the data storage block.

[0075] In the prior art, if the method of separating the modification of the memory state of the data storage block allocation / reclamation from the persistence of the allocation / reclamation event is not adopted, then in order not to affect the read operation, Copy-on-write is usually adopted, that is, the original data is first copied to the memory, the appended part is merged into the memory, and then the whole is overwritten to the disk space. Compared with the prior art, the present embodiment adopts the method of directly appending data at the end of the data storage block, which reduces the copy of the original data in the memory, makes the append operation more efficient, and at the same time the append process is not affected by the read operation and does not affect the original data.

[0076] As Figure 4 shown, the present invention also provides a system for managing a large-value data storage space for implementing the above method. The system includes an allocator and a metadata storage. The allocator is located in the memory and is used to manage the memory state of the data storage block. The metadata storage is located in the hard disk and is used to implement the metadata storage of the data storage block and the persistence of the change events of the data storage block state (allocation, reclamation). The separation of the process of changing the memory state of the data storage block and the persistence of the data block state change event is realized.

[0077] In a specific embodiment, the allocator dynamically generates a logical location identifier in the memory, which points to a newly allocated data storage block on the disk and is used to accurately record the starting position of data writing. As Figure 5 shown, the allocator includes a bitmap and a reference count table. The allocator manages the data storage bits through the bitmap and the reference count table in the memory. The bitmap is usually used to track the occupancy status of the storage block. The storage space in the hard disk is divided into data storage blocks of a fixed size, and each data storage block corresponds to a bit in the bitmap. Each bit represents whether a data storage block is occupied.

[0078] The metadata is stored in a specific independent area on the hard disk, storing the metadata for data storage blocks and implementing the persistence of events for allocating / reclaiming data storage blocks. The metadata refers to, for example, the location, data length, hash value, version information, etc. of the data storage blocks.

[0079] In a specific embodiment, it is necessary to persist the events of allocating / reclaiming data storage bits and the generated metadata of the corresponding data storage blocks to the metadata storage. The metadata includes, for example, the location, data length, hash value, version information, etc. of the data storage blocks. The metadata storage refers to a specific independent area on the hard disk for storing metadata information and bitmap. That is to say, it is necessary to persist the information indicating that the data storage bits are marked as allocated, that is, the changes in the data storage bits reflected by the bitmap in the memory, to the metadata storage; at the same time, it is also necessary to persist the metadata changes caused by the allocation / reclamation operations to the metadata storage.

[0080] In a specific implementation, the persistence process of the data storage bit allocation / reclamation events and metadata changes together can rely on the WriteBatch of RocksDB to form an atomic update.

[0081] The above are the specific embodiments of the present invention and the technical principles applied. If changes are made according to the concept of the present invention and the functions and effects generated therefrom do not exceed the spirit covered by the description and drawings, they should still fall within the protection scope of the present invention.< / chunkpos> < / chunkpos> < / chunkpos>

Claims

1. A method for managing large-value data storage space, characterized in that It includes the following steps: Manage the state in the memory of the data storage block; Implement the persistence of the data storage block state change event in the hard disk; The change of the memory state of the data storage block is separated from the process of persisting the change event in the hard disk.

2. The method for managing a large-value data storage space according to claim 1, wherein The state in the memory of the data storage block is managed by an allocator.

3. A method for managing a large-value data storage space according to any one of claims 1-2, characterized in that The persistence of the data storage block state change event is implemented in the metadata storage in the hard disk; changes to the metadata involved in the process of changing the data storage block also need to be persisted in the metadata storage.

4. A method for large-value data storage space management according to any one of claims 1-3, characterized in that The process of jointly persisting the data storage block state change event and the metadata change adopts atomic update.

5. A method for managing a large-value data storage space according to any one of claims 1-4, characterized in that The allocator includes a bitmap, and one bit in the bitmap corresponds to a data storage block, indicating whether the corresponding data storage block is occupied.

6. A method for managing a large-value data storage space according to any one of claims 1-5, characterized in that The allocator also includes a reference count table, which is used to record how many ongoing IO operations there are on the occupied data storage block.

7. A method for managing a large-value data storage space according to any one of claims 1-6, characterized in that The allocator manages and allocates data storage bits in the memory, and is responsible for modifying the memory state of the data storage block; the data storage bit is a logical position identifier dynamically generated by the allocator in the memory, and is used to record the starting position of data writing.

8. A method for managing a large-value data storage space according to any one of claims 1-7, characterized in that The method for managing the large-value data storage space includes writing data, specifically including the following steps: (101) In the memory, allocate a new data storage bit and point it to the corresponding hard disk space; (102) Write data to the hard disk space pointed to by the allocated data storage bit; (103) Generate corresponding metadata and persist it together with the event of allocating the data storage bit into the metadata storage.

9. A method for large-value data storage space management according to any one of claims 1-7, characterized in that The method for managing the large-value data storage space includes deleting data, specifically including the following steps: (201) Quickly load all allocation information in the memory; (202) Persist the deletion mark in the metadata storage; (203) Determine whether the read and write operations of the data storage block are completed. If completed, proceed to the next step (204). If not completed, delay the recovery and wait for the read and write to complete; (204) Modify the allocation of the data storage block in the memory and release the corresponding data storage block back to the resource pool.

10. A method for large-value data storage space management according to any one of claims 1-7, characterized in that The method for managing the large-value data storage space includes appending data, specifically including the following steps: (301) Modify the reference count table in the memory; (302) Directly append data to the corresponding hard disk space; (303) Generate corresponding metadata and persist it into the metadata storage.

11. A system for managing large-value data storage space, characterized in that It includes: Allocator, metadata storage; The allocator is located in the memory and is used to implement the modification of the memory state of the data storage block; The metadata storage is located in the hard disk and is used to implement the metadata storage of the data storage block and the persistence of the data storage block state change event; the change of the memory state of the data storage block and the process of persisting the data storage block state change event are separated.

Images