A block-based KV storage method and system based on configurable MMU

Abstract

The present invention discloses a block-type KV storage method and system based on a configurable MMU, and relates to the technical field of data storage. The method comprises: selecting and configuring the NSID type required in the MMU, determining the corresponding section size and Hash type; determining the size of the Hash table according to the Hash type, and calculating the number of sections; allocating a pointer to each section, and using a memory block to store pointers corresponding to different sections; submitting all parameters through the MMU configuration interface, and the hardware automatically manages the mapping relationship between the Hash table and the physical address space according to the configured parameters, and dynamically adjusts and optimizes. The present invention realizes flexible management of the physical address space while improving the system read and write performance by dynamically configuring and optimizing the NSID and its corresponding section.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a block-type KV storage method and system based on a configurable MMU. Background Art

[0002] KV storage, or Key-Value Store, is a very simple and efficient way to store data. It stores data in the form of key-value pairs, where each key is unique, similar to an "entry" in a dictionary, and each value is the data associated with this key. KV storage is widely used in scenarios where large amounts of data need to be quickly accessed, such as cache systems (such as Redis, Memcached), databases (such as Cassandra, DynamoDB), etc.

[0003] KV storage technology mainly relies on two implementation methods: LSM-tree (Log-Structured Merge-tree) and Hash table (Hashtable). LSM-tree optimizes write performance by merging data at different levels, but reading may involve multiple levels of search, resulting in high read latency. In addition, the implementation of LSM-tree is usually more complex and consumes relatively large system resources. Hash table mainly solves the problem of search efficiency, and its implementation method is relatively easy. Therefore, the existing KV-SSD (key-value solid-state drive) is mainly implemented based on Hash table.

[0004] In traditional hash tables, different data are sometimes assigned to the same location, which is called a "hash conflict". When a hash conflict occurs, the conflicting data is usually stored in a chain at the same location. This is called the "chain address method", that is, a linked list is established for each hash conflict. When the amount of data is large, these chains will become very long. For example, for a hash table of size m, there are m continuous chains from 0 to (m-1), which slows down data search and reduces system performance. At the same time, the data in the hash table is usually fixedly stored in a certain type of storage medium, such as DRAM with fast speed but small capacity or TLC with large capacity but slow speed. For different types of storage media (such as DRAM, SLC, MLC, TLC, etc.), a single hash table storage strategy is difficult to meet the diverse performance requirements of high speed and large capacity at the same time.

[0005] In addition, existing KV storage systems mainly rely on software to manage and update hash tables, which will occupy a large amount of host resources. Especially when the amount of data is large, the host's computing pressure will increase significantly, resulting in performance degradation. During data storage, storage space is often fragmented, that is, data is stored in discontinuous storage blocks, resulting in reduced space utilization, a large amount of storage space wasted, and slow data search. Summary of the invention

[0006] In view of the limitations of the chain address method in Hash conflicts, the inability of a single fixed storage path to simultaneously take into account the requirements of high performance and large capacity, the software-implemented KV storage method leading to system performance degradation, and the lack of effective utilization of fragmented space, the present invention proposes a block-based KV storage method and system based on a configurable MMU, which combines hardware and MMU technology to improve system performance and optimize the utilization of storage space.

[0007] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0008] A block-based KV storage method based on a configurable MMU, the method comprising:

[0009] Select and configure the required NSID type in the MMU according to the system read / write performance and storage requirements;

[0010] Determine the section size and hash type corresponding to the NSID type according to the NSID type;

[0011] Determine the size of the Hash table according to the Hash type, and calculate the number of sections based on the size of the Hash table and the size of the section;

[0012] In the MMU configuration interface, a pointer is allocated to each section, and a memory block is used to store pointers corresponding to different sections. The pointer points to the actual location in the physical address space and is used to store the section data.

[0013] After the configuration is completed, all parameters are submitted through the MMU configuration interface, and the hardware automatically manages the mapping relationship between the Hashtable and the physical address space according to the configured parameters;

[0014] During system operation, the hardware is dynamically adjusted and optimized based on real-time read and write operation requirements.

[0015] A further improvement of the present invention is that the pointers of all sections corresponding to the same NSID are stored in the continuous address range of the same memory block. The system first automatically configures the memory address where the pointer of the first section corresponding to the NSID is located, and then automatically calculates and configures the memory addresses of the pointers of the remaining sections based on the memory address.

[0016] A further improvement of the present invention is that the hardware automatically manages the mapping relationship between the hash table and the physical address space according to the configured parameters, specifically including: performing hash calculation according to the key of the key-value pair to determine the position of the key in the hash table, and then the hardware finds the pointer of the corresponding section through the MMU, locates the specific position of the physical address space according to the pointer, and performs corresponding add, delete, modify, and query operations.

[0017] A further improvement of the present invention is that the MMU can be configured with multiple NSIDs, each NSID is configured with multiple sections, and each section corresponds to a pointer; the storage position of each section under the same NSID in the physical address space is continuous, and the storage positions of different sections under the same NSID in the physical address space are discrete and non-continuous.

[0018] A further improvement of the present invention is that the dynamic adjustment and optimization includes:

[0019] The MMU can be used for dynamic configuration of multiple NSIDs, including changing the size of the section corresponding to the NSID, reallocating pointers for the adjusted section, and implementing dynamic changes of the corresponding hash table through hardware acceleration.

[0020] A further improvement of the present invention is that the dynamic adjustment and optimization also includes the MMU being able to further adjust the physical address space by dynamically changing the pointer corresponding to the section, including adjusting the position of the section within the same storage medium or migrating the section from one storage medium to another storage medium, and performing the conversion from the old Hashtable to the new Hash table through hardware acceleration, and the conversion is achieved by configuring the enablement of the Rehash command.

[0021] A further improvement of the present invention is that the enabling of the Rehash command is implemented by configuring the following steps:

[0022] Lock the current Hash table and prohibit new read and write operations;

[0023] Generate a new Hash table based on the new section distribution. After the addresses of all sections are reallocated, release the lock and enable the new Hash table for subsequent key-value pair management.

[0024] A block-type KV storage system based on a configurable MMU, the system comprising:

[0025] The MMU unit is used to automatically manage the mapping relationship between the hash table and the physical address space according to the configured parameters, and dynamically adjust and optimize according to the real-time read and write operation requirements;

[0026] An MMU configuration interface, which is used to select and configure the required NSID type according to the system's read and write performance and storage requirements, and determine the section size and hash type based on the NSID type, and then configure the hash table size, allocate section pointers, and map the pointers to actual locations in the physical address space;

[0027] Multiple storage units, used to store block-type KV data, where each storage unit includes multiple sections;

[0028] A Hash table is used to determine the position of a key in a Hash table based on the key in a key-value pair.

[0029] The beneficial effects of the present invention are as follows: based on the configurable MMU mapping relationship, the Hash table is discretized and mapped to different areas, which can make full use of the fragmented space and expand the new storage space. At the same time, the dynamic change of the MMU mapping table is realized through hardware acceleration to reduce the pressure on the host. By managing the Hash table in sections, the overhead of the mapping table is reduced, and the continuous address interval is fragmented to more effectively utilize the discrete space and improve the space utilization. By dynamically configuring and optimizing the NSID and its corresponding section, while improving the system's read and write performance, the flexible management of the physical address space is achieved, which not only efficiently manages the mapping between the Hash table and the physical address, but also supports advanced operations such as Rehash, which significantly enhances the flexibility and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0031] Figure 1 is a flow chart of the method of the present invention;

[0032] Figure 2 is a MMU mapping relationship diagram in an embodiment of the present invention;

[0033] Figure 3 4 is a system module diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0035] Example 1: Figure 1 As shown, this embodiment provides a block-based KV storage method based on a configurable MMU, comprising the following steps:

[0036] S1: Select and configure the required NSID type in the MMU according to the system read / write performance and storage requirements;

[0037] MMU: Memory Management Unit, also known as Memory Management Unit in Chinese, is a computer hardware responsible for processing memory access requests from the central processing unit (CPU);

[0038] NSID: Namespace Identifier, an identifier used to identify different storage spaces during memory mapping. By configuring the NSID type, the storage space can be divided into different sections (i.e., the area blocks into which you want to divide the hash table), and each section is mapped to a certain part of the actual physical storage medium.

[0039] Step S1 involves analyzing the current and expected workloads of the system to determine the best NSID configuration. For example, in a high-performance database system, if a large number of concurrent read and write requests are expected, then it may be necessary to select an NSID type that supports high concurrent access.

[0040] S2: Determine the corresponding section size and hash type according to the NSID type;

[0041] The section size determines how much data each section can hold, while the hash type (i.e., the hash algorithm used to map keys to storage locations for fast data lookup and storage, including MD5, SHA-1, SHA-256, CRC32, etc.) affects the data distribution and access efficiency. For a KV storage system that requires fast lookup, a hash function with a low conflict rate may be selected;

[0042] The NSID type determines the size and allocation of the hash table. Each NSID type corresponds to one or more sections, and the size of each section needs to be determined based on the requirements. If you need to divide a large hash table into smaller, easier-to-manage blocks, you can choose a smaller section size; for example, for high-performance applications, select CRC_32 as the hash type and set the section size to 2M.

[0043] S3: Determine the size of the hash table according to the hash type, and calculate the number of sections based on the size of the hash table and the size of the section. For example, if the size of the hash table is 16M and the size of each section is 4M, then 4 sections are required.

[0044] Through section management, storage space can be flexibly mapped to improve resource utilization.

[0045] S4: In the MMU configuration interface, a pointer is allocated to each section, and a memory block is used to store pointers corresponding to different sections. The pointer points to the actual location in the physical address space (i.e., the Physical Address Space) for storing section data.

[0046] Specifically, the pointers of all sections corresponding to the same NSID are stored in the continuous address range of the same memory block. The system first automatically configures the memory address of the pointer of the first section corresponding to the NSID, and then automatically calculates and configures the memory addresses of the pointers of the remaining sections based on the memory address.

[0047] For example, there is an NSID that is divided into 4 sections, and the pointers of these sections are stored in the memory at addresses 0x0 to 0xc. If the pointer of the first section is stored at 0x0, the system only needs to configure the address 0x0, and the pointer addresses of other sections can be 0x4, 0x8, and 0xc in sequence. By configuring continuous memory addresses, each section can be effectively managed and quickly accessed, improving the efficiency of storage operations and simplifying the complexity of configuration.

[0048] S5: After the configuration is completed, all parameters are submitted through the MMU configuration interface, including NSID, section size, Hash type, number of sections, pointers, physical address space, memory address, etc. The hardware gradually adds key-value pairs to the Hash table according to these configured parameters, and automatically manages the mapping relationship between the Hash table and the physical address space;

[0049] Step S5 specifically includes: performing a hash calculation according to the key of the key-value pair to determine the position of the key in the hash table, and then the hardware finds the pointer of the corresponding section through the MMU, locates the specific position of the physical address space according to the pointer, and performs the corresponding add, delete, modify, and query operations;

[0050] MMU can configure multiple NSIDs, each NSID configures multiple sections, and each section corresponds to a pointer; the storage location of each section under the same NSID in the physical address space is continuous, and the storage locations of different sections under the same NSID in the physical address space are discrete and non-continuous. For example, the size of the hash table corresponding to NSID0 is configured to be 1000, that is, there are 1000 continuous chains, from 0 to 999. If the section size configured for NSID0 is 100, then there are a total of 10 sections, that is, there are 10 pointers pointing to different locations. In this way, the continuous 1000 chains are divided into 10 discrete blocks, each of which is 100 continuous chains.

[0051] like Figure 2 As shown, NSID is Namespace identifier, section is the area block into which you want to divide the hash table, pointer is the starting position pointer of the real storage medium mapped to the section, and Physical AddressSpace refers to the real storage medium interval.

[0052] Nsidx(x=0,1,…,p-1): different NSIDs that can be supported;

[0053] sectiony (y=0,1,…,n-1): different sections divided in each NSID;

[0054] pointerz(z=0,1, …,n-1): different pointers configured in different sections;

[0055] 200K space: The size of a section is 200k.

[0056] S6: During system operation, the hardware is dynamically adjusted and optimized according to real-time read and write operation requirements;

[0057] The system monitors changes in storage requirements in real time, such as data growth causing section fullness or system performance degradation. When conditions that require adjustment are detected (such as insufficient section size or the need to optimize storage efficiency), the system triggers a dynamic adjustment process. Based on current demand and load, the system calculates the new size of the corresponding section under each NSID. Based on the new section size, the system plans how to reallocate the pointers to each section to ensure efficient use of storage space.

[0058] The system modifies the section size parameters corresponding to the NSID through the MMU configuration interface. The hardware automatically adjusts the section layout in the physical address space according to the new configuration and reallocates pointers for each adjusted section. These pointers will point to the new physical address space to accommodate the adjusted sections.

[0059] According to the new section size, the system may need to recalculate and update the mapping relationship in the Hash table (i.e., Rehash). This step ensures that the mapping relationship between the Hash table and the physical address space is still accurate. The hardware uses the built-in acceleration function to quickly complete the conversion from the old Hash table to the new Hash table to reduce the impact on system performance. After confirming that all adjustment operations are completed, the system resumes normal read and write operations on the relevant sections, and writes the data buffered during the adjustment to the new section location to ensure data consistency.

[0060] Further, step S6 includes:

[0061] MMU can be used for dynamic configuration of multiple NSIDs, including changing the size of the section corresponding to the NSID, reallocating pointers for the adjusted section, and dynamically changing the corresponding hash table through hardware acceleration; for example, in the example mentioned above, the configured NSID0 corresponds to a hash table of size 1000, and the section size can be changed from 100 to 200, so that the section size of NSID1 can be configured to be 200, and the pointers corresponding to each section can be configured. After configuration, the real hash table changes can be realized by hardware.

[0062] It also includes that the MMU can further adjust the physical address space by dynamically changing the pointer corresponding to the section, including adjusting the position of the section within the same storage medium (such as adjusting different positions within DRAM) or migrating the section from one storage medium to another storage medium (such as changing from DRAM to SLC or MLC, etc.), and performing the conversion from the old hash table to the new hash table through hardware acceleration, and the conversion is achieved by configuring the enablement of the Rehash command.

[0063] Example 2

[0064] S10: Configure the section size of NSID0 in MMU to 4M, and the Hash type to CRC_24, that is, the size of the Hash table is 16M, then the number of sections corresponding to NSID0 is 4;

[0065] S20: Select a memory with a width of 32 bits and a depth of 20, and write four 32-bit pointers in sequence from the starting position, as shown in the following table:

[0066] Table 1 Memory address and pointer mapping table configured by NSID0

[0067] memory address pointer 0x0 0x0 0x4 0x1000_0000 0x8 0x2000_0000 0xc 0x3000_0000

[0068] S30: continuously send add KV commands to the hardware module, and the hardware builds a Hash table according to the parameters configured in steps S10 and S20;

[0069] S40: Due to the change in storage space, the actual storage space of NSID0 needs to be changed, that is, the mapping relationship in MMU is changed, the section size of NSID1 in MMU is configured to be 4M, the Hash type is CRC_24, and four new pointers are written in sequence in the continuous positions avoiding NSID0 in memory, as shown in the following table:

[0070] Table 2 Memory address and pointer mapping table configured by NSID1

[0071] memory address pointer 0x100 0x5000_0000 0x104 0x6000_0000 0x108 0x7000_0000 0x10c 0x8000_0000

[0072] S50: After configuring step S40, the enable of the Rehash command is sent to the hardware, and the hardware moves the hash table of NSIDO0 in the actual storage space to the space location specified by NSID1 according to the configuration of NSID0 and NSID1 in the above steps.

[0073] Example 3: Figure 3 As shown, this embodiment provides a block-based KV storage system based on a configurable MMU, including:

[0074] The MMU unit is used to automatically manage the mapping relationship between the hash table and the physical address space according to the configured parameters, and dynamically adjust and optimize according to the real-time read and write operation requirements;

[0075] An MMU configuration interface, which is used to select and configure the required NSID type according to the system's read and write performance and storage requirements, and determine the section size and hash type based on the NSID type, and then configure the hash table size, allocate section pointers, and map the pointers to actual locations in the physical address space;

[0076] Multiple storage units, used to store block-type KV data, where each storage unit includes multiple sections;

[0077] A Hash table is used to determine the position of a key in a Hash table based on the key in a key-value pair.

[0078] In summary, the present invention discretizes the Hash table and maps it to different areas based on the configurable MMU mapping relationship, which can make full use of the fragmented space and expand the new storage space. At the same time, the dynamic change of the MMU mapping table is realized through hardware acceleration to reduce the pressure on the host. By managing the Hash table in sections, the overhead of the mapping table is reduced, and the continuous address interval is fragmented to more effectively utilize the discrete space and improve the space utilization. By dynamically configuring and optimizing the NSID and its corresponding section, while improving the read and write performance of the system, the flexible management of the physical address space is realized, which not only efficiently manages the mapping between the Hash table and the physical address, but also supports advanced operations such as Rehash, which significantly enhances the flexibility and scalability of the system.

[0079] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0080] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0081] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules.

[0082] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A block-based KV storage method based on a configurable MMU, characterized in that: The method comprises: Select and configure the required NSID type in the MMU according to the system read / write performance and storage requirements; Determine the section size and hash type corresponding to the NSID type according to the NSID type; Determine the size of the Hash table according to the Hash type, and calculate the number of sections based on the size of the Hash table and the size of the section; In the MMU configuration interface, a pointer is allocated to each section, and a memory block is used to store pointers corresponding to different sections. The pointer points to the actual location in the physical address space and is used to store the section data. After the configuration is completed, all parameters are submitted through the MMU configuration interface, and the hardware automatically manages the mapping relationship between the Hashtable and the physical address space according to the configured parameters; During system operation, the hardware is dynamically adjusted and optimized based on real-time read and write operation requirements.

2. According to the block-based KV storage method based on configurable MMU according to claim 1, it is characterized in that: The pointers of all sections corresponding to the same NSID are stored in the continuous address range of the same memory block. The system first automatically configures the memory address of the pointer of the first section corresponding to the NSID, and then automatically calculates and configures the memory addresses of the pointers of the remaining sections based on the memory address.

3. The block-based KV storage method based on a configurable MMU according to claim 1, characterized in that: The hardware automatically manages the mapping relationship between the hash table and the physical address space according to the configured parameters, specifically including: performing hash calculation according to the key of the key-value pair to determine the position of the key in the hash table, and then the hardware finds the pointer of the corresponding section through the MMU, locates the specific position of the physical address space according to the pointer, and performs corresponding add, delete, modify, and query operations.

4. The block-based KV storage method based on a configurable MMU according to claim 3 is characterized in that: The MMU can be configured with multiple NSIDs, each NSID is configured with multiple sections, and each section corresponds to a pointer; the storage location of each section under the same NSID in the physical address space is continuous, and the storage locations of different sections under the same NSID in the physical address space are discrete and non-continuous.

5. The block-based KV storage method based on a configurable MMU according to claim 1, characterized in that: The dynamic adjustment and optimization include: The MMU can be used for dynamic configuration of multiple NSIDs, including changing the size of the section corresponding to the NSID, reallocating pointers for the adjusted section, and implementing dynamic changes of the corresponding hash table through hardware acceleration.

6. The block-based KV storage method based on configurable MMU according to claim 5, characterized in that: The dynamic adjustment and optimization also includes that the MMU can further adjust the physical address space by dynamically changing the pointer corresponding to the section, including adjusting the position of the section within the same storage medium or migrating the section from one storage medium to another storage medium, and performing the conversion from the old hash table to the new hash table through hardware acceleration, and the conversion is achieved by configuring the enablement of the Rehash command.

7. The block-based KV storage method based on a configurable MMU according to claim 6, characterized in that: The configuration of enabling the Rehash command includes the following steps: Lock the current Hash table and prohibit new read and write operations; Generate a new Hash table based on the new section distribution. After the addresses of all sections are reallocated, release the lock and enable the new Hash table for subsequent key-value pair management.

8. A storage system based on a block-based KV storage method based on a configurable MMU according to any one of claims 1 to 7, characterized in that: The system comprises: The MMU unit is used to automatically manage the mapping relationship between the hash table and the physical address space according to the configured parameters, and dynamically adjust and optimize according to the real-time read and write operation requirements; An MMU configuration interface, which is used to select and configure the required NSID type according to the system's read and write performance and storage requirements, and determine the section size and hash type based on the NSID type, and then configure the hash table size, allocate section pointers, and map the pointers to actual locations in the physical address space; Multiple storage units, used to store block-type KV data, where each storage unit includes multiple sections; A Hash table is used to determine the position of a key in a Hash table based on the key in a key-value pair.

Images