Hardware implementation method for optimizing KV storage performance based on configurable logic

Abstract

The present invention discloses a hardware implementation method for optimizing KV storage performance based on configurable logic, including the following hardware implementation process: using CRC as a HASH function, and supporting 17 options, namely CRC16 to CRC32; using a chain address method to solve the HASH conflict problem, and the same chain is arranged from small to large according to the size of the key; adjusting the number of entries in the HEAD ZONE by adjusting the HASH function; wherein Key_info_buf is a unit containing a key and a pointer to a value. The present invention is a study based on a new KV storage form, KV-SSD, based on the implementation of HASH, hardware acceleration is achieved through a configurable logic, and the read and write performance can be adjusted in real time according to actual applications.

Description

Technical Field

[0001] The present invention relates to the technical field of hardware for optimizing storage performance, and in particular to a hardware implementation method for optimizing KV storage performance based on configurable logic. Background Art

[0002] At present, the main implementations of KV storage are LSM-tree (Log-Structured Merge-tree, a data structure commonly used in disk storage systems, especially in key-value storage and database management systems) and HASH (HashFunction, a method of creating a small digital "fingerprint" from any kind of data). HASH-based KV-SSD (Key-Value Store Solid State Drive, a solid-state storage device designed specifically for key-value storage data structures) is widely used in key-value storage systems as a new type of storage device.

[0003] In actual storage read and write applications, software implementation will result in high CPU usage; in hardware implementation, the HASH type is pre-set, and adjustment after actual application is not supported or the operation is complicated, resulting in an increase in the probability of HASH conflicts or a decrease in read and write performance, which brings inconvenience to actual applications. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above-mentioned problems existing in the existing KV-SSD read-write storage hardware implementation, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is to solve the problem that the existing KV-SSD read-write storage hardware does not support adjustment after actual application or the operation is complicated, resulting in an increased probability of HASH conflict or a decrease in read-write performance.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: a hardware implementation method for optimizing KV storage performance based on configurable logic, including the following hardware implementation process: using CRC as the HASH function, and supporting 17 options, namely CRC16 to CRC32; using the chain address method to solve the HASH conflict problem, and the same chain is arranged from small to large according to the size of the key; adjusting the number of entries in the HEAD ZONE by adjusting the HASH function; wherein, Key_info_buf is a unit containing a key and a pointer to a value; wherein, the HEAD ZONE is divided into two areas, the Head Zone and the Rand Zone, in order to distinguish the first Key_info_buf of each chain in the HASH TABLE from other Key_info_bufs on the chain, namely, the Head Zone refers to the area composed of the first Key_info_buf of each chain in the HASH TABLE; the Rand Zone Zone refers to the area composed of Key_info_buf except the first Key_info_buf of each chain; wherein, the HASH type is adjusted at any time to adjust the length of each chain, thereby adjusting the read and write performance at any time; wherein, the hardware implementation method includes the following implementation steps: S1: preliminarily estimate a HASH type based on the total amount of KV and expected performance of the namespace; S2: select the HASH type; S3: as the actual KV data is filled into the kv_hash_table, compare the performance and space actually calculated at this time with the performance and space estimated in step S1; S4: select a specific CRC type according to the selection rules of performance and space, and repeat step S2 until the performance and space are optimal; wherein, when selecting a specific CRC type according to the selection rules of performance and space, when the performance and space estimated in step S1 are higher than the performance and space actually calculated in step S3, select a CRC type smaller than step S1, and when the performance and space estimated in step S1 are lower than the performance and space actually calculated in step S3, select a CRC type larger than step S1.

[0008] The beneficial effects of the present invention are as follows:

[0009] 1. Reduce dependence on CPU through hardware implementation;

[0010] 2. Improved read and write performance through hardware acceleration;

[0011] 3. Multiple optional HASH types reduce space waste according to actual applications;

[0012] 4. Multiple optional HASH types can ensure controllable performance;

[0013] 5. Adjust the HASH type at any time according to the actual application background hardware to ensure the best performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0015] Figure 1 This is a schematic diagram of the Kv_hash_table provided by the present invention.

[0016] Figure 2 This is a schematic diagram of the old table provided by the present invention.

[0017] Figure 3 The present invention provides Figure 2 Schematic diagram of the new table resulting from the changes.

[0018] Figure 4 The present invention provides a schematic diagram of the implementation process. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0020] It should be noted in advance that HASH refers to a hash algorithm, also known as a hash algorithm or a hash function. A hash algorithm is an algorithm that converts input data of any length into output data of a fixed length. In the field of computer science, hash algorithms are widely used in data encryption, data integrity verification, data compression and other fields.

[0021] The hash algorithm has the following characteristics:

[0022] 1. The length of input data can be arbitrary, but the length of output data is fixed.

[0023] 2. Different input data will produce different output data through hash algorithm.

[0024] 3. The hash algorithm is a one-way function, that is, the corresponding output data can be calculated from the input data, but the original input data cannot be deduced from the output data.

[0025] 4. For the same input data, the output data obtained is fixed and is not affected by the environment or computer platform.

[0026] 5. Hash algorithms are efficient, meaning they have fast calculation speeds and can process large amounts of data.

[0027] Common hash algorithms include MD5, SHA-1, SHA-256, etc., which are widely used in data encryption, digital signatures, cryptography and other fields.

[0028] It should be noted in advance that KV-SSD (Key-Value Solid State Drive) is a data storage device based on a solid-state drive (SSD). It supports key-value pair storage and has the advantages of fast read and write, low power consumption, and low cost. It is widely used in data storage, cloud computing, the Internet of Things, and other fields. KV-SSD uses flash storage media, which has extremely high read and write speeds and durability, and can provide higher data storage performance and reliability than traditional mechanical hard drives. At the same time, KV-SSD can also achieve data sharding and load balancing through a distributed architecture, improving the overall performance and reliability of the system.

[0029] As a new type of storage device, HASH-based KV-SSD is widely used in key-value storage systems. In actual storage read-write applications, software implementation will result in high CPU usage; while in hardware implementation, the HASH type is pre-set, and it is not supported or complicated to adjust after actual application, resulting in an increase in the probability of HASH conflicts or a decrease in read-write performance, which brings inconvenience to actual applications.

[0030] Therefore, the present invention provides hardware for optimizing KV storage performance based on configurable logic, including the following hardware implementation process:

[0031] 1. Use CRC as the HASH function and support 17 options, namely CRC16~CRC32;

[0032] It should be noted that CRC is the abbreviation of Cyclic Redundancy Check, which is a commonly used error detection check code used to detect errors that may occur during data transmission. CRC uses the polynomial remainder method to implement verification. The check code is generated by XOR operation of the polynomial and the data block, and then the check code is modulo operation is performed to obtain the final verification result. The CRC algorithm can be applied to error detection during data transmission to ensure the integrity and reliability of the data. Different CRC algorithms may have different polynomial coefficients. Therefore, different CRC algorithms and polynomial coefficients may be selected for different application scenarios.

[0033] 2. Separate Chaining is used to solve the HASH conflict problem, and the same chain is arranged from small to large according to the size of the key. Please refer to Figure 1 ;

[0034] It should be noted that the Separate Chaining method is a method for resolving hash conflicts. In a hash table, when multiple keys are mapped to the same hash bucket, the Separate Chaining method stores these keys in the form of a linked list or other data structure, so that each hash bucket can store multiple key-value pairs. When a key-value pair needs to be found or inserted, the hash value of the key-value pair is first calculated, and then the corresponding hash bucket is found according to the hash value, and finally the corresponding key-value pair is found or inserted in the linked list of the hash bucket. Compared with other methods for resolving hash conflicts, the Separate Chaining method is simple and easy to implement, but it may cause space waste and reduce query efficiency. However, by setting a suitable hash function and an appropriate hash table size, the length of the linked list can be minimized, thereby improving the performance of the Separate Chaining method. The Separate Chaining method is often used to implement data structures such as hash tables, hash sets, and hash maps.

[0035] 3. Adjust the number of entries in the HEAD ZONE by adjusting the HASH function;

[0036] It should be noted that:

[0037] ①HEAD ZONE:

[0038] In some database systems, HEAD ZONE refers to the information area on a data block or page used to store metadata. This area usually contains a pointer to the actual location of the data, as well as control information for fast access and management.

[0039] In the file system, HEAD ZONE refers to the file header, which contains the file's metadata, such as file size, creation time, modification time, file owner, permission information, etc.

[0040] ②Entry:

[0041] In databases and data structures, an Entry usually refers to an entry in a data structure, which contains a key and its associated value. In a hash table, each Entry corresponds to a key-value pair, and the hash function is used to calculate the hash value of the key in order to quickly locate the address where the Entry is stored.

[0042] In a log file or transaction record, Entry refers to an entry in the record, which contains transaction status information, timestamp, operation type, etc.

[0043] It should be noted that different HASH functions determine different numbers of entries in the HEAD ZONE. Different lengths of a single chain determine different read and write performance. The longer the single chain, the worse the read and write performance. Therefore, the number of entries in the HEAD ZONE can be adjusted by adjusting the HASH function, which indirectly adjusts the statistical average number of each chain, and ultimately achieves the purpose of adjusting the read and write performance.

[0044] In the present invention, "HEAD ZONE" is mainly used to distinguish the first Key_info_buf of each chain in the HASH TABLE from other Key_info_bufs on the chain, and is divided into two zones, Head Zone and Rand Zone. That is, Head Zone refers to the zone composed of the first Key_info_buf of each chain in the HASH TABLE; Rand Zone refers to the zone composed of Key_info_bufs excluding the first Key_info_buf of each chain.

[0045] Among them, Key_info_buf is a unit containing key and pointer to value.

[0046] It should be noted that adjusting the number of entries in the HEAD ZONE by adjusting the HASH function means optimizing the structure of the hash table according to the needs of the application. Specifically, this can be achieved in the following ways:

[0047] a: Change the number of bits in the output of the hash function: If the number of entries in the HEAD ZONE needs to be increased, design a hash function that produces more different hash values. This usually involves changing the number of bits in the output of the hash function to create more unique locations in the same size address space.

[0048] b: Adjust the complexity of the hash function: By increasing the computational complexity of the hash function, the possibility of conflict is reduced, allowing more entries to be stored in the HEAD ZONE.

[0049] c: Use a dynamic hash table: In some cases, design a dynamic hash table whose size can be adjusted as needed. When the HEAD ZONE is full, the size of the HEAD ZONE can be expanded and all entries can be rehashed to maintain efficient access speed.

[0050] Specifically, the number of entries in the HEAD ZONE is adjusted by adjusting the HASH function: CRC is used as the HASH function, and the number of entries in the HEAD ZONE corresponding to different CRCs is different. For example, CRC16 corresponds to 2. 16 , CRC32 corresponds to 2 32 Through the configuration of the register, you can choose which CRC to adjust the number of entries in the HEAD ZONE. The existing chip technology selects a certain HASH function, or uses multiple HASHes to avoid HASH conflicts. The present invention uses the chain address method to solve the HASH conflict, and can adjust the number of entries in the HEAD ZONE with 17 configurable HASH functions.

[0051] Furthermore, the statistical average number of each chain is adjusted indirectly, and the purpose of adjusting the read and write performance is finally achieved: hardware is used to implement it (called REHASH operation). Through CPU configuration, the old HASH table is given, and then the HASH function type to be used in the new HASH table and the location to be created are specified. Then the hardware reads the old HASH Table and writes the new HASH Table by itself. After all the old data is read and the new data is written, the completion signal is notified to the CPU.

[0052] Specifically, the present invention can adjust the HASH type at any time according to actual conditions, thereby adjusting the length of each chain, and ultimately achieving the purpose of adjusting the read and write performance at any time. Figure 2 and Figure 3 As shown, an old HASH table is changed to a new HASH table to achieve the read / write performance and space size that meet your needs.

[0053] It should be noted that the HASH type can be adjusted at any time according to the actual situation. For example, before creating the old HASH Table, the estimated total data volume is 8x2 24The selected HASH function is CRC24. That is, HEAD ZONE has 2 24 Entries, that is, there are 2 24 chains, then according to the statistical distribution, each chain has an average of 8, and the selected HASH function is CRC25, which becomes 4x2 while the total number remains unchanged. 25 , i.e. a total of 2 25 There are 4 chains in each chain. In this way, by adjusting the HASH type, the length of each chain can be adjusted.

[0054] It should be noted that when changing from an old HASH table to a new HASH table, the change is the REHASH operation mentioned above. The purpose of the change is to improve the read and write performance. After the change, the single chain becomes shorter, and the read and write performance is improved. Another purpose is to avoid resource waste. For example, if the estimated data is too large in the early stage, the selected CRC is relatively large, and the number of HEADZONGs is relatively large, some of them will not be used, resulting in resource waste.

[0055] For further information, see Figure 4 The present invention also provides a hardware implementation method for optimizing KV storage performance based on configurable logic, which uses the hardware for optimizing KV storage performance based on configurable logic, including the following implementation steps:

[0056] S1: Preliminarily estimate a HASH type based on the total amount of KV in the namespace and the expected performance;

[0057] It should be noted that in computer programming, "namespace" is a mechanism for avoiding naming conflicts and organizing code. Namespace is a way to organize related classes, functions, variables, etc. together to make the code more readable, maintainable and easy to expand.

[0058] By using namespaces, different namespaces can be assigned to different code parts to ensure that there will be no name conflicts between them. This way, even if the same name exists in different libraries or modules, there will be no conflict. Namespaces allow developers to better manage and organize their own code, and also facilitate code reuse.

[0059] The syntax and usage of namespaces may be slightly different in different programming languages. For example, in C++, the keyword "namespace" is used to define a namespace, and the scope operator "::" is used to access members in the namespace. In C#, the keyword "namespace" is also used to define a namespace, and "." is used to access members in the namespace.

[0060] Using namespaces can improve the modularity and maintainability of code, making the development of large projects more efficient and orderly. It is an important organization and management tool in programming.

[0061] It should be noted that during the estimation process, the length of a chain can be calculated based on the size of key_info_buf set by the user, the time required to access a key_info_buf, the time used for HASH operations and the module for comparing KEY sizes, and the desired performance. Then, the number of chains can be calculated based on the total amount of data, thereby completing the HASH type estimation.

[0062] S2: Select HASH type;

[0063] S3: As the actual KV data is filled into kv_hash_table, the actual calculated performance and space are compared with the performance and space estimated in step S1 to evaluate whether the performance or space is optimal, that is, whether the performance can meet the demand and whether there is any space waste.

[0064] It should be noted that kv_hash_table is a common data structure, which is a key-value storage mechanism, usually used to implement hash tables. In programming, a hash table is an array that uses a hash function to index data and is used to store key-value pairs. Each key is converted into a number (hash value) through a hash function, which is used to index the position in the array, thereby quickly accessing the value stored at that position.

[0065] The main features of kv_hash_table include:

[0066] a. Key-value pair storage: Each element is a key-value pair, where the key is used to uniquely identify a value.

[0067] b. Hash function: used to convert a key into a hash value, which is used to determine the storage location in the array.

[0068] c. Array: A hash table is usually implemented as an array, where each slot in the array corresponds to a possible hash value.

[0069] d. Conflict resolution: When two keys produce the same hash value, it is called a conflict. Conflict resolution strategies include chain address method (using a linked list to store key-value pairs with the same hash value) and open address method (finding an empty slot to store the key-value pair).

[0070] e. Dynamic expansion: In order to maintain efficient performance, the hash table may dynamically expand as the number of elements increases, that is, increase the size of the array.

[0071] f. Iteration: kv_hash_table usually provides an iterator to traverse all key-value pairs.

[0072] In different programming languages ​​and libraries, kv_hash_table or similar structures may have different implementations and APIs. For example, in C++, you can use std::unordered_map or std::map, which are key-value pair containers based on hash tables. In Python, the dict type is a hash table used to store key-value pairs.

[0073] The benefit of using kv_hash_table is the ability to quickly find, insert, and delete key-value pairs, especially when processing large amounts of data, it provides relatively high data storage and retrieval efficiency.

[0074] It should be noted that: if we only look at performance, the greater the performance, the better; if we only look at space, the smaller the space, the better. However, performance and space are opposites. The greater the performance, the greater the space consumption. Similarly, the smaller the space, the lower the performance. Therefore, it is best to reduce the space under the premise of meeting your performance requirements.

[0075] To further explain the optimal strategy:

[0076] 1. The performance evaluation standard is the reading and writing speed, i.e. QPS (query per second), which is determined by the time it takes to read and write a Key_info_buf, and the unit is the unit of time.

[0077] 2. The evaluation standard of space is the size of the space occupied by the data, that is, the size of the data in KB / GB / TB, etc.

[0078] 3. In step S1, a performance and space size are estimated. However, when the actual application reaches step S3, the performance and the space size to be occupied are actually calculated. By comparing with the estimation in step S1, it can be determined whether there is a waste of space and performance.

[0079] For example, in step S1, the estimated performance is: 10us to read a key_info_buf, the space is 100M. But in S3, the actual calculated performance is: 5us to read a key_info_buf, and the total amount of data stored is only 80M. For users, 10us and 80M are enough, so 5us is a bit wasteful. In this way, CRC can be adjusted to make a single chain longer, the total number of chains is reduced, and the performance can be reduced to 10us; because the KI_BUF of HEAD_ZONE is a continuous address, the number of chains is reduced, and the space of HEAD_ZONE can be saved. The total data size is known in step S3, so the space required for RAND_ZONE can be calculated, so that it can be reduced from the original 100M to 80M.

[0080] S4: Select a specific CRC type based on the selection rules of performance and space, and repeat step S2 until the performance and space are optimal. If the performance is satisfactory, use a smaller CRC type as much as possible to save resources; if the performance is too low and needs to be improved, choose a larger CRC type, and finally choose a solution with the best performance and resources.

[0081] Specifically, when selecting a specific CRC type according to the selection rule of performance and space, when the performance is too high, a smaller CRC type is selected, and when the performance is too low, a larger CRC type is selected. That is, when the performance and space estimated in step S1 are higher than the performance and space actually calculated in step S3, a smaller CRC type than step S1 is selected, and when the performance and space estimated in step S1 are lower than the performance and space actually calculated in step S3, a larger CRC type than step S1 is selected.

[0082] It should be noted that the criteria for selecting a CRC type are usually based on the system's performance requirements and data length. Here, "too high performance" and "too low performance" refer to the system's requirements for data transmission speed and error detection capabilities. "Small" and "large" CRC types refer to the number of bits in the CRC.

[0083] Smaller CRC types: usually refers to CRC types with fewer bits, such as CRC-12 and CRC-24. The advantage of these CRC types is that they are fast in calculation and suitable for high-speed data transmission, but they can detect fewer types of errors and may not be able to detect some complex error patterns.

[0084] Larger CRC types: such as CRC-32, have more bits and can detect more error types, including some complex error patterns. Such CRC types can provide stronger error detection capabilities, but the calculation speed will be relatively slow, which may affect the data transmission throughput.

[0085] The specific judgment criteria are as follows:

[0086] a. Overperformance: When the data transmission speed of the system is much higher than the data transmission speed of the user's target, you can choose a CRC type with fewer bits, because fewer bits means fewer entries in the Headzone, that is, the number of chains in the hashtable is reduced. At this time, the size of the data space can be adjusted.

[0087] b. Performance is too low: When the system's data transmission speed is much lower than the user's target data transmission speed, you can choose a CRC type with more bits, because more bits means more entries in the Headzone, that is, more chains in the hashtable. In this way, the total amount of data remains unchanged, and the number of key_info_bufs in a single chain decreases, which can improve performance.

[0088] c. Smaller CRC type: Usually used in scenarios where the data transmission rate requirement is not very high but the storage space requirement is very high (for example, only relatively fragmented and discontinuous space can be provided). Because of the smaller CRC type, the number of chains is small and the number of single chains is large. That is, there are fewer heazdzones and more randzones. The storage space required for headzone entries is relatively continuous, and the space required for randzone key_info_buf can be any discontinuous address space.

[0089] d. Larger CRC type: Usually used in scenarios where data transmission rate requirements are very high and storage space requirements are not high (i.e., both continuous and discontinuous space can be provided). Because the larger CRC type, i.e., more chains and smaller number of each chain, can meet the requirements of relatively high transmission rate.

[0090] It is suitable for scenarios with extremely high requirements for data integrity, such as large storage systems and mission-critical control systems.

[0091] The present invention is based on a configurable HASH algorithm to achieve reasonable optimization of performance and resources from the perspective of hardware. In practical applications, the strategy can be reasonably adjusted at any time according to actual needs to avoid resource waste or performance degradation caused by estimation errors before application or later development changes.

[0092] The present invention is based on research on a new KV storage form, KV-SSD, which is based on the implementation of HASH, realizes hardware acceleration through a configurable logic, and can adjust the read and write performance in real time according to actual applications.

[0093] In actual operation, for example, in a read operation, the CPU only needs to send a read command and KEY to the hardware module, and the hardware can autonomously run the HASH operation, find the corresponding HEAD ZONE according to the result of the HASH operation, and implement the hardware reading the key_info_buf on the chain and performing a comparison. If a match or mismatch is found, it will be fed back to the CPU, thus avoiding the need for the CPU to perform the HASH operation, and reading the key_info_buf and comparison operations on the chain in turn, thus freeing up the CPU.

[0094] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A hardware implementation method for optimizing KV storage performance based on configurable logic, characterized in that: The hardware implementation process includes the following: Use CRC as the HASH function and support 17 options, namely CRC16~CRC32; The chain address method is used to solve the HASH conflict problem, and the same chain is arranged from small to large according to the size of the key; Adjust the number of entries in the HEAD ZONE by adjusting the HASH function; Among them, Key_info_buf is a unit containing key and pointer to value; Among them, HEAD ZONE is divided into two areas, Head Zone and Rand Zone, in order to distinguish the first Key_info_buf of each chain in HASH TABLE from other Key_info_buf on the chain. That is, Head Zone refers to the area composed of the first Key_info_buf of each chain in HASHTABLE; Rand Zone refers to the area composed of Key_info_buf except the first Key_info_buf of each chain; Among them, the HASH type can be adjusted at any time to adjust the length of each chain, thereby adjusting the read and write performance at any time; The hardware implementation method includes the following implementation steps: S1: Preliminarily estimate a HASH type based on the total amount of KV in the namespace and the expected performance; S2: Select HASH type; S3: As the actual KV data is filled into kv_hash_table, the performance and space actually calculated at this time are compared with the performance and space estimated in step S1; S4: Select a specific CRC type according to the selection rules of performance and space, and repeat step S2 until the performance and space are optimized; Among them, when selecting a specific CRC type based on the performance and space selection rules, when the performance and space estimated in step S1 are higher than the performance and space actually calculated in step S3, a CRC type smaller than step S1 is selected; when the performance and space estimated in step S1 are lower than the performance and space actually calculated in step S3, a CRC type larger than step S1 is selected.

Images