Efficient kv encoding storage method based on redis protocol

Abstract

The application provides a high-efficiency KV encoding storage method based on a Redis protocol, and has the characteristics that all Key meta information of a data object is stored in an object key; and descriptive information of the data object, keys of each field in the data object, and values of each field in the data object are all stored in an object value. Preferably, in the high-efficiency KV encoding storage method based on the Redis protocol, the object value is encoded in blocks in a fixed length, a block header of each block stores descriptive information of the data object value, and the rest of each block stores the keys and values of the fields. Preferably, in the high-efficiency KV encoding storage method based on the Redis protocol, the fixed length of the block is an integer multiple of an operating system block. The encoding and storage method of the application reduces the number of I / Os, the number of Key encodings, saves computing resources, and improves the read-write efficiency.

Description

Technical Field

[0001] This protocol pertains to the field of NoSQL storage, specifically to an efficient key-value encoding storage method based on the Redis protocol. Background Technology

[0002] Redis is a very popular NoSQL protocol, suitable for solving problems that are not suitable for traditional relational databases. As an in-memory database, all data in Redis resides in memory, making it particularly suitable for handling small amounts of frequently accessed data. However, when massive amounts of data exceed the memory capacity, it needs to be written to disk. In this case, a Redis+KV ​​storage solution, such as RocksDB, is required.

[0003] In the integration of Redis with key-value stores, encoding and decoding are crucial steps. The standard Redis protocol, when managing complex data objects such as hashes, lists, sets, and sorted sets, encodes the Meta and Field elements of the data objects separately. The logical structure of this encoding is as follows: Figure 1 As shown (the encoding of Fields may differ slightly depending on the business scenario). For example, for a List type data object, a key-value pair (meta_key, meta_value) is needed to store the metadata of the entire key (such as the number of members in the List, expiration time, etc.), and another key-value pair (field_key, field_value) is needed to store the name and value of the member. For a Sorted set data object, due to the existence of two attributes, score and rank, it may even be necessary to have one key-value pair to store the metadata of the entire key, one key-value pair to store the score information, and another key-value pair to store the rank information corresponding to each member.

[0004] This method has the following drawbacks:

[0005] (1) When the number of object members is 1, compared with the Set / Get command, this data type has 1 more I / O. First, it must obtain the MetaKey to determine whether the member has expired. If not, it continues to obtain the Value corresponding to the Key.

[0006] (2) When the number of object members is n, it takes n+1 I / O operations to search for members within a range.

[0007] (3) Updating each member requires two I / O operations: one to update the member's value and the other to update the meta information of the number of its members.

[0008] (4) The number of keys has increased, which has increased the compaction burden of the disk KV.

[0009] Furthermore, since the file system performs disk I / O operations in the form of operating system blocks and the write operations to disk are random, when using the above encoding scheme, multiple fields of a data object will be scattered in different data blocks after being written to disk, which greatly reduces the I / O efficiency when reading and rereading the same data object. Summary of the Invention

[0010] In view of the shortcomings of the prior art described above, the present invention provides an efficient key-value encoding storage method based on the Redis protocol, characterized by including: the step of storing all key meta-information of a data object into an object key; and the step of storing the descriptive information of the data object, the keys of each field in the data object, and the values ​​of each field in the data object into an object value.

[0011] Preferably, in the above-described efficient KV encoding storage method based on the Redis protocol, the object values ​​are encoded in blocks aligned to a fixed length, the header of each block stores descriptive information of the data object value, and the remainder of each block stores the key and value of the field.

[0012] Preferably, in the above-described efficient KV encoding storage method based on the Redis protocol, the fixed length of the block is an integer multiple of the operating system block.

[0013] Preferably, in the above-described efficient KV encoding storage method based on the Redis protocol, the block header stores the total number of field members of the data object, the lifecycle of the data object, and the number of fields stored in the block.

[0014] Preferably, in the above-described efficient KV encoding storage method based on the Redis protocol, the length of the field key is recorded in 2 bytes at the end of each block; and the length of the field value is recorded in 2 bytes.

[0015] Preferably, in the above-described efficient KV encoding storage method based on the Redis protocol, the block header of the block also stores the index information of the next block.

[0016] Preferably, in the above-described efficient KV encoding storage method based on the Redis protocol, the block header of the block further includes a reserved field.

[0017] Preferably, in the above-mentioned efficient KV encoding storage method based on the Redis protocol, the length of the field key is recorded using 2 bytes immediately after the reserved field, and then the string of the field key is directly saved. The length of the field value is recorded using 2 bytes after the field key string, and then the field value is directly saved. Attached Figure Description

[0018] Figure 1 This is a logical structure diagram of the existing KV encoding based on Redis;

[0019] Figure 2 This is a logical structure diagram of ObjValue according to one embodiment of the present invention;

[0020] Figure 3 This is the logical structure diagram of ObjValue according to the second embodiment of the present invention. Detailed Implementation

[0021] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0022] The core of the encoding scheme of this invention is a single KV encoding. First, the first implementation method is introduced, which includes the step of setting the data object key ObjKey, which corresponds to the previous MetaKey. It mainly records the meta-information of all keys of the data object, including the slot of the distributed cluster, the object name (i.e., RAW_KEY), and other information.

[0023] The encoding scheme in Example 1 also includes the step of merging meta_value, field_key, and field_value into an object value ObjValue. ObjValue is encoded in blocks aligned to a fixed length. This fixed length is chosen based on the data block size during disk I / O operations by the file system, typically an integer multiple of the operating system block size. For commonly used disk file systems, this fixed length is set to 4KB in this embodiment. According to the Redis protocol's recommended standard, the value cannot exceed 512MB, thus there are 131,072 coded ObjValue blocks.

[0024] Each block's header records descriptive information about the data object, such as its size and lifecycle (descriptive information about the value). In this example, the block header is a fixed length of 32 bytes and specifically records the following information:

[0025] Fields Length (Byte) illustrate obj_size 8 The current size of this structure, i.e., the total number of field members. ttl 8 The lifecycle of this object field_size 8 The number of fields stored in this block rsd 8 Reserved fields

[0026] The remaining space in the block is used to store the field values. Since each field_key and field_value is variable-length, when inserted into the block, two bytes are used at the end to record the length of each field_key and field_value (this scheme is designed for small keys, so two bytes can record a maximum length of 65536). This allows the specified field to be retrieved through mathematical operations. Operations within the block use start and end pointers; when the start and end pointers meet, the space is full; when the end pointers meet, the block is full. The logical structure of the ObjValue block in Example 1 is as follows: Figure 2 As shown.

[0027] This invention also provides a second embodiment: In Embodiment Two, ObjValue is also encoded in blocks aligned to a fixed length. The difference from Embodiment One is that the block header of each block not only records the size and lifecycle of the data object, but also records the index information of the next block. After the reserved field (rsd) of the block, two bytes are used to record the length of the field key (field_key), followed directly by the string of field_key. After the field_key string, two bytes are used to record the length of the field value (field_value), followed directly by the value of field_value. The logical structure of the ObjValue block in Embodiment Two is as follows: Figure 3 As shown. Compared with Embodiment 1, the advantage of Embodiment 2 is that it can greatly improve storage efficiency in scenarios where the number and type of fields of data objects change—it only requires reallocating blocks for storage based on the changed fields, and then updating the information of the next block in the original block.

[0028] The above encoding method firstly reduces the number of I / O operations. For operations on a single data object, a single I / O operation is sufficient to obtain the Meta and Field member information, completing read, write, and local lookup operations. Secondly, it reduces the number of encoded keys, effectively compressing the encoding of fields. Furthermore, in a disk-based KV storage system, reducing the number of keys reduces compaction pressure, saving computational resources. In addition, file systems perform disk I / O operations in data blocks. Using the encoding method of this invention, ObjValue information is more easily stored contiguously on disk, further improving read and write efficiency. Therefore, this invention solves the problems of excessive I / O operations and discontinuous disk storage in existing technologies, and significantly saves computational resources, thus possessing high industrial application value.

Claims

1. An efficient KV encoding storage method based on Redis protocol, characterized in that, The method comprises: a step of storing all key meta-information of a data object into an object key; a step of storing descriptive information of the data object, keys of fields in the data object, and values of the fields in the data object into an object value; wherein the object value is encoded in blocks aligned in fixed length, a block header of each block stores the descriptive information of the data object, and the rest of each block stores the keys and values of the fields.

2. The Redis protocol-based efficient KV coding storage method according to claim 1, characterized in that, The fixed length of the block is an integer multiple of the operating system block.

3. The Redis protocol-based efficient KV coding storage method according to claim 1, characterized in that, The block header of the block further stores the total number of field members of the data object, the life cycle of the data object, and the number of fields saved in the block.

4. The Redis protocol-based efficient KV coding storage method according to claim 1, characterized in that, The length of the field key is recorded in 2 bytes at the tail of each block; the length of the field value is recorded in 2 bytes.

5. The Redis protocol-based efficient KV coding storage method according to claim 3, characterized in that, The block header of the block further stores index information of the next block.

6. The Redis protocol-based efficient KV coding storage method according to claim 3, characterized in that, The block header of the block further comprises a reserved field.

7. The Redis protocol-based efficient KV coding storage method according to claim 6, characterized in that, The reserved field is followed by 2 bytes recording the length of the field key, and then directly saving the string of the field key; the string of the field key is followed by 2 bytes recording the length of the field value, and then directly saving the field value.

Images