A key-value pair processing method and data processing device

Abstract

A key-value pair processing method includes: receiving a first key-value pair; when the first key-value pair includes a large key, sharding the large key; generating a global identifier for each shard; generating basic key-value pairs according to each shard and the global identifier of the shard in the order of sharding, each basic key-value pair includes a basic key and a value corresponding to the basic key, and the global identifier included in the value of each basic key-value pair is the same as the global identifier included in the key of the next basic key-value pair; writing all generated basic key-value pairs into a key-value storage device. In this way, the large key can be split and stored, and all large key shards can be found according to the global identifier to restore the large key, thereby realizing the persistence of the large key. The present application also provides a data processing device capable of implementing the above-mentioned key-value pair processing method.

Description

Technical Field

[0001] The present application relates to the field of data storage, and in particular to a key-value pair processing method and a data processing device. Background Art

[0002] A key-value (KV) storage database is a non-relational data storage database. In a key-value storage database, data is stored in the form of key-value pairs, with the key being the index of the value, and the value being the stored data. In a key-value storage database, the size of the key cannot exceed the maximum key length set by the key-value storage database. Such a key is also called a basic key. For some scenarios where applications need to use large keys (the length of the large key exceeds the maximum key length), the large keys cannot be stored directly in the key-value storage database. However, large keys stored in memory are easily lost, so a certain method needs to be provided to persist the large keys to ensure data security and reliability.

[0003] To address the above issues, there is currently a key-value storage method that is roughly as follows: hash the large key to a hash value that does not exceed the maximum key length, and save the hash value on the disk. When the large key stored in the memory is lost due to power outage or other reasons, the large key can be obtained based on the hash value.

[0004] In actual key indexes, multiple keys are usually arranged in order. However, after the hash value is obtained by hash mapping the large key among multiple keys, the hash value will replace the position of the large key. Since the hash value is completely different from the original large key, the hash value will disrupt the order relationship between the previous and next keys, resulting in multiple keys being disordered, so it is impossible to traverse multiple keys in order. Summary of the invention

[0005] In view of this, the present application provides a key-value pair processing method and a data processing device, which can persist large keys and traverse all keys in order.

[0006] The first aspect provides a key-value pair processing method, in which after receiving a first key-value pair, when the first key-value pair includes a large key, the large key is sharded, and a global identifier is generated for each shard; a basic key-value pair is generated according to each shard and the global identifier of the shard in the sharding order; and all generated basic key-value pairs are written into a key-value storage device. Each basic key-value pair includes a basic key and a value corresponding to the basic key, and the global identifier included in the value of each basic key-value pair is the same as the global identifier included in the key of the next basic key-value pair.

[0007] When the data processing device loses power, the program crashes, or other reasons cause the large key stored in the memory to be lost, the above method can ensure that the large key is stored in the key-value storage device, thus solving the problem that the large key is easily lost when stored in the memory. Also, a global identifier is set in the value of the previous basic key-value pair and the key of the next basic key-value pair, so that the next basic key-value pair can be found according to the global identifier, until all basic key-value pairs are found, and each fragment of the large key can be obtained, and all fragments of the large key can be used to form the large key. In the key-value storage system, each basic key-value pair serves as a node in the forest, and all basic key-value pairs form a forest. By traversing the forest, all keys received by the data processing device can be traversed in order.

[0008] In a possible implementation, generating a basic key-value pair according to each shard and the global identifier of the shard in the order of shards includes: when the shard is the first shard, generating a first shard identifier; generating a key of the first basic key-value pair according to the first shard identifier and the first shard; generating a value of the first basic key-value pair according to the index identifier and the global identifier of the first shard; when the shard is neither the first shard nor the last shard, generating the shard identifier; generating a key of the basic key-value pair according to the shard, the shard identifier and the global identifier of the previous shard; generating a value of the basic key-value pair according to the index identifier and the global identifier of the shard; when the shard is the last shard, generating the last shard identifier; generating a key of the last basic key-value pair according to the last shard identifier, the global identifier of the second-to-last shard and the last shard; generating a value of the last basic key-value pair according to the data identifier and the value of the first key-value pair. In this way, the basic key-value pairs corresponding to all shards can be generated according to the order of shards. The keys of different basic key-value pairs can be distinguished by the shard identifier and the global identifier.

[0009] In another possible implementation, after all generated basic key-value pairs are written to the key-value storage device, a large key of the first key-value pair is received; the large key is sharded; basic key-value pairs are obtained from each shard in the sharding order; and the value of the first key-value pair is parsed from the value of the last basic key-value pair. Since the next basic key-value pair is a child node of the previous basic key-value pair in the key-value storage system, each shard can be determined according to the node order, all shards are combined into a large key, and the value corresponding to the large key can be read from the last basic key-value pair.

[0010] In another possible implementation, obtaining a basic key-value pair according to each shard in the shard order includes: when the shard is the first shard, generating a first basic key according to the first shard identifier and the first shard; reading a value corresponding to the first basic key from a key-value storage device; when the shard is not the first shard, parsing a global identifier of the previous shard from the value of a previous basic key-value pair; generating a basic key corresponding to the shard according to the shard, the shard identifier, and the global identifier of the previous shard, and reading a value corresponding to the basic key from the key-value storage device. After all basic key-value pairs corresponding to the first key-value pair are stored in the key-value storage device, each basic key-value pair can be obtained from the key-value storage device in sequence.

[0011] In another possible implementation, after receiving the second key-value pair, when the second key-value pair includes a large key, the large key of the second key-value pair is sharded; when the first i shards of the second key-value pair are the same as the first i shards of the first key-value pair, the first i basic key-value pairs corresponding to the first key-value pair are used as the first i basic key-value pairs of the second key-value pair; starting from the i+1th shard, a global identifier is generated for each shard; basic key-value pairs are generated in turn according to each shard and the global identifier of the shard, and all generated basic key-value pairs are written to the key-value storage device. i is a positive integer. After all generated basic key-value pairs are written to the key-value storage device, the first i basic key-value pairs generated according to the large key of the first key-value pair can obtain the first i shards of the large key of the second key-value pair, and the (i+1)th basic key-value pair to the last basic key-value pair generated according to the large key of the second key-value pair can obtain the (i+1)th shard to the last shard of the large key of the second key-value pair. For the second key-value pair including the large key, only the basic key-value pairs of different parts need to be stored, thus saving storage space.

[0012] The second aspect provides a data processing device, which includes an input module, a processing module and a communication interface; the input module is used to receive a first key-value pair; the processing module is used to shard the large key when the first key-value pair includes a large key; generate a global identifier for each shard; generate basic key-value pairs according to each shard and the global identifier of the shard in the order of shards, each basic key-value pair includes a basic key and a value corresponding to the basic key, and the global identifier included in the value of each basic key-value pair is the same as the global identifier included in the key of the next basic key-value pair; write all the generated basic key-value pairs into a key-value storage device through the communication interface.

[0013] In a possible implementation, the processing module is specifically used to generate a first shard identifier when the shard is the first shard; generate a key of a first basic key-value pair based on the first shard identifier and the first shard; generate a value of the first basic key-value pair based on the index identifier and the global identifier of the first shard; generate a shard identifier when the shard is neither the first shard nor the last shard; generate a key of a basic key-value pair based on the shard identifier, the global identifier of the previous shard, and the shard; generate a value of the basic key-value pair based on the index identifier and the global identifier of the shard; generate a last shard identifier when the shard is the last shard; generate a key of a last basic key-value pair based on the last shard, the last shard identifier, and the global identifier of the second-to-last shard; and generate a value of the last basic key-value pair based on the data identifier and the value of the first key-value pair.

[0014] In another possible implementation, after the processing module writes all generated basic key-value pairs into a key-value storage device, the input module is further used to receive a large key of a first key-value pair; the processing module is further used to shard the large key; obtain basic key-value pairs according to each shard in shard order; and parse the value of the first key-value pair from the value of the last basic key pair.

[0015] In another possible implementation, the processing module is specifically used to generate a first basic key according to the first shard identifier and the first shard when the shard is the first shard; read the value corresponding to the first basic key from the key-value storage device; when the shard is not the first shard, parse the global identifier of the previous shard from the value of the previous basic key-value pair; generate the basic key corresponding to the shard according to the shard, the shard identifier and the global identifier of the previous shard, and read the value corresponding to the basic key from the key-value storage device; and parse the value of the first key-value pair from the value of the last basic key-value pair.

[0016] In another possible implementation, the input module is also used to receive a second key-value pair; the processing module is also used to shard the large key of the second key-value pair when the second key-value pair includes a large key; when the first i shards of the large key of the second key-value pair are the same as the first i shards of the large key of the first key-value pair, starting from the i+1th shard of the large key of the second key-value pair, generate a global identifier for each shard; generate a basic key-value pair in turn according to each shard and the global identifier of the shard, where i is a positive integer; and write all the generated basic key-value pairs into a key-value storage device.

[0017] The third aspect provides a data processing device, which includes a processor and a memory; the memory is used to store program instructions; by calling the program instructions stored in the memory, the processor is used to execute the key value processing method of the first aspect.

[0018] The fourth aspect provides a data processing system, comprising a data processing device and a key-value storage device; the data processing device is used to execute the steps performed by the data processing device in the key-value pair processing method of the first aspect; the key-value storage device is used to execute the steps performed by the key-value storage device in the key-value pair processing method of the first aspect.

[0019] The fifth aspect provides a computer storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the key-value pair processing method of the first aspect.

[0020] A sixth aspect provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the methods of the above aspects.

[0021] The seventh aspect provides a chip system, which includes a processor for supporting a data processing device to implement the functions involved in the above aspects, for example, sending or processing instructions and / or data involved in the above key-value pair processing method. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the key-value pair processing method. The chip system can be composed of a chip, or it can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A A schematic diagram of a data processing system in this application;

[0023] Figure 1B is another schematic diagram of the data processing system in this application;

[0024] Figure 2 A schematic diagram of the structure of the data processing device in this application;

[0025] Figure 3 is a flow chart of the key value processing method in this application;

[0026] Figure 4 is another flow chart of the key value processing method in this application;

[0027] Figure 5A A schematic diagram of the basic key-value pairs in this application;

[0028] Figure 5B is another schematic diagram of the basic key-value pairs in this application;

[0029] Figure 5C is another schematic diagram of the basic key-value pairs in this application;

[0030] Figure 6 A schematic diagram of reading the value corresponding to the large key in this application;

[0031] Fig. 7A is another schematic diagram of the basic key-value pairs in this application;

[0032] Figure 7B is another schematic diagram of the basic key-value pairs in this application;

[0033] Figure 8 Another schematic diagram for reading the value corresponding to the large key in this application;

[0034] Fig. 9 A storage diagram of two large keys with the same shard in this application;

[0035] Fig.10 This is another structural schematic diagram of the data processing device in this application. DETAILED DESCRIPTION

[0036] First, some of the terms that appear in the embodiments of this application are introduced:

[0037] A key-value pair consists of a key and a value. In the key-value storage device 20, data is stored in the form of a key-value pair, where the key is the index of the value and the value is the stored data. According to the key length, the key can be divided into a big key and a base key. The key length refers to the size of the key.

[0038] A large key refers to a key whose key length exceeds a specified length. In one implementation, the specified length is the maximum key length allowed by the key-value storage device, such as 1 kilobyte (KB), 2KB, or 4KB. It should be understood that the specific value of the specified length can be set according to actual conditions, and can be greater than 1KB or less than 1KB. The length unit of the large key can be a storage unit of KB, megabyte (MB), gigabyte (GB), or a larger size. Since the large key does not meet the storage requirements of the key-value storage device 20, the large key cannot be directly stored in the key-value storage device 20.

[0039] A basic bond is a bond whose length does not exceed the specified length.

[0040] A basic key-value pair consists of a basic key and a value, and the basic key is used to find the value corresponding to the basic key in a key-value storage device.

[0041] The fragment identifier is used to identify the fragment number, for example, the fragment identifier can be a number, such as a fragment serial number, or a combination of one or more characters and numbers or other types of identifiers.

[0042] The global identifier is used to index shards. Each shard has a unique global identifier in the key-value database. The global identifier can be a universally unique identifier (UUID). Specifically, the key of the first base key-value may not include the global identifier, and the value of the last base key-value may not include the global identifier. If there are n base key-value pairs in total, then the base key-value pairs between the first base key-value pair and the last base key-value pair include the 2nd to the (n - 1)th base key-value pairs. The key and value of the ith base key-value pair each include a global identifier. The global identifier included in the key of the ith base key-value pair is the same as the global identifier included in the value of the (i - 1)th base key-value pair, and the global identifier included in the value of the ith base key-value pair is the same as the global identifier included in the key of the (i + 1)th base key-value pair, where i is an integer and 1 < i < n. In this way, the value of the base key-value pair can be found according to the key of each base key-value pair, and the global identifier included in the key of the next base key-value pair can be obtained from the value of each base key-value pair, so as to obtain the key of the next base key-value pair. By executing sequentially, all base key-value pairs can be found.

[0043] The application scenario of the present application will be introduced below. The key-value pair processing method of the present application can be applied to a data processing system. Refer to Figure 1A and Figure 1B , the data processing system may include a data processing device 10 and a key-value storage device 20.

[0044] The data processing device 10 may be a host, a server, or an electronic device with computing capabilities. The key-value storage device 20 may be an independent storage server or a distributed storage server. Among them, the key-value storage device 20 may include one or more disks or one or more solid state disks (SSDs). The data processing device 10 and the key-value storage device 20 may be directly connected through a communication cable or may be connected through a wired network or a wireless network.

[0045] An application program is deployed on the data processing device 10, and the application program can communicate with the database program. The user can input a key or a key-value pair in the application program. After the data processing device 10 obtains the above key or key-value pair, the data processing device 10 can send read operations and write operations to the key-value storage device 20. Alternatively, when the user inputs a key or a key-value pair in the database program, after the data processing device 10 obtains the above key or key-value pair, the data processing device 10 can also send read operations and write operations to the key-value storage device 20. The user input method can be mouse and keyboard input, voice input, touch input, etc.

[0046] Refer to Figure 1A, to perform a write operation, the data processing apparatus 10 provides a key and a value to the key-value storage device 20 to write the key-value pair into the key-value storage device 20 , with the key serving as an index to the value.

[0047] See also Figure 1B To perform a read operation, the data processing device 10 provides a key to the key-value storage device 20, and the key-value storage device 20 finds the value according to the key and provides the value to the data processing device 10. Therefore, in a key-value storage system, the key is an index used to access the value, and the value is the data to be accessed. Generally, the length of the key and the value can be fixed or variable.

[0048] The following introduces a method in which a data processing device 10 can implement the key-value pair of the present application. Figure 2 is a schematic diagram of a data processing device 10. Figure 2 The data processing device 10 includes an input module 201, a processor 202, a communication interface 203, a memory 204 and a display 205. The input module 201, the processor 202, the communication interface 203, the memory 204 and the display 205 are connected via a bus 206.

[0049] The input module 201 may be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the data processing device 10. Specifically, the input module 201 may include, but is not limited to, a physical keyboard, a mouse, a touch screen, and function keys (such as a volume control key, a switch key, etc.).

[0050] It should be understood that the processor 202 mentioned in the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0051] It should also be understood that the memory 204 mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0052] It should be noted that when the processor 202 is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory 204 can be integrated into the processor 202. It should be noted that the memory 204 described herein is intended to include but is not limited to these and any other suitable types of memory. By calling the program stored in the memory 204, the processor 202 can execute the key-value pair processing method described in the embodiment of the present application.

[0053] The communication interface 203 is used to communicate with the key-value storage device 20. The processor 202 sends instructions, such as write instructions or read instructions, to the key-value storage device 20 through the communication interface 203. Alternatively, the processor 202 obtains data from the key-value storage device 20 through the communication interface 203.

[0054] The display 205 may be used to display information input by the user or information provided to the user and various menus of the data processing device. The display 205 may include a display panel, which may be configured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0055] It should be understood that although Figure 2 Not shown, the data processing device may also include a power supply, an audio module, a communication module, etc.

[0056] To solve the problem of large key persistence, this application provides a new key-value pair processing method, which stores the shards of large keys through multiple basic key-value pairs, thereby storing large keys persistently. Figure 3 , an embodiment of the key-value pair processing method of the present application includes:

[0057] Step 301: Receive a first key-value pair.

[0058] The first key-value pair may be any key-value pair input by the user in the application or database program, or any key-value pair obtained by the data processing device from other devices (such as terminals or servers). After the user inputs the key-value pair in the application or database program, the data processing device may receive the key-value pair, and receiving the key-value pair may be considered as the data processing device detecting the key-value pair input by the user. It is understandable that the data processing device may also obtain the key-value pair from other terminals or servers.

[0059] Step 302: When the first key-value pair includes a large key, the large key is sharded.

[0060] After receiving the first key-value pair, the key of the first key-value pair is compared with the preset length. If the key of the first key-value pair is greater than the preset length, it is determined that the first key-value pair includes a large key; if the key of the first key-value pair is less than or equal to the preset length, it is determined that the first key-value pair does not include a large key. The preset length can be the difference between the maximum key length allowed by the key-value storage device and the length of the shard identifier. The length of each shard identifier can be set according to actual needs, for example, the length of each shard identifier is the same.

[0061] The size of each shard can be set according to preset rules.

[0062] In one implementation, the total number of shards is 2.

[0063] The first segment length L1 satisfies the following formula: L1 = L base -L s .L base is the maximum length of the base key, Ls Is the fragment identifier length.

[0064] The second segment length L2 satisfies the following formula: L2 = L big -L1.L big is the length of the major key.

[0065] In another implementation, the total number of shards is n, and n is greater than 2.

[0066] The length L1 of the first segment satisfies the following formula: L1 = L base -L s .

[0067] The length of the i-th segment is L i Satisfies the following formula: L i =L base -L s -L u .L u is the length of the global identifier. The lengths of global identifiers of different shards can be the same. i is a positive integer greater than 1 and less than n.

[0068] The length of the nth fragment is L n Satisfies the following formula:

[0069] It should be understood that the above method is to generate the minimum number of slices. It should be understood that the slice length is not limited to the above example, and its length can be set shorter according to actual conditions. For example, the length of the first slice can be set according to the length formula of the i-th slice.

[0070] Step 303: Generate a global identifier for each shard.

[0071] Step 304: Generate a basic key-value pair according to each shard and the global identifier of the shard in the shard order.

[0072] Each basic key-value pair includes a basic key and a value corresponding to the basic key, and the global identifier included in the value of each basic key-value pair is the same as the global identifier included in the key of the next basic key-value pair. It should be noted that the value of the last basic key-value pair may not include the global identifier of the last shard, and the global identifier of the last shard may not be generated in step 303. The last shard is the last shard.

[0073] Step 305: Write all generated basic key-value pairs into a key-value storage device.

[0074] In some key-value databases, write operations can be implemented through the set interface.

[0075] In this embodiment, after the large key is sharded, a basic key-value pair is generated according to each shard, and then all the basic key-value pairs are stored in the key-value storage device. Even if the large key stored in the memory is lost due to power failure of the data processing device, program crash or other reasons, the large key is still stored in the key-value storage device, thus solving the problem that the large key is easily lost when stored in the memory.

[0076] Secondly, a global identifier is set in the value of the previous basic key-value pair and the key of the next basic key-value pair, so that the next basic key-value pair can be found according to the global identifier until all basic key-value pairs are found, and the large key can be restored according to all basic key-value pairs. It can be understood that the value corresponding to the large key can also be read according to all basic key-value pairs, thereby restoring the key-value pair input by the user.

[0077] Again, according to the above method, the large key can be split and stored, and the data processing device can receive keys of any size, so that the restriction on the key size can be removed in the application program or database program.

[0078] The process of creating a basic key-value pair is described in detail below. In an optional embodiment, step 304 includes:

[0079] When the shard is the first shard, generate a first shard identifier; generate a key of a first basic key-value pair according to the first shard identifier and the first shard; generate a value of the first basic key-value pair according to the index identifier and the global identifier of the first shard;

[0080] When the shard is neither the first shard nor the last shard, generate the shard identifier; generate the key of the basic key-value pair according to the shard identifier, the global identifier of the previous shard, and the shard; generate the value of the basic key-value pair according to the index identifier and the global identifier of the shard;

[0081] When the shard is the last shard, generate the last shard identifier; generate the key of the last basic key-value pair according to the last shard, the last shard identifier and the global identifier of the second-to-last shard; generate the value of the last basic key-value pair according to the data identifier and the value of the first key-value pair.

[0082] In this embodiment, each shard corresponds to a basic key-value pair one-to-one, and the order of the shards is the same as the order of the basic key-value pairs. The shard identifier can be a number, a character, or a combination of numbers and characters. For example, the shard identifiers of the 1st, 2nd, ..., Nth shards are 1, 2, ..., N respectively. Or, the shard identifiers of the 1st, 2nd, ..., Nth shards are shard1, shard2, ..., shardN respectively. The above are schematic examples of shard identifiers, and the specific forms of shard identifiers are not limited to the above examples.

[0083] Specifically, the first shard identifier and the first shard are combined to form the key of a first basic key-value pair; the index identifier and the global identifier of the first shard are combined to form the value of the first basic key-value pair.

[0084] If the second shard is the last shard, the second shard identifier, the global identifier of the first shard, and the second shard are used to form the key of the second basic key-value pair, and the data identifier and the value of the first key-value pair are used to form the value of the second basic key-value pair.

[0085] If the second shard is not the last shard, the second shard identifier, the global identifier of the first shard, and the second shard are combined into the key of the second basic key-value pair, and the index identifier and the global identifier of the second shard are combined into the value of the second basic key-value pair; then other basic key-value pairs are generated in sequence until the second-to-last basic key-value pair; the last shard identifier, the global identifier of the second-to-last shard, and the last shard are combined into the key of the last basic key-value pair; the data identifier and the value of the first key-value pair are combined into the value of the last basic key-value pair. In this way, the basic key-value pairs corresponding to all shards can be generated in the order of shards.

[0086] It should be understood that in the key of each basic key-value pair, the order of the shard identifier, the global identifier and the shard is not limited to the above example, and can also be set in other orders, which is not limited here. In the value of each basic key-value pair, the order of the index identifier and the global identifier is not limited to the above example, and the order of the data identifier and the global identifier is not limited to the above example, and can also be set in other orders, which is not limited here.

[0087] It should be understood that all shards can be generated at once, and then a basic key-value pair is generated for each shard in turn. Alternatively, a shard and a basic key-value pair corresponding to the shard are generated, and then the next shard and the basic key-value pair corresponding to the next shard are generated, until the last shard and all the last basic key-value pairs are generated.

[0088] In another optional embodiment, when the first key-value pair does not include a large key, a basic key-value pair is created according to the first key-value pair, and the basic key-value pair is written into the key-value storage device.

[0089] Specifically, if the key of the first key-value pair is less than or equal to the preset length, it is determined that the first key-value pair does not include the large key. The key of the basic key-value pair is generated according to the shard identifier and the key of the first key-value pair, and the value of the basic key-value pair is generated according to the data identifier and the value of the first key value. In this way, when the key-value pair received by the data processing device does not include the large key, it can also be stored.

[0090] When reading data, after inputting the key of the above basic key-value pair, read the value of the above basic key-value pair, and then parse the value of the first key-value pair therefrom.

[0091] The above describes the process of storing basic key-value pairs. Based on the above embodiments, the following describes the process of reading basic key-value pairs:

[0092] See also Figure 4 , an embodiment of the key-value pair processing method of the present application includes:

[0093] Step 401: Receive a major key of a first key-value pair.

[0094] Step 402: Segment the large key.

[0095] Step 403: Obtain basic key-value pairs from each shard in the shard order.

[0096] The global identifier included in the value of each basic key-value pair is the same as the global identifier included in the key of the next basic key-value pair.

[0097] Step 404: parse the value of the first key-value pair from the value of the last basic key-value pair.

[0098] In some key-value databases, obtaining the value corresponding to a base key can be achieved through the get interface.

[0099] In this embodiment, in the key-value storage system, the next basic key-value pair is a child node of the previous basic key-value pair, so each shard can be obtained according to the node order, and then all the shards are combined into a large key. In the key index of the key-value storage system, each basic key-value pair is a node of the forest, and all basic key-value pairs form a forest. By traversing the forest, all keys received by the data processing device can be read in sequence.

[0100] In an optional embodiment, step 403 includes: when the shard is the first shard, generating a first basic key according to the first shard identifier and the first shard; reading the value corresponding to the first basic key from the key-value storage device; when the shard is not the first shard, parsing the global identifier of the previous shard from the value corresponding to the previous basic key; generating a basic key corresponding to the shard according to the shard identifier, the global identifier of the previous shard, and the shard, and reading the value corresponding to the basic key from the key-value storage device. The previous basic key refers to the basic key corresponding to the previous shard. Taking the second shard as an example, the previous shard of the second shard refers to the first shard, and the basic key corresponding to the previous shard refers to the first basic key. In this way, the value of each key-value pair can be read from the key-value storage device.

[0101] The process of storing other key-value pairs is introduced below. In another optional embodiment, the key-value pair processing method further includes:

[0102] Receive a second key-value pair; when the second key-value pair includes a large key, shard the large key of the second key-value pair; when the first i shards of the large key of the second key-value pair are the same as the first i shards of the large key of the first key-value pair, generate a global identifier for each shard starting from the i+1th shard of the large key of the second key-value pair; generate a basic key-value pair according to each shard and the global identifier of the shard in turn; write all the generated basic key-value pairs into a key-value storage device.

[0103] In this embodiment, the second key-value pair refers to any key-value pair received after the first key-value pair is received.

[0104] The total number of shards included in the major key of the first key-value pair is n, and the total number of shards included in the major key of the second key-value pair is m. When the first i shards of the major key of the second key-value pair are the same as the first i shards of the major key of the first key-value pair, the basic key-value pairs corresponding to the first i shards of the second major key are also the same as the basic key-value pairs corresponding to the first i shards of the first major key, wherein the global identifiers of the first i shards of the second major key are the same as the global identifiers of the first i shards of the first major key. For the (i+1)th shard to the mth shard of the second major key, the global identifier of each shard is different from the global identifier of the shard of the first major key.

[0105] Starting from the (i+1)th shard of the second key-value pair, a global identifier is generated for each shard, and a basic key-value pair is generated according to each shard of the second key and the global identifier of the shard. The method for generating a basic key-value pair for each shard is the same as Figure 3 Step 304 in the illustrated embodiment or the optional embodiment is similar. i is a positive integer. For example, for the (i+1)th shard of the second largest key, the key of the (i+1)th basic key-value pair is generated according to the shard identifier of the (i+1)th shard, the global identifier of the i-th shard of the first largest key, and the (i+1)th shard, and the value of the (i+1)th basic key-value pair is generated according to the index identifier and the global identifier of the (i+1)th shard.

[0106] It should be understood that the first i shards of the large key of the second key-value pair are stored in the first i basic key-value pairs of the first key-value pair, and the (i+1)th shard to the last shard of the large key of the second key-value pair are stored in the (i+1)th basic key-value pair to the last basic key-value pair of the second key-value pair.

[0107] The following is an introduction to the process of reading the second key-value pair. When the user enters the main key of the second key-value pair, the data processing device segments the main key into slices, obtains basic key-value pairs from each slice in the order of the slices, and parses the value of the second key-value pair from the value of the last basic key-value pair.

[0108] Among them, the step of obtaining the basic key-value pair according to the shards in the order of shards is similar to step 403, and the details can be found in the corresponding description in the previous text. For example, the i-th shard of the second largest key is the same as the i-th shard of the first largest key, and the global identifier of the i-th shard of the second largest key is the global identifier of the i-th shard of the first largest key. For the (i+1)th shard of the second largest key, the (i+1)th basic key of the second largest key is generated according to the shard identifier of the (i+1)th shard, the global identifier of the i-th shard of the first largest key, and the (i+1)th shard, and the corresponding value is read according to the above (i+1)th basic key. In this way, the complete second key-value pair can be restored based on the basic key-value pairs of the same part and the basic key-value pairs of different parts.

[0109] It should be understood that according to the above key-value pair processing method, in the key-value storage system, the latter basic key-value pair is a child node of the former basic key-value pair, so each major key can be obtained according to the node order.

[0110] Taking the key-value pair input by the user as an example, one or more basic key-value pairs can be generated according to the input key-value pair, and each basic key-value pair can serve as a node.

[0111] The input key-value pair may or may not include a large key. For an input key-value pair that does not include a large key, the basic key-value pair generated from the input key-value pair constitutes a single node, i.e., a tree without child nodes. For an input key-value pair that includes a large key, its large key may include the same first i fragments or different first i fragments, where i is a positive integer.

[0112] For multiple input key-value pairs with the same first i shards, all basic key-value pairs generated based on the multiple input key-value pairs form a tree. Among them, the first basic key-value pair is used as the root node, the basic key-value pair corresponding to the last shard of the large key in each input key-value pair is used as a leaf node, and other basic key-value pairs are used as intermediate nodes. The root node and the intermediate node can have one or more child nodes. Among them, the first i basic key-value pairs are the same, and the i+1th basic key-value pairs of different input key-value pairs are used as different child nodes of the i-th basic key-value pair.

[0113] For an input key-value pair that does not have the same shard as other input key-value pairs, all basic key-value pairs generated from the input key-value pair form a non-branching tree.

[0114] For all two input key-value pairs with different shards, all the basic key-value pairs generated from the above two input key-value pairs form two trees. Similarly, in the key-value storage system, all the basic key values ​​generated from all input key-value pairs form a forest. In the key index, for multiple keys with a sequential relationship, the ordered traversal of multiple keys can be achieved through the shard index value and global identifier of each key.

[0115] The key-value pair processing method of the present application is described in detail below using a specific application scenario. In this application scenario, the maximum key length allowed by the key-value storage device is 1KB as an example.

[0116] When the user needs to store data, the user enters a key-value pair in the application. The key of the key-value pair is recorded as user_key1, and the value of the first key-value pair is recorded as user_value1. Taking the key length of user_key1 as 2.5KB as an example, the data processing device compares the key length of user_key1 with the maximum key length. Since 2.5KB is greater than 1KB, it can be determined that user_key1 belongs to a large key. User_key1 is fragmented. The fragment lengths are 1020B, 956B, and 584B as examples. According to the fragment length, user_key1 is divided into 3 fragments: user_key1_0, user_key1_1, and user_key1_2.

[0117] For user_key1_0, generate a shard identifier 0 and a global index identifier uuid10;

[0118] See also Figure 5A , 0 and user_key1_0 are combined into a key 5011 of the first basic key-value pair 501, and the key 5011 can be represented by 0|user_key1_0. The index identifier is 1, and the data identifier is 0. 1 and uuid10 are combined into a value 5012 of the first basic key-value pair, for example, 1|uuid10.

[0119] See also Figure 5B For user_key1_1, generate a shard identifier 1 and a global index identifier uuid11; combine 1, uuid10 and user_key1_1 to form a key 5021 of the second basic key-value pair 502, such as 1|uuid10|user_key1_1. Combine the index identifier 1 and uuid11 to form a value 5022 of the second basic key-value pair 502, such as 1|uuid11.

[0120] See also Figure 5C , since user_key1_2 is the last shard, generate shard identifier 2, combine 2, uuid11 and user_key1_2 to form key 5031 of the third basic key-value pair 503, such as 2︱uuid11︱user_key1_2. Combine data identifier 0 and value value1 of the first key-value pair to form value 5032 of the third basic key-value pair 503, such as 0︱user_value1.

[0121] All the generated key-value pairs can be stored separately, or all the key-value pairs can be stored together in a key-value storage device after being generated.

[0122] See also Figure 6 When the user needs to read user_value1, the user inputs user_key1, the data processing device determines that user_key1 is greater than the preset length, and then fragments user_key1 into user_key1_0, user_key1_1, and user_key1_2. The fragment identifier 0 and user_key1_0 form the key 5011 of the first basic key 501 (i.e., 0︱user_key1_0), and the value 5022 (i.e., 1︱uuid10) is found according to the key 5011. For the next basic key-value pair, uuid10 is taken out from the value 5022, and then the fragment identifier 1, uuid10, and user_key1_1 form the key 5021 of the second basic key value (i.e., 1︱uuid10︱user_key1_1), and the value 5022 (i.e., 1︱uuid11) is found according to the key 5021. For the last basic value pair, take out uuid11 from the value 5022, and then use the shard identifier 2, uuid11 and user_key1_2 to form the key 5031 of the third basic key value (i.e. 2︱uuid11︱user_key1_2), and find the value 5032 (i.e. 0︱user_value1) according to the key 5031, and parse out user_value1 from it.

[0123] When the user wants to store another key-value pair, the key-value pair is recorded as (user_key2, user_value2). If user_key2 belongs to the big key, then user_key2 is sharded. Assume that 4 shards are obtained, and the 4 shards are user_key2_0, user_key2_1, user_key2_2, and user_key2_3.

[0124] Compare the shards of user_key2 with the shards of user_key1, starting from the first shards of the two large keys. If the first shards of the two large keys are different, then follow the above data storage method to generate basic key-value pairs based on the shards of user_key2 and then store them.

[0125] See also Fig. 7A and Figure 7B, if the first and second shards of the two large keys are the same, then compare the third shards. If the third shards are different, it is determined that user_key2_0, user_key2_1 are the same as user_key1_0, user_key1_1. Thus, starting from user_key2_2, basic key-value pairs 703 and basic key-value pairs 704 are generated according to each shard and the global identifier of the shard. Basic key-value pairs 703 and basic key-value pairs 704 are respectively taken as (2︱uuid11︱user_key2_2, 1︱uuid22) and (3︱uuid22︱user_key2_3, 0︱user_value2) as examples, and the set interface is called to store the above basic key-value pairs in the key-value pair storage device. Among them, the key 7031 of the basic key-value pair 703 is (2︱uuid11︱user_key2_2), the value 7032 of the basic key-value pair 703 is (1︱uuid22), the key 7041 of the basic key-value pair 704 is (3︱uuid22︱user_key2_3), and the value 7042 of the basic key-value pair 704 is (0︱user_value2).

[0126] When the user wants to read user_value2, the user inputs user_key2, and the data processing device segments user_key2 into user_key2_0, user_key2_1, user_key2_2, user_key2_3, i.e. user_key1_0, user_key1_1, user_key2_2, user_key2_3.

[0127] See also Figure 8 , key 5011 (i.e. 0︱user_key1_0) is generated based on user_key1_0, and value 5012 (i.e. 1︱uuid10) is found based on key 5011. Key 5021 (i.e. 1︱uuid10︱user_key1_1) is composed based on shard ID 1, uuid10 and user_key1_1, and value 5022 (i.e. 1︱uuid11) is found based on key 5021. Key 7031 (i.e. 2︱uuid11︱user_key2_2) is composed based on shard ID 2, uuid11 and user_key2_2, and value 7032 (i.e. 1︱uuid22) is found based on key 7031. According to 3, uuid22 and user_key2_3, key 7041 (i.e. 3︱uuid22︱user_key2_3) is formed. According to key 7041, value 7042 (i.e. 0︱user_value2) is found, and user_value2 is parsed from it.

[0128] In the key-value storage system, the basic key-value pairs 501, 502, 503, 703 and 704 are stored in the following form: Fig. 9 As shown. It can be seen that the above method only stores part of the shards of user_key2. When reading data, you can read part of the basic key-value pairs corresponding to user_key1 and other basic key-value pairs corresponding to user_key2 to read user_value2. This can save storage space and achieve ordered traversal.

[0129] In this application, A|B represents a complete data consisting of A and B. It should be understood that A or B can be a number, a symbol or a string, and the symbol | is used for separation for ease of reading and understanding. The symbol | in A|B|C has the same meaning.

[0130] The present application provides a data processing device capable of implementing the method in the above embodiment. Fig.10 , in an optional embodiment, the data processing device 1000 includes an input module 1001, a processing module 1002 and a communication interface 1003;

[0131] Input module 1001, used for receiving a first key-value pair;

[0132] The processing module 1002 is used for sharding the large key when the first key-value pair includes the large key;

[0133] The processing module 1002 is further used to generate a basic key-value pair according to each shard and the global identifier of the shard in the shard order;

[0134] The processing module 1002 is further configured to write all generated basic key-value pairs into a key-value storage device via the communication interface 1003 .

[0135] In an optional embodiment, the processing module 1002 is specifically configured to perform the following steps:

[0136] When the shard is the first shard, generate a first shard identifier; generate a key of a first basic key-value pair according to the first shard identifier and the first shard; generate a value of the first basic key-value pair according to the index identifier and the global identifier of the first shard;

[0137] When the shard is neither the first shard nor the last shard, generate the shard identifier; generate the key of the basic key-value pair according to the shard identifier, the global identifier of the previous shard, and the shard; generate the value of the basic key-value pair according to the index identifier and the global identifier of the shard;

[0138] When the shard is the last shard, generate the identifier of the last shard; generate the key of the last basic key-value pair according to the identifier of the last shard, the global identifier of the second-to-last shard and the last shard; generate the value of the last basic key-value pair according to the data identifier and the value of the first key-value pair.

[0139] In another alternative embodiment,

[0140] The input module 1001 is further used to receive the main key of the first key-value pair;

[0141] The processing module 1002 is also used to perform sharding on the large key;

[0142] The processing module 1002 is further used to obtain a basic key-value pair from each shard in the shard order;

[0143] The processing module 1002 is further configured to parse the value of the first key-value pair from the value of the last basic key pair.

[0144] In another optional embodiment, the processing module 1002 is specifically configured to perform the following steps:

[0145] When the shard is the first shard, generating a first basic key according to the first shard identifier and the first shard; reading a value corresponding to the first basic key from a key-value storage device;

[0146] When the shard is not the first shard, the global identifier of the previous shard is parsed from the value of the previous basic key-value pair; the basic key corresponding to the shard is generated according to the shard, the shard identifier and the global identifier of the previous shard, and the value corresponding to the basic key is read from the key-value storage device.

[0147] In another alternative embodiment,

[0148] The input module 1001 is further used to receive a second key-value pair;

[0149] The processing module 1002 is also used to shard the large key of the second key-value pair when the second key-value pair includes a large key; when the first i shards of the second key-value pair are the same as the first i shards of the first key-value pair, starting from the i+1th shard, generate a global identifier for each shard; generate a basic key-value pair according to each shard and the global identifier of the shard in turn, where i is a positive integer; and write all the generated basic key-value pairs into the key-value storage device through the communication interface 1003.

[0150] It should also be noted that the device embodiments described above are schematic, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme in the present application. In addition, in the drawings of the device embodiments of the present application, some connection relationships between modules or units indicate that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. Other connection relationships between modules or units indicate that they are electrically connected, which can be specifically implemented as one or more circuits.

[0151] The present application provides a computer storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the steps executed by the data processing device in the above embodiments.

[0152] The data processing device in the present application may specifically be a chip in the data processing device, and the chip includes: a processing unit and a communication unit. The processing unit may be a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit. The processing unit may execute computer instructions stored in the storage unit, so that the data processing device executes the data processing method in the above embodiment or the optional embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit located outside the chip in the data processing device, such as a ROM or a static storage device that can store static information and other types, such as a RAM, etc. The processor mentioned in any of the above may be a general-purpose central processing unit, a microprocessor, an ASIC or one or more integrated circuits for executing the program of the above-mentioned key-value pair processing method.

[0153] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, disk or optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment of the present application.

[0154] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0155] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium, etc.

[0156] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A key-value pair processing method, characterized in that: include: receiving a first key-value pair; When the first key-value pair includes a large key, sharding the large key; Generate a global identifier for each shard; Generate basic key-value pairs according to each shard and the global identifier of the shard in the order of shards, each basic key-value pair including a basic key and a value corresponding to the basic key, and the global identifier included in the value of each basic key-value pair is the same as the global identifier included in the key of the next basic key-value pair; Write all generated base key-value pairs to the key-value storage device.

2. The method according to claim 1, characterized in that: Generating a basic key-value pair according to each shard and the global identifier of the shard in the shard order includes: When the shard is the first shard, generate a first shard identifier; generate a key of a first basic key-value pair according to the first shard identifier and the first shard; generate a value of the first basic key-value pair according to the index identifier and the global identifier of the first shard; When the shard is neither the first shard nor the last shard, generate an identifier for the shard; generate a key of a basic key-value pair according to the identifier of the shard, the global identifier of the previous shard, and the shard; generate a value of the basic key-value pair according to the index identifier and the global identifier of the shard; When the shard is the last shard, generate a last shard identifier; generate the key of the last basic key-value pair according to the identifier of the last shard, the global identifier of the second-to-last shard and the last shard; generate the value of the last basic key-value pair according to the data identifier and the value of the first key-value pair.

3. The method according to claim 1, characterized in that After writing all generated basic key-value pairs into the key-value storage device, the method further includes: Receive the major key of the first key-value pair; Slice the large key; Get the basic key-value pairs from each shard in the shard order; The value of the first key-value pair is parsed from the value of the last basic key-value pair.

4. The method according to claim 3, characterized in that The obtaining of the basic key-value pairs according to each shard in the shard order includes: When the shard is the first shard, generating a first basic key according to the first shard identifier and the first shard; reading a value corresponding to the first basic key from a key-value storage device; When the shard is not the first shard, the global identifier of the previous shard is parsed from the value of the previous basic key-value pair; the basic key corresponding to the shard is generated according to the identifier of the shard, the global identifier of the previous shard and the shard, and the value corresponding to the basic key is read from the key-value storage device.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: receiving a second key-value pair; When the second key-value pair includes a large key, sharding the large key of the second key-value pair; When the first i shards of the major key of the second key-value pair are the same as the first i shards of the major key of the first key-value pair, starting from the i+1th shard of the major key of the second key-value pair, generate a global identifier for each shard; Generate a basic key-value pair according to each shard and the global identifier of the shard in turn, where i is a positive integer; Write all generated base key-value pairs to the key-value storage device.

6. A data processing device, characterized in that: The data processing device includes an input module, a processing module and a communication interface; The input module is used to receive a first key-value pair; The processing module is configured to, when the first key-value pair includes a large key, perform sharding on the large key; The processing module is further used to generate a global identifier for each shard; The processing module is further used to generate a basic key-value pair according to each shard and the global identifier of the shard in the order of the shards, each basic key-value pair including a basic key and a value corresponding to the basic key, and the global identifier included in the value of each basic key-value pair is the same as the global identifier included in the key of the next basic key-value pair; The processing module is further configured to write all generated basic key-value pairs into the key-value storage device via the communication interface.

7. The device according to claim 6, characterized in that The processing module is specifically used to generate a first shard identifier when the shard is the first shard; generate a key of a first basic key-value pair according to the first shard identifier and the first shard; and generate a value of the first basic key-value pair according to the index identifier and the global identifier of the first shard; When the shard is neither the first shard nor the last shard, generate an identifier for the shard; generate a key of a basic key-value pair according to the identifier of the shard, the global identifier of the previous shard, and the shard; generate a value of the basic key-value pair according to the index identifier and the global identifier of the shard; When the shard is the last shard, generate a last shard identifier; generate the key of the last basic key-value pair according to the identifier of the last shard, the global identifier of the second-to-last shard and the last shard; generate the value of the last basic key-value pair according to the data identifier and the value of the first key-value pair.

8. The device according to claim 6, characterized in that The input module is further used to receive the main key of the first key-value pair; The processing module is further used to perform sharding processing on the large key; The processing module is further used to obtain basic key-value pairs according to each shard in the shard order; The processing module is further used to parse the value of the first key-value pair from the value of the last basic key-value pair.

9. The device according to claim 8, characterized in that The processing module is specifically configured to, when the shard is the first shard, generate a first basic key according to the first shard identifier and the first shard; and read a value corresponding to the first basic key from the key-value storage device; When the shard is not the first shard, the global identifier of the previous shard is parsed from the value of the previous basic key-value pair; the basic key corresponding to the shard is generated according to the identifier of the shard, the global identifier of the previous shard and the shard, and the value corresponding to the basic key is read from the key-value storage device.

10. The device according to any one of claims 6 to 9, characterized in that The input module is further used to receive a second key-value pair; The processing module is further configured to, when the second key-value pair includes a large key, perform sharding on the large key of the second key-value pair; when the first i shards of the large key of the second key-value pair are the same as the first i shards of the large key of the first key-value pair, generate a global identifier for each shard starting from the i+1th shard of the large key of the second key-value pair; Generate basic key-value pairs in turn according to each shard and the global identifier of the shard, where i is a positive integer; write all generated basic key-value pairs into the key-value storage device.

11. A computer storage medium comprising instructions, characterized in that: When the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 5.

Images