Data processing method and apparatus, and electronic device

Abstract

The embodiment of the application provides a data processing method, device and electronic equipment, the method comprises: obtaining to be stored data, the to be stored data comprises at least two key-value pair data, each key-value pair data in at least two key-value pair data comprises: string and the value corresponding to string;Determine any one key-value pair data in at least two key-value pair data, according to the string size ordering of key-value pair data, determine the storage location of each character in the corresponding key-value bucket in string, and store each character of string to the corresponding key-value bucket;According to the storage order of the value corresponding to string, determine the storage location of the value corresponding to string in value partition and store the value corresponding to string to value partition;According to the storage location of each character in the corresponding key-value bucket in string and the storage location of the value corresponding to string in value partition, establish index mapping table.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing of Fintech, and relates to but is not limited to a data processing method and device and electronic equipment. BACKGROUND

[0002] With the development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to Fintech, but due to the security and real-time requirements of the financial industry, higher requirements are also put forward for the technology.

[0003] In the financial field, a large amount of financial data needs to be stored and queried. In the related technology, the HashMap data structure provided by the Java Development Kit (JDK) is generally used to store financial data, and the data structure can be directly accessed according to the key to read the corresponding data. However, due to the generic mechanism of Java, the data type stored in the HashMap data structure can only be a wrapper type, so the integer type and other basic types of data need to be converted into the corresponding wrapper type before storage. The JDK wrapper type consumes more memory than the basic type, which makes the memory overhead of the HashMap data structure larger. It can be seen that the HashMap data structure consumes a large amount of additional memory space to improve the reading efficiency, so the HashMap data structure is not an optimal data structure selection when the data volume is large and the memory is limited.

[0004] In the related technology, there are also some data structures developed based on third-party open source collection frameworks, such as SparseArray data structure, ConcurrentHashMap data structure, and Guava Cache data structure, etc. However, these data structures are only optimized for a certain scenario, and generally have the problems of large memory overhead or low reading efficiency. SUMMARY

[0005] The embodiments of the present application mainly provide a data processing method, device, electronic equipment and computer storage medium.

[0006] The technical scheme of the embodiments of the present application is as follows:

[0007] The embodiments of the present application provide a data processing method, which comprises:

[0008] Obtaining to-be-stored data, the to-be-stored data comprising at least two key-value pair data, each key-value pair data in the at least two key-value pair data comprising a string and a value corresponding to the string;

[0009] determining a storage position of each character in the string in the corresponding key-value bucket according to the size ordering of the string of the key-value pair data; and storing each character of the string into the corresponding key-value bucket according to the storage position of each character in the corresponding key-value bucket;

[0010] determining a storage position of the value corresponding to the string in the value partition according to the storage order of the value corresponding to the string; and storing the value corresponding to the string into the value partition according to the storage position of the value corresponding to the string in the value partition;

[0011] establishing an index mapping table according to the storage position of each character in the string in the corresponding key-value bucket and the storage position of the value corresponding to the string in the value partition, the index mapping table being used to represent an index mapping relationship between the storage position of the string and the storage position of the value corresponding to the string.

[0012] In some embodiments of the present application, the determining of the storage position of each character in the string in the corresponding key-value bucket according to the size ordering of the string of the key-value pair data comprises: determining an i-th character of the string of the key-value pair data, i being greater than or equal to 1; starting from i being equal to 1, searching for whether there is a character identical to the i-th character in the key-value bucket corresponding to the i-th character, and if there is no character identical to the i-th character, determining the storage position of the i-th character in the corresponding key-value bucket according to the size ordering of the i-th character in the corresponding key-value bucket; and if there is a character identical to the i-th character, increasing the value of i by 1 and repeating the step of searching for whether there is a character identical to the i-th character.

[0013] In the embodiments of the present application, by traversing each character of the string of the key-value pair data, it is searched for whether there is a character identical to the i-th character in the key-value bucket corresponding to the i-th character, so that the i-th character is stored into the corresponding key-value bucket according to the size ordering, so as to obtain the size ordering position of the string when searching for the corresponding value according to the string.

[0014] In some embodiments of the present application, the key-value bucket corresponding to each character comprises at least one bucket, and each bucket in the at least one bucket is used to store the ASCII value of the character and the number of consecutive repetitions of the character; after the step of searching for whether there is a character identical to the i-th character, the method further comprises: determining a target bucket in the key-value bucket corresponding to the i-th character, the target bucket representing a bucket storing the ASCII value of the character identical to the i-th character; and increasing the number of consecutive repetitions of the character in the target bucket by 1.

[0015] In the embodiments of the present application, the bucket array corresponding to each character includes at least one bucket, which is used to store the ASCII value of the corresponding character and the number of consecutive repetitions of the character. When the same character needs to be stored in the target bucket, only the number of consecutive repetitions of the character in the target bucket needs to be increased by 1, so that the storage of the character can be realized, the number of repeated characters in the key-value bucket can be effectively reduced, and the memory overhead can be reduced.

[0016] In some embodiments of the present application, before determining the storage position of the value corresponding to the string in the value partition according to the storage order of the value corresponding to the string, the method further comprises: determining the value partition in which the value corresponding to the string is stored according to the type of the value corresponding to the string and / or the access frequency of the value corresponding to the string.

[0017] In the embodiments of the present application, the value partition in which the value corresponding to the string is stored can be determined according to the type and / or access frequency of the value corresponding to the string. The value partitions are managed according to different types and / or access frequencies of the values. By allocating a continuous memory space, all values can be uniformly stored, and the memory fragmentation caused by the separate storage of different types of values can be reduced.

[0018] In some embodiments of the present application, the establishing of the index mapping table according to the storage positions of each character in the corresponding key-value bucket and the storage position of the value corresponding to the string in the value partition comprises: determining the size ordering position of the string according to the storage positions of each character in the corresponding key-value bucket; determining the partition type code of the value partition in which the value corresponding to the string is stored; determining the internal code of the value corresponding to the string in the value partition according to the storage position of the value corresponding to the string in the value partition; determining the first mapping relationship between the size ordering position of the string and the partition type code of the value partition of the value corresponding to the string, and the second mapping relationship between the size ordering position of the string and the internal code of the value corresponding to the string in the value partition, according to the size ordering position of the string, the partition type code of the value partition of the value corresponding to the string, and the internal code of the value corresponding to the string in the value partition; and establishing the index mapping table according to the first mapping relationship and the second mapping relationship.

[0019] In the embodiments of the present application, the index mapping table can be established based on the size ordering position of the string, the partition type code of the value partition of the value corresponding to the string, and the internal code of the value corresponding to the string in the value partition. When the value corresponding to the string is read according to the string, the storage position of the value corresponding to the string can be determined based on the corresponding index mapping table, so that the value corresponding to the string can be read, the time complexity is low when the value corresponding to the string is read, and the reading efficiency is high.

[0020] In some embodiments of the present application, the method further comprises: obtaining a string of key-value pair data to be read; determining a size order position of the string of key-value pair data to be read; determining, according to the size order position of the string of key-value pair data to be read, the partition type code of the value corresponding to the string of key-value pair data to be read and the internal code of the partition of the value corresponding to the string of key-value pair data to be read in the index mapping table; determining the storage position of the value corresponding to the string of key-value pair data to be read in the value partition according to the partition type code and the internal code of the partition; and reading the value corresponding to the string of key-value pair data to be read according to the storage position of the value corresponding to the string of key-value pair data to be read in the value partition.

[0021] In the embodiments of the present application, by determining the size order position of the string of key-value pair data to be read, the partition type code of the value corresponding to the string of key-value pair data to be read and the internal code of the partition can be determined based on the index mapping table, so that the storage position of the value corresponding to the string of key-value pair data to be read in the value partition can be determined, that is, the value corresponding to the string can be read according to the determined storage position.

[0022] In some embodiments of the present application, the determination of the size order position of the string of key-value pair data to be read comprises: determining the jth character of the string of key-value pair data to be read, j being greater than or equal to 1; starting from j being equal to 1, accumulating the number of characters smaller than the jth character in the key-value bucket corresponding to the jth character, and finding whether there is a character identical to the jth character, if there is no character identical to the jth character, determining the size order position of the string in the key-value bucket according to the number of characters smaller than the jth character; and if there is a character identical to the jth character, increasing the value of j by 1 and repeating the steps of accumulating the number of characters smaller than the jth character in the key-value bucket corresponding to the jth character and finding whether there is a character identical to the jth character until there is no character identical to the jth character.

[0023] In the embodiments of the present application, by sequentially traversing each character of the string of key-value pair data to be read, for the jth character, the number of characters smaller than the jth character is accumulated in the corresponding key-value bucket, and when there is no character identical to the jth character in the corresponding key-value bucket, the size order position of the string in the key-value bucket is determined. Since the size order of the characters is determined based on the ASCII values of the characters, the complexity of O(1) can be used to determine the size order of the characters.

[0024] In some embodiments of the present application, the method further comprises: obtaining a string of key-value pair data to be read; determining, according to the index mapping table, a storage location of a value corresponding to the string of key-value pair data to be read; and reading the value corresponding to the string of key-value pair data to be read according to the storage location of the value corresponding to the string of key-value pair data to be read.

[0025] In the embodiments of the present application, by determining the string of key-value pair data to be read, the storage location of the value corresponding to the string of key-value pair data to be read is determined based on the index mapping table, so that the value corresponding to the string of key-value pair data to be read can be read according to the determined storage location.

[0026] The embodiments of the present application provide a data processing apparatus, which comprises:

[0027] The obtaining module is configured to obtain to-be-stored data, wherein the to-be-stored data comprises at least two key-value pair data, and each key-value pair data in the at least two key-value pair data comprises a string and a value corresponding to the string;

[0028] The first determining module is configured to determine, for any key-value pair data in the at least two key-value pair data, a storage location of each character in the string in a corresponding key-value bucket according to a size order of the string of the key-value pair data, and store each character of the string to the corresponding key-value bucket according to the storage location of the character in the corresponding key-value bucket;

[0029] The second determining module is configured to determine a storage location of the value corresponding to the string in a value partition according to a storage order of the value corresponding to the string, and store the value corresponding to the string to the value partition according to the storage location of the value corresponding to the string in the value partition;

[0030] The processing module is configured to establish an index mapping table according to the storage location of each character in the string in the corresponding key-value bucket and the storage location of the value corresponding to the string in the value partition, wherein the index mapping table is used to represent an index mapping relationship between the string and the storage location of the value corresponding to the string.

[0031] The embodiments of the present application provide an electronic device, which comprises:

[0032] The memory is configured to store executable instructions;

[0033] The processor is configured to execute the executable instructions stored in the memory to implement any one of the above data processing methods.

[0034] The embodiment of the present application provides a computer readable storage medium, which stores executable instructions, and is used for implementing any one of the data processing methods when executed by a processor.

[0035] It can be seen that, in the embodiment of the present application, the to-be-stored data includes at least two key-value pair data, wherein each key-value pair data includes a string and a value corresponding to the string, any one key-value pair data is determined, the storage positions of each character in the string in the corresponding key-value bucket are determined according to the size of the string of the key-value pair data, and each character of the string is stored into the corresponding key-value bucket according to the storage position; the storage position of the value corresponding to the string in the value partition is determined according to the storage order of the value corresponding to the string, and the value corresponding to the string is stored into the value partition according to the storage position; and the index mapping relationship between the storage positions of the string and the value corresponding to the string is established according to the storage positions of each character of the string in the corresponding key-value bucket and the storage position of the value corresponding to the string in the value partition.

[0036] It can be seen that, in the embodiment of the present application, the values of different types are continuously stored in the value partition, so that the values of the basic types do not need to be converted into corresponding wrapper types before being stored, there is no loss of object type wrapping, the memory overhead of the values of the basic types can be effectively reduced, and in addition, the values of different types are continuously stored, so that the memory fragmentation caused by the separate storage of values of different types is reduced. In addition, in the embodiment of the present application, each character of the string is stored into the corresponding key-value bucket according to the size of the string of the key-value pair data, so that when the value corresponding to the string is searched according to the string, the size of the string can be obtained, and the storage position of the value corresponding to the string is determined based on the corresponding index mapping relationship, so that the value corresponding to the string can be read, and the time complexity is low when the value corresponding to the string is read, and the reading efficiency is high. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 An internal storage structure of a HashMap data structure in the related technology provided by the embodiment of the present application is shown in a schematic diagram;

[0038] Figure 2 An internal storage structure of a SparseArray data structure in the related technology provided by the embodiment of the present application is shown in a schematic diagram;

[0039] Figure 3 A flowchart of a data processing method provided by the embodiment of the present application is shown in a schematic diagram;

[0040] Figure 4 A structure schematic diagram of a key-value bucket provided by the embodiment of the present application is shown in a schematic diagram;

[0041] Figure 5A structural schematic diagram of a bucket provided by an embodiment of the present application;

[0042] Figure 6 A schematic diagram of storing each character of a string into a corresponding key-value bucket provided by an embodiment of the present application;

[0043] Figure 7 A comparison diagram of memory overhead of a data processing method provided by an embodiment of the present application;

[0044] Figure 8 A schematic diagram of a first mapping relationship provided by an embodiment of the present application;

[0045] Figure 9 A schematic diagram of a second mapping relationship provided by an embodiment of the present application;

[0046] Figure 10 A schematic diagram of a third mapping relationship provided by an embodiment of the present application;

[0047] Figure 11 A schematic diagram of a fourth mapping relationship provided by an embodiment of the present application;

[0048] Figure 12 A first schematic diagram of a data structure of a data processing method provided by an embodiment of the present application;

[0049] Figure 13 A first schematic diagram of a data structure of a data processing method provided by an embodiment of the present application;

[0050] Figure 14 A flow schematic diagram of a first specific implementation of a data processing method provided by an embodiment of the present application;

[0051] Figure 15 A flow schematic diagram of a second specific implementation of a data processing method provided by an embodiment of the present application;

[0052] Figure 16 An optional component structural schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0053] Figure 17 An optional component structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are only used to explain the present application, and are not used to limit the present application. In addition, the embodiments provided below are used to implement some embodiments of the present application, and the technical solutions described in the embodiments of the present application can be implemented in any combination manner without conflict.

[0055] It should be noted that, in the embodiments of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or device comprising a series of elements not only includes the elements explicitly listed, but also includes other elements not explicitly listed, or further includes elements inherent in the implementation of the method or device. Without more limitations, the element defined by the sentence "comprising a" does not exclude the presence of other related elements (such as steps in the method or units in the device, for example, the unit can be part of the circuit, part of the processor, part of the program or software, etc.) in the method or device comprising the element.

[0056] For example, the data processing method provided by the embodiments of the present application comprises a series of steps, but the data processing method provided by the embodiments of the present application is not limited to the steps described, and similarly, the data processing device provided by the embodiments of the present application comprises a series of modules, but the device provided by the embodiments of the present application is not limited to comprising the modules explicitly described, and can further comprise modules required to be set when obtaining relevant information or processing based on the information.

[0057] In the related art, based on the data structure provided by JDK, data can be stored. Due to the generic mechanism of Java, the data of the basic type needs to be converted into the corresponding wrapper class to be stored. Here, the wrapper class is to convert the data of the basic type into an object. The storage structure of a Java object usually includes an object header, instance data and alignment padding. The object header is used to store the mark bit (Mark Word) and type pointer (Class Pointer) of the object. The object header occupies 4 bytes on a 32-bit system and 8 bytes on a 64-bit system. The mark bit is used to mark the runtime information of an object, and the type pointer indicates a pointer of the object to its metadata. The type pointer occupies 4 bytes on a 32-bit system and 8 bytes on a 64-bit system. It should be noted that if the instance object is an array, 4 bytes will be additionally occupied to store the length of the array; the instance data is used to store all member variables defined by the object. When the occupied bytes of the object are not multiples of 8, the Java virtual machine (JVM) will add appropriate alignment padding to make the occupied bytes of the object multiples of 8.

[0058] For example, an instance object internally stores an int type field and a byte type field, and the instance object occupies 24 bytes on a 64-bit system, wherein the mark bit of the object header occupies 8 bytes, the type pointer of the object header occupies 4 bytes, the int type field in the instance data occupies 4 bytes, the byte type field occupies 1 byte, and the alignment padding part occupies 7 bytes.

[0059] For the HashMap data structure provided based on JDK, the internal storage structure is as shown in Figure 1 Figure 1 FIG. 1 is a schematic diagram of the internal storage structure of the HashMap data structure in the related technology provided by an embodiment of the present application, the HashMap data structure is composed of a hash bucket array 101 and a plurality of data nodes (Node) 102, wherein the hash bucket array 101 is an array for storing the plurality of data nodes 102, and is composed of an object header with an array length and an array element set; the data node Node class is inherited from Map.Entry, and is a basic storage unit in the HashMap data structure, and the internal structure of the data node includes: key-value data (Key, Value), a hash value (Hash) of the data node, and a reference (next) to the next data node.

[0060] According to the internal structure of the data node, taking an Integer data type as an example, it can be determined that the storage of one data node includes: the mark bit of the object header occupies 8 bytes, the type pointer of the object header occupies 4 bytes, the key in the instance data occupies 4 bytes, the value in the instance data occupies 4 bytes, the hash value of the data node occupies 4 bytes, the reference to the next data node occupies 4 bytes, and the alignment padding part occupies 4 bytes, so the storage of one data node is 32 bytes, and if 32 Integer key-value data are stored by using the data node, 32 data nodes occupy 1024 bytes in total. It can be seen that the HashMap data structure provided based on JDK consumes a large amount of additional memory space in exchange for its read-write performance, which makes the HashMap data structure not an optimal data structure selection in the application of large data and limited memory.

[0061] Meanwhile, due to the generic mechanism of Java, the data type stored in the HashMap data structure can only be a wrapper type. Therefore, the basic type data such as an integer needs to be converted into a corresponding wrapper type before being stored, which causes additional performance loss of boxing and unboxing when storing or reading data, and the memory overhead of storing the wrapper type data is also larger than that of storing the basic type data.

[0062] ​For example, when creating an array of integer data using the primitive type `int[]`, only 4 bytes are needed to store the integer data for each element. However, when creating an array using the wrapper type `Integer[]`, in addition to allocating 4 bytes to store the instance data, an extra 12 bytes are needed to store its object header, and 4 bytes are needed to store its reference relationships. Therefore, the actual overhead is 4 + 12 + 4 = 20 bytes. It can be seen that wrapper type data incurs an additional 16 bytes of overhead. Therefore, the HashMap data structure is not suitable for storing large amounts of data with limited memory; using primitive types to store data is more memory-efficient.

[0063] Among related technologies, there are also some data structures developed based on third-party open-source collection frameworks, such as the SparseArray data structure, the fastutil data structure, the ConcurrentHashMap data structure, and the Guava Cache data structure.

[0064] The SparseArray data structure, also known as a sparse array, is a data structure provided by the Android system for storing integer data. Unlike the HashMap data structure, which uses a one-dimensional array and a singly linked list, the SparseArray data structure uses two one-dimensional arrays, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of the internal storage structure of the SparseArray data structure in the related technology provided in the embodiments of this application. One array is used to store mKey, and another array is used to store mValue. The indices of mKey and mValue are also shown. [] They are mutually corresponding, and SparseArray is more efficient and lightweight than HashMap. However, the disadvantages of SparseArray are: (1) it can only store integer keys; (2) it needs to copy the array when performing insertion and garbage collection (GC) operations, which is less efficient; (3) SparseArray reads data by binary search, with a time complexity of O(logN). When the amount of data is huge, the efficiency of reading data is low, and it is slower than HashMap.

[0065] The fastutil data structure is a high-performance collection framework, and the internal storage structure of the fastutil data structure is similar to that of the SparseArray data structure. Different types of values need to be stored separately, and only basic type values can be stored. Therefore, the fastutil data structure does not need to box and unbox the basic type data, so that the memory overhead of the fastutil data structure is low when storing and reading data, and the reading speed is fast. However, the fastutil data structure has the following disadvantages: (1) the storage of the key is not optimized; (2) the type of the value needs to be determined in advance, and then the corresponding object is created, and different types of values need to create different objects. When there are multiple types of values, multiple objects need to be created, and when searching for values, multiple objects need to be traversed, which reduces the reading efficiency.

[0066] The ConcurrentHashMap data structure is a thread-safe HashMap data structure, and its internal storage structure is similar to that of the HashMap data structure, which is also based on array, linked list and red-black tree structure to store data. Compared with the HashTable data structure in the same JDK, it has less lock competition and higher read-write efficiency. However, the storage of the ConcurrentHashMap data structure is still the HashMap data structure, and its disadvantage is still the large memory overhead.

[0067] The Guava Cache data structure is a local cache tool library, which uses multiple segments to provide a thread-safe storage structure that supports high-concurrency scenarios. However, the Guava Cache data structure has the disadvantage of large memory overhead, which consumes a large amount of additional memory space to improve reading efficiency.

[0068] In view of the problems in the related art, the technical scheme of the embodiments of the present application is proposed. The embodiments of the present application can be applied to terminals and servers. Here, the terminal can be a thin client, a thick client, a handheld or laptop device, a microprocessor-based system, a programmable consumer electronics product, a network personal computer, a small computer system, etc. The server can be a small computer system, a large computer system, and a distributed cloud computing technology environment including any of the above systems, etc.

[0069] Electronic devices such as terminals, servers, etc. can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in local or remote computer system storage media including storage devices.

[0070] Figure 3 A flowchart of a data processing method provided by an embodiment of the present application is shown in FIG. 1, which can include the following steps: Figure 3

[0071] Step 301: Obtain to-be-stored data, the to-be-stored data including at least two key-value pair data, each key-value pair data in the at least two key-value pair data including a string and a value corresponding to the string.

[0072] In an embodiment of the present application, the to-be-stored data includes at least two key-value pair data, each key-value pair data in the key-value pair data including a string and a value corresponding to the string. Here, the string represents a unique identifier of the key-value pair data, denoted as key, and the value corresponding to the string represents actual storage content of the to-be-stored data, denoted as value. Each key-value pair data can be denoted as (key, value). The key-value pair data is actually a mapping model, and a hash function is used to realize mapping of the string to the value, i.e., the string is a unique identifier for finding the corresponding value, and each string corresponds to a value.

[0073] For example, a key-value pair data is (“pbc_ldt”, “-99”), where the string “pbc_ldt” is a unique identifier of the key-value pair data, and the value “-99” is actual storage content of the key-value pair data. According to the string “pbc_ldt”, a unique value “-99” can be determined.

[0074] Step 302: Determine any key-value pair data in the at least two key-value pair data, sort the strings of the key-value pair data according to their sizes, determine storage positions of each character in the string in a corresponding key-value bucket, and store each character of the string to the corresponding key-value bucket according to the storage positions of the characters in the corresponding key-value bucket.

[0075] ​In the embodiments of the present application, the string of the key-value pair data can be a string of characters composed of numbers, letters and underscores, and each character in the string can be extracted. For example, a string is "pbc_ld2", which includes the first character "p", the second character "b", the third character "c", the fourth character "_", the fifth character "l", the sixth character "d", and the seventh character "2".

[0076] In the embodiments of the present application, the key-value bucket represents a storage space for storing each character in the string of the key-value pair data, that is, each character in the string is stored in the corresponding key-value bucket, and each character corresponds to a key-value bucket.

[0077] For example, the first string is "abc", the first character of which is "a", the second character of which is "b", and the third character of which is "c"; the second string is "def", the first character of which is "d", the second character of which is "e", and the third character of which is "f". The character "a" of the first string and the character "d" of the second string are stored in the key-value bucket corresponding to the first character, the character "b" of the first string and the character "e" of the second string are stored in the key-value bucket corresponding to the second character, and the character "c" of the first string and the character "f" of the second string are stored in the key-value bucket corresponding to the third character.

[0078] In the embodiments of the present application, the comparison between strings is a comparison between single characters or strings in dictionary order, and the size of the characters is generally compared according to the American Standard Code for Information Interchange (ASCII) value. The size comparison between strings is based on the comparison of the first character from the left of the string. The string with the larger first character is determined as the larger string, and the string with the smaller first character is determined as the smaller string. If the first characters are equal, the size of the next character is compared, and the size comparison between strings is determined according to the size relationship between the unequal characters. If all characters in the strings are equal, it is considered as the same string.

[0079] For example, one string is "EFG", and the other string is "EAFG". The size of the two strings is compared. The first character of the two strings is "E". Therefore, the size of the next character is compared. The second characters of the two strings are "F" and "A" respectively. Since the ASCII code value of the character "F" is greater than that of the character "A", the string "EFG" is greater than the string "EAFG".

[0080] In the embodiment of the present application, according to the string size sorting of the key-value pair data, the storage positions of each character in the corresponding key-value bucket are determined from the first character on the left of the string, and each character of the string is stored in the corresponding key-value bucket according to the storage position of each character in the corresponding key-value bucket, and a terminator "-1" is stored after the last character of the string to identify a string, and the terminator "-1" is stored in the corresponding key-value bucket.

[0081] Step 303: determining the storage position of the value corresponding to the string in the value partition according to the storage order of the value corresponding to the string, and storing the value corresponding to the string into the value partition according to the storage position of the value corresponding to the string in the value partition.

[0082] In the embodiment of the present application, the value partition represents a storage space for storing the value corresponding to the string, and the value corresponding to the string can be basic type data or wrapped type data. The value partition can directly store the basic type data, thereby reducing the memory overhead generated when the basic type data is converted into wrapped type data and then stored.

[0083] In the embodiment of the present application, the storage position of the value corresponding to the nth to-be-stored string is the nth position in the value partition. The value corresponding to the string is stored in the value partition according to the storage position of the value corresponding to the string in the value partition.

[0084] Step 304: establishing an index mapping table according to the storage positions of each character in the corresponding key-value bucket and the storage position of the value corresponding to the string in the value partition, and the index mapping table is used to represent the index mapping relationship between the storage positions of the string and the value corresponding to the string.

[0085] In the embodiment of the present application, the index mapping table is established according to the storage positions of each character in the corresponding key-value bucket and the storage position of the value corresponding to the string in the value partition. When the value corresponding to the string is read according to the string, the index mapping relationship between the storage positions of the string and the value corresponding to the string can be determined through the index mapping table, that is, the storage position of the value corresponding to the string can be determined, and thus the value corresponding to the string can be read.

[0086] It can be seen that, in the embodiment of the application, by storing different types of values continuously in the value partition, the values of the basic type do not need to be converted into the corresponding wrapper type before storage, there is no loss of object type wrapping, and the memory overhead of the values of the basic type can be effectively reduced. In addition, in the embodiment of the application, the characters of the string are stored in the corresponding key-value bucket according to the size sorting of the string of the key-value pair data, so that when the corresponding value is searched according to the string, the size sorting position of the string can be obtained, and the storage position of the value corresponding to the string is determined based on the corresponding index mapping relationship, so that the value corresponding to the string can be read. Therefore, the time complexity is low when the value corresponding to the string is read, and the reading efficiency is high.

[0087] In actual application, steps 301 to 304 can be implemented based on a processor of an electronic device. The processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that the electronic device implementing the functions of the processor can also be other devices, and the embodiments of the application are not limited thereto.

[0088] In some embodiments, the determining of the storage positions of the characters in the corresponding key-value bucket according to the size sorting of the string of the key-value pair data comprises:

[0089] determining an i-th character of the string of the key-value pair data, i being greater than or equal to 1;

[0090] starting from i being equal to 1, searching, in the key-value bucket corresponding to the i-th character, whether there is a character identical to the i-th character, and if there is no character identical to the i-th character, determining the storage position of the i-th character in the corresponding key-value bucket according to the size sorting of the i-th character in the corresponding key-value bucket;

[0091] If there is a character same as the i-th character, let the value of i be added by 1, and repeat the step of searching whether there is a character same as the i-th character.

[0092] In the embodiment of the present application, the key value bucket corresponding to each character in the string can be represented in the form of a bucket array, where the bucket array is a two-dimensional array. Figure 4 A structural diagram of the key value bucket provided by the embodiment of the present application is shown in FIG. 2. Figure 4 The horizontal axis represents m key value buckets, where m = 1, 2, 3, …, 64, and the m-th key value bucket represents the key value bucket corresponding to the m-th character in the string. It should be noted that the maximum value of m is 64, that is, there are at most 64 key value buckets, which can store a 64-bit character string.

[0093] In the embodiment of the present application, the m key value buckets can be represented as m bucket arrays, and the vertical axis represents that each bucket array includes n buckets, where n = 1, 2, 3, …, the n-th bucket represents the n-th position in the corresponding bucket array according to the ascending order of the characters. Each position in the bucket array represents a bucket, so the coordinates of the bucket can be represented by (m, n). It should be noted that for the first key value bucket, it stores the first character of the string, and the ASCII value of the character ranges from 32 to 126. Subtracting 32 from the ASCII value of the first character, the ASCII value of the first character ranges from 0 to 95, so for the first key value bucket, the maximum value of n in the bucket array is 95, that is, there are at most 96 buckets in the first bucket array.

[0094] In the embodiment of the present application, Figure 5 A structural diagram of the bucket provided by the embodiment of the present application is shown in FIG. 3. Figure 5 The bucket is used to store the ASCII value of a character and the number of consecutive repetitions of the character, and the ASCII value of the character and the number of consecutive repetitions of the character in the bucket occupy 8 bytes respectively.

[0095] Exemplarily, a string includes 8 characters, the ASCII values of the 8 characters are: char1, char2, char3, char4, char5, char6, char7, char8 respectively, and the continuous repetition times corresponding to the 8 characters are: count1, count2, count3, count4, count5, count6, count7, count8 respectively, then the 8 characters of the string are stored in 8 buckets respectively, which are: [char1, count1], [char2, count2], [char3, count3], [char4, count4], [char5, count5], [char6, count6], [char7, count7], [char8, count8].

[0096] In the embodiment of the present application, the continuous repetition times of a character represent the number of characters with the same ASCII value as the i-th character in the target range of the bucket array corresponding to the i-th character. Here, the target range of the bucket array corresponding to the i-th character includes two cases: (1) i = 1, that is, the i-th character is the first character of the string, and the target range of the bucket array corresponding to the i-th character is: all the buckets in the bucket array corresponding to the first character; (2) i > 1, that is, the i-th character is a character other than the first character in the string, and the target range of the bucket array corresponding to the i-th character is: the buckets in the bucket array corresponding to the i-th character, in which the ASCII values of the first character to the i-1-th character are equal to the ASCII values of the first character to the i-1-th character in the string to which the i-th character belongs.

[0097] It should be noted that, since the characters in the bucket array are stored in ascending order, the target range of the bucket array corresponding to the i-th character includes the highest point high and the lowest point low, and the ASCII value of the character corresponding to the lowest point low is less than the ASCII value of the character corresponding to the highest point high, so the target range can be represented as [lowest point low, highest point high].

[0098] In the embodiment of the present application, if there is a character with the same ASCII value as the i-th character in the target range of the bucket array corresponding to the i-th character, it means that the i-th character needs to be stored repeatedly, and only the continuous repetition times of the character in the bucket with the same ASCII value as the i-th character in the bucket array need to be increased by 1, so as to realize the storage of the i-th character.

[0099] If the ASCII value of the i-th character does not exist in the target range of the bucket array corresponding to the i-th character, it is indicated that the i-th character does not exist in the corresponding bucket array, and the storage position of the i-th character in the corresponding bucket array is determined according to the size of the i-th character in the target range of the corresponding bucket array, and the ASCII value of the i-th character and the continuous repetition number of the i-th character are stored in the bucket at the storage position.

[0100] For example, the string to be stored is "pbc", that is, the first character "p" of the string is stored in the first key-value bucket, the second character "b" is stored in the second key-value bucket, and the third character "c" is stored in the third key-value bucket. Figure 6 An example of storing each character of the string in the corresponding key-value bucket is provided in the embodiment of the present application, as shown in FIG. 1. Figure 6 The bucket array of the first key-value bucket includes four buckets, which are [98, 1], [100, 1], [112, 2] and [114, 1]. The bucket array of the second key-value bucket includes five buckets, which are [114, 1], [95, 1], [98, 1], [100, 1] and [105, 1]. The bucket array of the third key-value bucket includes five buckets, which are [97, 1], [54, 1], [95, 1], [108, 1] and [112, 1]. As can be seen, five characters are stored in the first key-value bucket, and the ASCII values of the five characters are 98, 100, 112, 112 and 114, respectively. Five characters are stored in the second key-value bucket, and the ASCII values of the five characters are 114, 95, 98, 100 and 105, respectively. Five characters are stored in the third key-value bucket, and the ASCII values of the five characters are 97, 54, 95, 108 and 112, respectively. Therefore, it can be determined that the ASCII values of the characters of the first string are {98, 114, 97}, the ASCII values of the characters of the second string are {100, 95, 54}, the ASCII values of the characters of the third string are {112, 98, 95}, the ASCII values of the characters of the fourth string are {112, 100, 108}, and the ASCII values of the characters of the fifth string are {114, 105, 112}.

[0101] The target range of the bucket array corresponding to the first character is all the buckets in the first key-value bucket. The ASCII value of the first character "p" of the string to be stored is 112, so it can be determined that there is a bucket equal to the ASCII value of the character "p" in the target range of the corresponding bucket array, which is bucket [112, 2], therefore, the number of continuous repetitions of the character in the bucket [112, 2] is added by 1, so that the storage data in the bucket is updated to [112, 3], indicating that the ASCII value of the character stored in the bucket is 112, and the number of repetitions of the character is 3, that is, 3 characters "p" are stored in the first key-value bucket.

[0102] The target range of the bucket array corresponding to the second character is the buckets in the bucket array corresponding to the second character, the ASCII value of the first character of which is equal to the ASCII value of the first character "p" of the string to be stored "pbc", as shown in Figure 6 The first character of the buckets [98, 1] and [100, 1] is 112, which is equal to the ASCII value of the first character "p" of the string to be stored "pbc", so it can be determined that the target range of the bucket array corresponding to the second character is the buckets [98, 1] and [100, 1]. The ASCII value of the second character "b" of the string to be stored is 98, so it can be determined that there is a bucket equal to the ASCII value of the character "b" in the target range of the corresponding bucket array, which is bucket [98, 1], therefore, the number of continuous repetitions of the character in the bucket [98, 1] is added by 1, so that the storage data in the bucket is updated to [98, 2], indicating that the ASCII value of the character stored in the bucket is 98, and the number of repetitions of the character is 2, that is, 2 characters "b" are stored in the second key-value bucket.

[0103] The target range of the bucket array corresponding to the third character is the buckets in the bucket array corresponding to the third character, the ASCII value of the first character of which is equal to the ASCII value of the first character "p" of the string to be stored "pbc", and the ASCII value of the second character is equal to the ASCII value of the second character "b" of the string to be stored "pbc", as shown in Figure 6As shown, the first character of the string to which the bucket [95, 1] belongs is 112, and the second character is 98, which is equal to the ASCII value of the first character "p" and the second character "b" of the to-be-stored string "pbc", so it can be determined that the target range of the bucket array corresponding to the third character is: bucket [95, 1]. The ASCII value of the second character "c" of the to-be-stored string is 99, so it can be determined that there is no bucket equal to the ASCII value of the character "c" in the target range of the corresponding bucket array. Therefore, according to the size ordering of the character "c" in the target range of the corresponding bucket array, the character "c" is stored in the first position after the bucket [95, 1], and then the third key-value bucket including 6 buckets, [97, 1], [54, 1], [95, 1], [99, 1], [108, 1], [112, 1], is obtained, which indicates that the fourth position of the second key-value bucket stores the bucket [99, 1], that is, the second key-value bucket stores one character "c".

[0104] In the embodiment of the present application, when storing each character in the string into the corresponding key-value bucket, it is required to meet that the string is stored in dictionary order, that is, the string is arranged in ascending order, so it is required to determine the storage position of each character in the string in the corresponding key-value bucket, and the storage position of the i-th character in the string is determined by traversing the i-th character in the string of the key-value pair data, and the storage position of the i-th character in the corresponding bucket array is determined one by one, where i is greater than or equal to 1.

[0105] In the embodiment of the present application, starting from i equal to 1, when i is equal to 1, the first character of the string of the key-value pair data is determined, and it is judged whether there is a character equal to the ASCII value of the first character in the bucket array corresponding to the first character. If not, it indicates that there is no first character in the corresponding bucket array, and the size ordering position of the first character in the corresponding bucket array is determined according to the size ordering of the first character in the corresponding bucket array, where the size ordering is recorded as position, position = the number of characters with ASCII values less than the ASCII value of the first character in the bucket array corresponding to the first character. According to the position, the first character is stored in the corresponding bucket array, that is, the first character is stored in the position after all characters with ASCII values less than the ASCII value of the first character in the corresponding bucket array.

[0106] In the embodiment of the present application, if there is a character equal to the ASCII value of the first character in the bucket array corresponding to the first character, it indicates that the size ordering of the string cannot be determined according to the first character of the string of the key-value pair data, and the size ordering of the string needs to be determined according to the size ordering of the next character, so the value of i is increased by 1, and the step of searching whether there is a character equal to the i+1-th character is performed.

[0107] In the embodiment of the present application, when i is greater than 1, the i-th character of the string of the key-value pair data is determined, the number of all characters in the bucket array corresponding to the i-th character is obtained, and the target range of the bucket array corresponding to the i-th character is determined. The target range of the bucket array corresponding to the i-th character can be determined according to the number of characters greater than the (i-1)-th character, the number of characters equal to the (i-1)-th character, and the number of characters one level higher than the (i-1)-th character in the target range of the bucket array corresponding to the (i-1)-th character. Here, the character one level higher than the (i-1)-th character refers to the (i-1)-th character of all strings greater than the string to which the highest point high of the target range of the bucket array corresponding to the (i-1)-th character belongs in the bucket array corresponding to the (i-1)-th character.

[0108] For example, the string to be stored is “pbc”, and each character of the string to be stored “pbc” needs to be stored in the key-value bucket shown in the figure, that is, the first character “p” of the string needs to be stored in the first key-value bucket, the second character “b” of the string needs to be stored in the second key-value bucket, and the third character “c” of the string needs to be stored in the third key-value bucket. Figure 6

[0109] For example, the third character “c” of the string to be stored is “pbc”. According to the foregoing analysis, the target range of the bucket array corresponding to the second character “b” of the string to be stored “pbc” is [98, 1] and [100, 1], that is, the highest point of the target range of the bucket array corresponding to the second character is [100, 1], and the string to which [100, 1] belongs is the fourth string. Since the string greater than the fourth string is the fifth string, it can be determined that the character one level higher than the second character “b” of the string to be stored “pbc” is the second character of the fifth string.

[0110] In the embodiment of the present application, the sum of the number of characters greater than the (i-1)-th character and the number of characters one level higher than the (i-1)-th character in the target range of the bucket array corresponding to the (i-1)-th character is recorded as bigs, and the number of characters equal to the (i-1)-th character in the target range of the bucket array corresponding to the (i-1)-th character is recorded as preEquals. The initial value of bigs is 0, and the initial value of preEquals is 0.

[0111] ​Before determining the target range of the bucket array corresponding to the i-th character, the bucket array corresponding to the (i-1)-th character is traversed to determine bigs and preEquals. It should be noted that when the (i-1)-th character has a stop symbol "-1" in the high-level character, the stop symbol "-1" is not counted in the number of high-level characters of the (i-1)-th character. Then the highest point high and the lowest point low of the target range are determined, where the highest point high = the number of all characters in the bucket array corresponding to the i-th character - bigs; and the lowest point low = the highest point high - preEquals.

[0112] In the target range of the bucket array corresponding to the i-th character, it is determined whether there is a character with an ASCII value equal to that of the i-th character. If not, the storage position of the i-th character in the corresponding key-value bucket is determined according to the size order of the i-th character in the corresponding bucket array. The storage position of the i-th character in the corresponding key-value bucket is denoted as k, k = high - bigs', where bigs' represents the number of characters with an ASCII value greater than that of the i-th character in the target range of the bucket array corresponding to the i-th character.

[0113] In the target range of the bucket array corresponding to the i-th character, it is determined whether there is a character with an ASCII value equal to that of the i-th character. If not, the storage position of the i-th character in the corresponding key-value bucket is determined according to the size order of the i-th character in the corresponding bucket array. The storage position of the i-th character in the corresponding key-value bucket is denoted as k, k = high - bigs', where bigs' represents the number of characters with an ASCII value greater than that of the i-th character in the target range of the bucket array corresponding to the i-th character.

[0114] In the target range of the bucket array corresponding to the i-th character, if there is a character with an ASCII value equal to that of the i-th character, it means that the size order of the string cannot be determined according to the i-th character of the string of the key-value pair data, and the size order of the string needs to be determined according to the size order of the next character. Therefore, the value of i is increased by 1, and the step of searching whether there is a character identical to the (i+1)-th character is performed.

[0115] It can be seen that, in the embodiments of the present application, by traversing each character of the string of the key-value pair data, it is determined whether there is a character identical to the i-th character in the bucket array corresponding to the i-th character, so that the i-th character is stored in the corresponding key-value bucket according to the size order, so as to obtain the size order position of the string when searching for the corresponding value according to the string.

[0116] In some embodiments, the key-value bucket corresponding to each character includes at least one bucket, and each bucket in the at least one bucket is used to store the ASCII value of the character and the number of consecutive repetitions of the character.

[0117] After the character identical to the i-th character, the method further comprises:

[0118] In the key-value bucket corresponding to the i-th character, a target bucket is determined, the target bucket representing a bucket storing an ASCII value of the character identical to the i-th character;

[0119] The number of continuous repetitions of the character in the target bucket is incremented by 1.

[0120] In the embodiment of the application, if there is a character identical in ASCII value to the i-th character in the target range of the key-value bucket corresponding to the i-th character, it is indicated that there is a bucket storing the i-th character in the key-value bucket corresponding to the i-th character. Therefore, the target bucket is determined. Here, the target bucket represents a bucket storing an ASCII value of the character identical to the i-th character. The number of continuous repetitions of the character in the target bucket is incremented by 1, so that the storage of the i-th character is realized.

[0121] In the embodiment of the application, the maximum value of the number of continuous repetitions of the character can be 128. Based on the number of continuous repetitions of the character in the bucket, a bucket can be used to store 128 identical characters at most. If a separate storage mode of each character is used, 128 buckets are needed to store 128 identical characters respectively. This storage mode makes the number of repeated characters in the key-value bucket larger and the memory overhead larger.

[0122] In the embodiment of the application, the storage of the corresponding character is realized by incrementing the number of continuous repetitions of the character in the target bucket. This storage mode can reduce the number of repeated characters in the key-value bucket. The higher the repetition rate of the identical character is, the larger the memory overhead saved is. In actual application scenarios, the strings of the key-value pair data generally follow a certain naming rule, which makes the identical characters exist in each string and the repetition rate high. Therefore, the larger the amount of key-value pair data is, the higher the repetition rate of the identical characters in each string is, and the smaller the memory space consumed when stored in the corresponding key-value bucket is.

[0123] In the embodiment of the application, the strings of 10,000 key-value pair data are compared and verified, Figure 7 The comparison chart of the memory overhead of the data processing method provided in the embodiment of the application is shown in FIG. 2. Figure 7 Compared with the memory space consumed by the HashMap data structure and the fastUtil data structure, the memory overhead of the data processing method provided in the embodiment of the application is less than 20% of the memory overhead of the fastUtil data structure, and the memory usage is greatly saved.

[0124] It can be seen that, in the embodiments of the present application, the key-value bucket corresponding to each character includes at least one bucket for storing the ASCII value of the corresponding character and the number of consecutive repetitions of the character. When the same character needs to be stored in the target bucket, only the number of consecutive repetitions of the character in the target bucket needs to be incremented by 1, so that the storage of the character can be realized, which can effectively reduce the number of repeated characters in the key-value bucket and reduce the memory overhead.

[0125] In some embodiments, before determining the storage location of the value corresponding to the string in the value partition according to the storage order of the value corresponding to the string, the method further comprises:

[0126] According to the type of the value corresponding to the string and / or the access frequency of the value corresponding to the string, the value partition for storing the value corresponding to the string is determined.

[0127] In the embodiments of the present application, the type of the value corresponding to the string can include object type, byte type, floating-point type, and integer type, and the value partition corresponding thereto can be divided into four partitions for storing values of object type, byte type, floating-point type, and integer type.

[0128] In the embodiments of the present application, when the value partition for storing the value corresponding to the string is determined according to the type of the value corresponding to the string, the type of the value corresponding to the string is first determined, and then the value corresponding to the string is stored in the last position of the corresponding partition.

[0129] For example, the values stored in the integer type partition of the value partition include [10, 20, 30], and if the value corresponding to the string of the key-value pair data to be stored is "40", it can be determined that the value corresponding to the string is of integer type, and therefore the value corresponding to the string is stored in the last position of the integer type partition, so that the values stored in the integer type partition include [10, 20, 30, 40].

[0130] In the embodiments of the present application, the value corresponding to the string can also be divided into values with high access frequency and values with low access frequency, and the value partition corresponding thereto can also be divided into two regions for storing high-frequency access region and low-frequency access region. The high-frequency access region is used to store values with high access frequency to achieve the purpose of compressing the memory space.

[0131] In the embodiments of the present application, when the value partition for storing the value corresponding to the string is determined according to the access frequency of the value corresponding to the string, it is first determined whether the value corresponding to the string exists in the high-frequency access region. If it exists, it can be determined that the value corresponding to the string has been stored in the high-frequency access region and does not need to be stored repeatedly. If it does not exist, it means that the value corresponding to the string is a value with low access frequency, which is stored in the low-frequency access region.

[0132] In the embodiments of the present application, according to the type and the access frequency of the value corresponding to the string, the value partition can also be divided into eight zones, which are respectively: a low-frequency access zone for storing object type, a low-frequency access zone for storing byte type, a low-frequency access zone for storing float type, a low-frequency access zone for storing integer type, a high-frequency access zone for storing object type, a high-frequency access zone for storing byte type, a high-frequency access zone for storing float type, and a high-frequency access zone for storing integer type.

[0133] As can be seen, in the embodiments of the present application, the value partition for storing the value corresponding to the string can be determined according to the type and / or the access frequency of the value corresponding to the string, and the value partitions are managed by partition according to different types and / or access frequencies of the value. By allocating a continuous memory space, all values can be uniformly stored, and the memory fragmentation caused by separate storage of different types of values can be reduced.

[0134] In some embodiments, the index mapping table is established according to the storage positions of each character in the string in the corresponding key-value bucket and the storage position of the value corresponding to the string in the value partition, and the index mapping table comprises:

[0135] determining the size ordering position of the string according to the storage positions of each character in the string in the corresponding key-value bucket;

[0136] determining the partition type code of the value partition for storing the value corresponding to the string, and determining the internal code of the value corresponding to the string in the value partition according to the storage position of the value corresponding to the string in the value partition;

[0137] determining the first mapping relationship between the size ordering position of the string and the partition type code of the value partition for the value corresponding to the string, and the second mapping relationship between the size ordering position of the string and the internal code of the value corresponding to the string in the value partition, according to the size ordering position of the string, the partition type code of the value partition for the value corresponding to the string, and the internal code of the value corresponding to the string in the value partition;

[0138] establishing the index mapping table according to the first mapping relationship and the second mapping relationship.

[0139] In the embodiments of the present application, the size ordering position of the string represents the size ordering position of the string of the key-value pair data to be stored in all strings. Since the storage positions of each character in the string in the corresponding key-value bucket are determined according to the size ordering of each character in the target range of the corresponding bucket array, the size ordering position of the string can be determined according to the storage positions of each character in the string in the corresponding key-value bucket.

[0140] In the embodiment of the present application, the size comparison between strings is based on the comparison of the first character from the left of the string. The string with the larger first character is determined as the larger string, and the string with the smaller first character is determined as the smaller string. If the first characters are equal, the size of the next character is compared, until there is a character that is not equal. The size order between the strings is determined according to the size relationship between the characters that are not equal.

[0141] For example, there are four strings, string 1, string 2, string 3, and string 4. The ASCII values of the first characters of the four strings are char1, char2, char3, and char4, respectively. The first characters of the four strings are stored in the corresponding bucket array in ascending order. If char1 < char2 < char3 < char4, the size order of the four strings can be determined as string1 < string2 < string3 < string4. The size order positions of the four strings are: the size order position of string1 is the first, recorded as Position1 = 1; the size order position of string2 is the second, recorded as Position2 = 2; the size order position of string3 is the third, recorded as Position3 = 3; and the size order position of string4 is the fourth, recorded as Position1 = 4.

[0142] In the embodiment of the present application, the partition type code represents the type code corresponding to each partition of the value partition. For example, according to the type of the value corresponding to the string, the type of the value partition can be divided into four partitions, including: a partition for storing object types, the corresponding partition type code can be 00; a partition for storing byte types, the corresponding partition type code can be 01; a partition for storing floating point types, the corresponding partition type code can be 10; and a partition for storing integer types, the corresponding partition type code can be 11.

[0143] Exemplarily, according to the type and the access frequency of the value corresponding to the string, the value partition can be divided into eight partition types, including: the partition type code corresponding to the low-frequency access area for storing the object type can be 000; the partition type code corresponding to the low-frequency access area for storing the byte type can be 001; the partition type code corresponding to the low-frequency access area for storing the float type can be 010; the partition type code corresponding to the low-frequency access area for storing the integer type can be 011; the partition type code corresponding to the high-frequency access area for storing the object type can be 100; the partition type code corresponding to the high-frequency access area for storing the byte type can be 101; the partition type code corresponding to the high-frequency access area for storing the float type can be 110; and the partition type code corresponding to the high-frequency access area for storing the integer type can be 111.

[0144] In the embodiment of the application, the internal coding of the value corresponding to the string in the value partition is used to indicate the storage position of the value corresponding to the string in the partition of the corresponding type. Since the storage position of the value corresponding to the string in the value partition is determined according to the storage order, the internal coding of the value corresponding to the string in the value partition can be determined according to the storage order of the value corresponding to the string in the value partition.

[0145] Exemplarily, the values stored in the integer type partition in the value partition include [10, 20, 30], if the value corresponding to the string of the key-value pair data to be stored is “40”, it can be determined that the value corresponding to the string is of the integer type, and thus the value corresponding to the string is stored in the last position of the integer type partition, so that the values stored in the integer type partition include [10, 20, 30, 40], the storage order of the value corresponding to the string in the value partition is the fourth stored data, and thus the internal coding of the value corresponding to the string in the value partition can be determined as 4.

[0146] In the embodiment of the application, the partition start coding of the value corresponding to the string in the value partition can also be determined. Here, the partition start coding is used to indicate the storage position of the value with the internal coding of 1 in each type of partition in the value partition. The partition start coding can be defined separately, and the entire index mapping table can be avoided from being updated in the value partition in-place copy movement.

[0147] Exemplarily, according to the type and the access frequency of the value corresponding to the string, the value partition can be divided into eight partition types, and thus the partition start coding can be set as an integer array including eight elements corresponding to the start coding of the eight types of partitions.

[0148] In the embodiment of the present application, the storage position of the value corresponding to the string in the value partition can be determined by determining the type of the partition in which the value corresponding to the string is stored, the internal coding in the partition, and the partition start coding of the partition.

[0149] In the embodiment of the present application, the first mapping relationship is established according to the size sorting position of the string and the partition type coding of the value partition corresponding to the string, and an example is shown in Figure 8 . Figure 8 A schematic diagram of the first mapping relationship provided by the embodiment of the present application includes the size sorting position Position of the string and the partition type coding of the value partition corresponding to the string. Here, the partition type coding is stored in decimal numbers and can also be converted into binary numbers.

[0150] In the embodiment of the present application, the second mapping relationship is established according to the size sorting position of the string and the internal coding of the value corresponding to the string in the value partition, and an example is shown in Figure 9 . Figure 9 A schematic diagram of the second mapping relationship provided by the embodiment of the present application stores the size sorting position Position of the string and the internal coding index of the value corresponding to the string in the value partition by using a one-dimensional array.

[0151] In the embodiment of the present application, the third mapping relationship is also established according to the partition type coding of the value partition corresponding to the string and the partition start coding of the value partition, and an example is shown in Figure 10 . Figure 10 A schematic diagram of the third mapping relationship provided by the embodiment of the present application includes the partition type coding of the value partition corresponding to the string and the partition start coding region_index of the value partition. Here, the partition type coding is stored in decimal numbers and can also be converted into binary numbers. Thus, the start coding of the partition can be determined according to the partition type coding of the value partition corresponding to the string.

[0152] In the embodiment of the present application, the fourth mapping relationship can also be established according to the size sorting position of the string, the partition type coding of the value partition corresponding to the string, and the internal coding of the value corresponding to the string in the value partition, and an example is shown in Figure 11 . Figure 11 A schematic diagram of the fourth mapping relationship provided by the embodiment of the present application includes the size sorting position Position of the string and an integer data. The integer data can be converted into 32-bit binary data. Thus, the low 3 bits of the 32-bit binary data can represent the partition type coding of the value partition corresponding to the string, and the high 29 bits can represent the internal coding of the value corresponding to the string in the value partition.

[0153] It can be seen that, in the embodiments of the present application, the index mapping table can be established based on the size sorting position of the string, the partition type code of the value partition corresponding to the value of the string, and the internal encoding of the value of the string in the value partition, so that when the value corresponding to the string is read according to the string, the storage position of the value corresponding to the string can be determined based on the corresponding index mapping table, and the value corresponding to the string can be read, thereby reducing the time complexity and improving the reading efficiency when the value corresponding to the string is read.

[0154] In some embodiments, the method further comprises:

[0155] obtaining a string of key-value pair data to be read;

[0156] determining the size sorting position of the string of key-value pair data to be read;

[0157] According to the size sorting position of the string of key-value pair data to be read, determining, in the index mapping table, the partition type code of the value corresponding to the string of key-value pair data to be read, and the internal encoding of the partition of the value corresponding to the string of key-value pair data to be read;

[0158] According to the partition type code and the internal encoding of the partition, determining the storage position of the value corresponding to the string of key-value pair data to be read in the value partition;

[0159] According to the storage position of the value corresponding to the string of key-value pair data to be read in the value partition, reading the value corresponding to the string of key-value pair data to be read.

[0160] In the embodiments of the present application, the size sorting position of the string of key-value pair data to be read represents the size sorting position of the string of key-value pair data to be read in all strings. Since the characters of the string are stored in size order in the target range of the corresponding bucket array, the size sorting position of the string can be determined by determining the storage position of each character in the string of key-value pair data to be read in the corresponding key-value bucket.

[0161] In the embodiments of the present application, the index mapping table is established based on the first mapping relationship and the second mapping relationship. Here, the first mapping relationship is a mapping relationship between the size ordering position of a string and the partition type code of the value partition corresponding to the value of the string, and the second mapping relationship is a mapping relationship between the size ordering position of a string and the internal code of the value partition in which the value corresponding to the string is located. Therefore, when the value corresponding to the string of the key-value pair data needs to be read, the size ordering position of the string of the key-value pair data to be read can be determined, and the partition type code of the value partition of the value corresponding to the string and the internal code of the value partition in which the value corresponding to the string is located can be determined in the index mapping table according to the size ordering position of the string of the key-value pair data to be read.

[0162] In the embodiments of the present application, the partition start code of the value partition of the value corresponding to the string to be read can also be determined based on the third mapping relationship between the partition type code of the value partition of the value corresponding to the string and the partition start code of the value partition according to the partition type code of the value partition of the value corresponding to the string of the key-value pair data to be read.

[0163] In the embodiments of the present application, according to the determined partition type code of the value partition, it can be determined in which type of partition the value corresponding to the string of the key-value pair data to be read is stored, and according to the determined internal code of the value partition of the corresponding type and the internal code of the value partition, the storage position of the value corresponding to the string of the key-value pair data to be read in the partition of the corresponding type can be determined. The determined storage position of the value corresponding to the string of the key-value pair data to be read in the partition of the corresponding type is denoted as pos, and pos = region_index + index.

[0164] In the embodiments of the present application, the fourth mapping relationship between the size ordering position of a string, the partition type code of the value partition of the value corresponding to the string and the internal code of the value partition in which the value corresponding to the string is located can also be established, and the corresponding integer data can be determined based on the fourth mapping relationship after the size ordering position of the string of the key-value pair data to be read is determined. The integer data is 32-bit binary data, the partition type code of the value partition of the value corresponding to the string of the key-value pair data to be read can be determined by obtaining the low 3 bits of the integer data, the internal code of the value partition of the value corresponding to the string of the key-value pair data to be read can be determined by obtaining the high 29 bits of the integer data, and thus the storage position of the value corresponding to the string of the key-value pair data to be read in the value partition can be determined.

[0165] It can be seen that, in the embodiment of the application, by determining the size order position of the string of the to-be-read key-value pair data, the partition type code of the value corresponding to the string of the to-be-read key-value pair data and the internal code of the partition can be determined based on the index mapping table, so that the storage position of the value corresponding to the string of the to-be-read key-value pair data in the value partition can be determined, that is, the value corresponding to the string can be read according to the determined storage position.

[0166] In some embodiments, the determining the size order position of the string of the to-be-read key-value pair data comprises:

[0167] determining the jth character of the string of the to-be-read key-value pair data, j being greater than or equal to 1;

[0168] starting from j being equal to 1, accumulating the number of characters smaller than the jth character in the key-value bucket corresponding to the jth character, and finding whether there is a character identical to the jth character, and if there is no character identical to the jth character, determining the size order position of the string in the key-value bucket according to the number of characters smaller than the jth character;

[0169] if there is a character identical to the jth character, increasing the value of j by 1, and repeating the steps of accumulating the number of characters smaller than the jth character in the key-value bucket corresponding to the jth character and finding whether there is a character identical to the jth character until there is no character identical to the jth character.

[0170] In the embodiment of the application, the size order position of the string of the to-be-read key-value pair data is denoted as p, and when the size order position of the string of the to-be-read key-value pair data is determined, the size order position of the string of the to-be-read key-value pair data is first initialized, and p is set to 0.

[0171] In the embodiment of the application, starting from j being equal to 1, each character in the string of the to-be-read key-value pair data is sequentially traversed. When j is equal to 1, the first character of the string of the to-be-read key-value pair data is first determined, the size order of the first character in the corresponding bucket array is determined, and p is set to the number of characters smaller than the first character in the ASCII value in the bucket array corresponding to the first character. Then it is judged whether there is a character with an ASCII value equal to that of the first character in the bucket array corresponding to the first character, and if there is no such character, p determined according to the first character of the string of the to-be-read key-value pair data is the size order position of the string of the to-be-read key-value pair data.

[0172] In the embodiment of the present application, if there is a character with the same ASCII value as the first character in the bucket array corresponding to the first character, it indicates that the size order of the string of the key-value pair data to be read cannot be determined according to the first character of the string of the key-value pair data to be read, and the size order of the string needs to be determined according to the size order of the next character. Therefore, in the case where the first character is not the last character of the string of the key-value pair data to be read, the value of j is increased by 1, and the steps of accumulating the number of characters smaller than the jth character and finding whether there is a character identical to the jth character in the bucket array corresponding to the jth character are performed.

[0173] In the embodiment of the present application, when the first character of the string is stored in the bucket, the ASCII value of the first character can be stored by subtracting 31 from the ASCII value of the first character. Therefore, the range of the ASCII value of the first character is between 1 and 95, and when the size value of the first character of the string of the key-value pair data to be read is determined, the read range is reduced to 1 / 95.

[0174] In the embodiment of the present application, when j is greater than 1, the jth character of the string of the key-value pair data is determined, the number of all characters in the bucket array corresponding to the ith character is obtained, and the target range of the bucket array corresponding to the ith character is determined. The target range of the bucket array corresponding to the jth character can be determined according to the number of characters greater than the j-1th character, the number of characters equal to the j-1th character, and the number of high-level characters of the j-1th character in the target range of the bucket array corresponding to the j-1th character. Here, the high-level character means that the j-1th character of all strings greater than the string to which the highest point high of the target range of the bucket array corresponding to the j-1th character belongs in the bucket array corresponding to the j-1th character.

[0175] In the embodiment of the present application, the sum of the number of characters greater than the j-1th character and the number of high-level characters of the j-1th character in the target range of the bucket array corresponding to the j-1th character is recorded as bigs, and the number of characters equal to the j-1th character in the target range of the bucket array corresponding to the j-1th character is recorded as preEquals. The initial value of bigs is 0, and the initial value of preEquals is 0.

[0176] In the embodiment of the present application, before determining the target range of the bucket array corresponding to the jth character, the bucket array corresponding to the (j-1)th character is traversed to determine bigs and preEquals. It should be noted that when the (j-1)th character has a high-level character, the stop symbol "-1" is not counted in the number of high-level characters of the (j-1)th character. Then the highest point high and the lowest point low of the target range are determined. Here, the highest point high is equal to the number of all characters in the bucket array corresponding to the jth character minus bigs; and the lowest point low is equal to the highest point high minus preEquals.

[0177] In the embodiment of the present application, in the target range of the bucket array corresponding to the jth character, it is determined whether there is a character with an ASCII value equal to that of the jth character. If not, p is accumulated according to the number of characters with an ASCII value less than that of the jth character in the target range of the bucket array corresponding to the jth character in the size order of the string of the key-value pair data, that is, p is equal to p plus the number of characters with an ASCII value less than that of the jth character in the target range of the bucket array corresponding to the jth character. And the accumulated p is determined as the size order position of the string of the key-value pair data to be read.

[0178] In the embodiment of the present application, if there is a character with an ASCII value equal to that of the jth character in the target range of the bucket array corresponding to the jth character, it means that the size order of the string of the key-value pair data to be read cannot be determined according to the jth character of the string of the key-value pair data, and the size order position of the string of the key-value pair data to be read needs to be determined according to the size order of the next character. Therefore, in the case where the jth character is not the last character of the string of the key-value pair data to be read, the value of j is increased by 1, and the steps of accumulating the number of characters less than the jth character and searching for a character equal to the jth character in the bucket array corresponding to the jth character are executed.

[0179] In the embodiment of the present application, after determining the target range of the bucket array corresponding to the jth character, the target range of the bucket array corresponding to the jth character can also be traversed to determine the number of characters with an ASCII value equal to that of the jth character in the target range of the bucket array corresponding to the jth character, which is recorded as equals. equals can be used to determine the target range of the bucket array corresponding to the (j+1)th character, that is, preEquals is equal to equals.

[0180] As can be seen, in this embodiment, by sequentially traversing each character of the string of the key-value pair data to be read, for the j-th character, the number of characters smaller than the j-th character is accumulated in the corresponding bucket array. When there is no character in the corresponding bucket array that is the same as the j-th character, the size sorting position of the string in the key-value bucket is determined. Since the size sorting of characters is determined based on the ASCII value of the characters, the size sorting of characters can be determined in O(1) complexity. Furthermore, the target range of the next character is determined based on the number of characters that are equal to the previous character in the target range. Therefore, the search complexity of each character is also close to O(1), which is significantly reduced compared with the HashMap data structure with a search complexity of O(logn) and the fastUtil data structure with a search complexity of O(n).

[0181] In some embodiments, the method further includes:

[0182] Get the string of the key-value pair data to be read;

[0183] Based on the index mapping table, determine the storage location of the value corresponding to the string of the key-value pair data to be read;

[0184] The value corresponding to the string of the key-value pair data to be read is read according to the storage location of the string corresponding to the string of the key-value pair data to be read.

[0185] In the embodiments of this application, Figure 12 The first schematic diagram of the data structure of the data processing method provided in the embodiments of this application is shown below. Figure 12 The data structure includes: a first key-value bucket module 1201, a first index mapping table 1202, and a first value partition 1203. The first key-value bucket module 1201 is used to store each character in the string. The key-value bucket corresponding to each character in the string can be represented by a bucket array. Each position in the bucket array represents a bucket, and the buckets are used to store the ASCII value of the corresponding character and the number of consecutive repetitions of the character.

[0186] In the embodiments of this application, see Figure 12 The first index mapping table 1202 includes: a first mapping relationship between the sorting position of the string and the partition type encoding of the value partition corresponding to the string; a second mapping relationship between the sorting position of the string and the internal encoding of the value partition corresponding to the string in the value partition; and a third mapping relationship between the partition type encoding of the value partition corresponding to the string and the partition start encoding of the value partition.

[0187] In the embodiments of this application, see Figure 12The first value partition 1203 includes eight areas, namely: a low-frequency access area for storing object types, a low-frequency access area for storing byte types, a low-frequency access area for storing floating-point types, a low-frequency access area for storing integer types, a high-frequency access area for storing object types, a high-frequency access area for storing byte types, a high-frequency access area for storing floating-point types, and a high-frequency access area for storing integer types.

[0188] In this embodiment, by obtaining the string of the key-value pair data to be read, the size sorting position of the string can be determined. Based on the first index mapping table 1202, the partition type code of the value partition corresponding to the string, the internal code (index) of the value within the value partition, and the region_index of the partition start code (region_index) of the value within the value partition can be determined. According to the determined partition type code, internal code (index), and region_index, the storage location of the value corresponding to the string of the key-value pair data to be read in the first value partition 1203 can be determined. Therefore, the value (Value) corresponding to the string of the key-value pair data to be read can be read according to the determined storage location.

[0189] In the embodiments of this application, Figure 13 The first schematic diagram of the data structure of the data processing method provided in the embodiments of this application is shown below. Figure 13 The data structure includes: a second key-value bucket module 1301, a second index mapping table 1302, and a second value partition 1303. The second key-value bucket module 1301 is used to store each character in the string. The key-value bucket corresponding to each character in the string can be represented by a bucket array. Each position in the bucket array represents a bucket, and the buckets are used to store the ASCII value of the corresponding character and the number of consecutive repetitions of the character.

[0190] In the embodiments of this application, see Figure 13 The second index mapping table 1302 includes a fourth mapping relationship established based on the sorting position of the string, the partition type encoding of the value partition corresponding to the string, and the internal encoding of the value in the value partition. Here, the partition type encoding of the value partition corresponding to the string and the internal encoding of the value in the value partition are represented by an integer, denoted as the value position encoding. The value partition 1303 includes three areas: a partition for storing byte types, a partition for storing floating-point types, and a partition for storing integer types.

[0191] In the embodiment of the present application, by obtaining the string of the key-value pair data to be read, the size order position position of the string of the key-value pair data to be read can be determined, so that based on the second index mapping table 1302, the value partition type code of the value partition corresponding to the value of the string of the key-value pair data to be read can be determined, by obtaining the high 29 bits of the integer data, the internal code of the value partition corresponding to the value of the string of the key-value pair data to be read can be determined, so that the storage position of the value corresponding to the string of the key-value pair data to be read in the value partition can be determined, and the value Value corresponding to the string of the key-value pair data to be read is read.

[0192] As can be seen, in the embodiment of the present application, by determining the string of the key-value pair data to be read, the storage position of the value corresponding to the string of the key-value pair data to be read is determined based on the index mapping table, so that the value corresponding to the string of the key-value pair data to be read can be read according to the determined storage position.

[0193] Figure 14 The flowchart of the first specific implementation of the data processing method provided in the embodiment of the present application is shown in FIG. 13A. Figure 14 As shown in FIG. 13A, the flowchart can include the following steps.

[0194] Step 1401: sequentially traversing each character of the string of the key-value pair data to be stored.

[0195] Step 1402: determining the i-th character of the key-value pair data to be stored.

[0196] Step 1403: determining whether i is equal to 1, if yes, executing step 1404, if no, executing step 1408.

[0197] Step 1404: determining whether there is a character equal to the first character in the bucket array corresponding to the first character, if yes, executing step 1405, if no, executing step 1406.

[0198] Step 1405: determining the target bucket in the bucket array corresponding to the first character, and adding 1 to the continuous repetition number of the character in the target bucket, and executing step 1416.

[0199] Step 1406: determining the number of characters less than the first character in the bucket array corresponding to the first character, and recording the number as position.

[0200] Step 1407: storing the first character into the corresponding bucket array according to position, and executing step 1416.

[0201] Step 1408: obtaining the number of all characters in the bucket array corresponding to the i-th character.

[0202] Step 1409: Determine the number of characters greater than the (i-1)th character in the target range of the bucket array corresponding to the (i-1)th character, and the number of characters one level higher than the (i-1)th character, denoted as bigs.

[0203] Step 1410: Determine the number of characters equal to the (i-1)th character in the target range of the bucket array corresponding to the (i-1)th character, denoted as preEquals.

[0204] Step 1411: According to the number of all characters in the bucket array corresponding to the ith character, bigs and preEquals, determine the target range of the bucket array corresponding to the ith character, including the highest point high and the lowest point low.

[0205] Step 1412: According to the size sorting of the ith character in the corresponding bucket array, determine the storage position k of the ith character in the corresponding key-value bucket.

[0206] Step 1413: Determine whether there is a character equal to the ith character in the target range of the bucket array corresponding to the ith character, if yes, execute step 1414, if no, execute step 1415.

[0207] Step 1414: In the bucket array corresponding to the ith character, determine the target bucket, and add 1 to the continuous repetition times of the character in the target bucket, execute step 1416.

[0208] Step 1415: According to the storage position k, store the ith character into the corresponding bucket array.

[0209] Step 1416: Determine whether the ith character is the last character of the to-be-stored key-value pair data, if yes, execute step 1417, if no, execute step 1401.

[0210] Step 1417: Determine whether there is a value corresponding to the string of the to-be-stored key-value pair data in the high-frequency access area of the value partition, if yes, execute step 1418, if no, execute step 1420.

[0211] Step 1418: According to the storage position of each character in the corresponding key-value bucket, and the storage position of the value corresponding to the string in the value partition, establish an index mapping table.

[0212] Step 1419: Determine the starting code of the value partition, and end the process.

[0213] Step 1420: According to the storage order of the value corresponding to the string of the to-be-stored key-value pair data, store the value corresponding to the string into the value partition.

[0214] Figure 15A flowchart of a second specific implementation of the data processing method provided by the embodiments of the present application is shown in Figure 15 The flowchart can include the following steps:

[0215] Step 1501: sequentially traversing each character of the string of the key-value pair data to be read.

[0216] Step 1502: determining the jth character of the key-value pair data to be read.

[0217] Step 1503: determining whether j is equal to 1, if yes, executing step 1504, if no, executing step 1511.

[0218] Step 1504: determining the number of characters smaller than the first character in the bucket array corresponding to the first character, denoted as p.

[0219] Step 1505: determining whether there is a character equal to the first character in the bucket array corresponding to the first character, if yes, executing step 1510, if no, executing step 1506.

[0220] Step 1506: determining the size sorting position of the string of the key-value pair data to be read according to p.

[0221] Step 1507: determining the partition type code, the internal code of the partition, and the starting code of the partition of the value partition corresponding to the value corresponding to the string of the key-value pair data to be read according to the size sorting position of the string of the key-value pair data to be read based on the index mapping table.

[0222] Step 1508: determining the storage position of the value corresponding to the string of the key-value pair data to be read in the value partition according to the determined partition type code, the internal code of the partition, and the starting code of the partition.

[0223] Step 1509: reading the value corresponding to the string of the key-value pair data to be read, and ending the flowchart.

[0224] Step 1510: determining whether the jth character is the last character of the key-value pair data to be read, if yes, executing step 1506, if no, executing step 1501.

[0225] Step 1511: obtaining the number of all characters in the bucket array corresponding to the jth character.

[0226] Step 1512: determining the number of characters greater than the j-1th character in the target range of the bucket array corresponding to the j-1th character, and the number of characters one level higher than the j-1th character, denoted as bigs.

[0227] Step 1513: Determine the number of characters equal to the (j-1)th character in the target range of the bucket array corresponding to the (j-1)th character, denoted as preEquals.

[0228] Step 1514: According to the number of all characters in the bucket array corresponding to the jth character, bigs, and preEquals, determine the target range of the bucket array corresponding to the jth character, including the highest point high and the lowest point low.

[0229] Step 1515: Determine the number of characters less than the jth character in the target range of the bucket array corresponding to the jth character, and accumulate it in p.

[0230] Step 1516: Determine whether there is a character equal to the jth character in the target range, if yes, execute step 1510, if no, execute step 1506.

[0231] In a specific example, there are four key-value pair data, which are: the first key-value pair data ("pbc_ldt", "-99"), the second key-value pair data ("pbc_ldar", "-88"), the third key-value pair data ("d_6m_id", "-77"), and the fourth key-value pair data ("br_als", "-88888"). The characters in the strings of the four key-value pair data are stored in the corresponding key buckets in turn, and the values corresponding to the strings of the four key-value pair data are stored in the value partition in turn.

[0232] The first stored is the first key-value pair data ("pbc_ldt", "-99"), and the ASCII values of the characters of the string "pbc_ldt" of the first key-value pair data are: {112, 98, 99, 95, 108, 100, 116} in turn, wherein the ASCII value of the first character minus 32 is 80. The six characters of the string of the first key-value pair data are stored in the first to sixth key buckets respectively, and a delimiter "-1" is stored in the seventh key bucket as the end symbol of the string of the first key-value pair data. At this time, only the string of the first key-value pair data is stored, and therefore, the size ordering position of the string of the first key-value pair data is 0.

[0233] The value "-99" corresponding to the string of the first key-value pair data is an integer type and has low access frequency, and is stored in the low-frequency access partition for storing integer type in the value partition. It can be determined that the data stored in the partition is {-99}, the partition type code is 011, the internal partition code is 0, and the partition start code is {0, 0, 0, 0, 0, 0, 0, 0}. The index mapping table {0} is established according to the size ordering position of the string of the first key-value pair data, the partition type code, and the internal partition code.

[0234] The second stored is the second key-value pair data ("pbc_lDar", "-88"), the ASCII values of each character of the string "pbc_lDar" of the second key-value pair data are: {112, 98, 99, 95, 108, 109, 97, 114} in turn, wherein the ASCII value of the first character is 80 after subtracting 32. According to the size ordering of the string of the second key-value pair data, the seven characters of the string of the second key-value pair data are respectively stored into the first to seventh key-value buckets, and a stopper "-1" is stored in the eighth key-value bucket. Since the first to fifth characters of the string of the second key-value pair data are the same as the first to fifth characters of the string of the first key-value pair data, and the sixth character of the string of the second key-value pair data is smaller than the sixth character of the string of the first key-value pair data, it can be determined that the string of the first key-value pair data is smaller than the string of the second key-value pair data, and the size ordering position of the string of the first key-value pair data is 0, and the size ordering position of the string of the second key-value pair data is 1.

[0235] The value "-88" corresponding to the string of the second key-value pair data is an integer type, and has a low access frequency, so it is stored in the low-frequency access partition for storing integer types in the value partition, and it can be determined that the data stored in the partition is {-99, -88}, the partition type code is 011, the partition internal code is 1, and the partition start code is {0, 0, 0, 0, 0, 0, 0, 0}. According to the mapping relationship of the size ordering position of the string of the second key-value pair data, the partition type code, and the partition internal code, it is stored in the index mapping table {0, 1}.

[0236] The third stored is the third key-value pair data ("d_6m_id", "-77"), the ASCII values of each character of the string "d_6m_id" of the third key-value pair data are: {100, 95, 54, 109, 95, 105, 100} in turn, wherein the ASCII value of the first character is 68 after subtracting 32. According to the size ordering of the string of the third key-value pair data, the seven characters of the string of the third key-value pair data are respectively stored into the first to seventh key-value buckets, and a stopper "-1" is stored in the eighth key-value bucket. Since the first character of the string of the third key-value pair data is smaller than the first character of the string of the first key-value pair data and the first character of the string of the second key-value pair data, it can be determined that the string of the third key-value pair data is smaller than the string of the first key-value pair data, and the string of the first key-value pair data is smaller than the string of the second key-value pair data, and the size ordering position of the string of the third key-value pair data is 0, the size ordering position of the string of the first key-value pair data is 1, and the size ordering position of the string of the second key-value pair data is 2.

[0237] The value corresponding to the string of the third key-value pair data "-77" is an integer type, and has a low access frequency, so it is stored in the low-frequency access partition for storing integer types in the value partition. It can be determined that the data stored in the partition is {-99, -88, -77}, the partition type code is 011, the internal code of the partition is 2, and the starting code of the partition is {0, 0, 0, 0, 0, 0, 0, 0}. According to the mapping relationship of the size sorting position of the string of the third key-value pair data, the partition type code, and the internal code of the partition, it is stored in the index mapping table {2, 0, 1}.

[0238] The fourth stored is the third key-value pair data ("br_als", "-88888"). The ASCII values of the characters of the string "br_als" of the third key-value pair data are {98, 114, 95, 97, 108, 115} in turn, and the ASCII value of the first character is 66 after subtracting 32. According to the size sorting of the fourth key-value pair data string, the six characters of the fourth key-value pair data string are stored in the first to sixth key-value buckets respectively, and a delimiter "-1" is stored in the seventh key-value bucket. Since the first character of the fourth key-value pair data string is smaller than the first character of the first key-value pair data string, the first character of the second key-value pair data string, and the first character of the third key-value pair data string, it can be determined that the fourth key-value pair data string is smaller than the third key-value pair data string, the third key-value pair data string is smaller than the first key-value pair data string, and the first key-value pair data string is smaller than the second key-value pair data string. The size sorting position of the fourth key-value pair data string is 0, the size sorting position of the third key-value pair data string is 1, the size sorting position of the first key-value pair data string is 2, and the size sorting position of the second key-value pair data string is 3.

[0239] The value corresponding to the string of the fourth key-value pair data "-88888" is an integer type, and has a low access frequency, so it is stored in the low-frequency access partition for storing integer types in the value partition. It can be determined that the data stored in the partition is {-99, -88, -77, -88888}, the partition type code is 011, the internal code of the partition is 3, and the starting code of the partition is {0, 0, 0, 0, 0, 0, 0, 0}. According to the mapping relationship of the size sorting position of the fourth key-value pair data string, the partition type code, and the internal code of the partition, it is stored in the index mapping table {3, 2, 0, 1}.

[0240] When the value corresponding to the string "d_6m_id" needs to be read, the size order position of the string "d_6m_id" in the other three strings is determined first, and it is found that the size order position of the string "d_6m_id" is 1. Then, according to the index mapping table, it is determined that the partition type code of the value corresponding to the string "d_6m_id" is 011, that is, the low-frequency access partition for storing integer type, and according to {3, 2, 0, 1}, it is determined that the internal partition code of the value corresponding to the string "d_6m_id" is 2. Then, according to the partition type code, the partition start code and the internal partition code, it is determined that the storage location of the value corresponding to the string "d_6m_id" is the third value in the low-frequency access partition for storing integer type, that is, "-77".

[0241] On the basis of the data processing method proposed in the foregoing embodiment, the embodiment of the present application further proposes a data processing device; Figure 16 An optional component structure diagram of the data processing device provided by the embodiment of the present application is shown in FIG. 16, which can include: Figure 16

[0242] The obtaining module 1601 is configured to obtain to-be-stored data, wherein the to-be-stored data includes at least two key-value pair data, and each key-value pair data includes a string and a value corresponding to the string.

[0243] The first determining module 1602 is configured to determine, according to the size order of the string of each key-value pair data in the at least two key-value pair data, the storage location of each character in the string in the corresponding key-value bucket, and store each character of the string into the corresponding key-value bucket according to the storage location of each character in the corresponding key-value bucket.

[0244] The second determining module 1603 is configured to determine, according to the storage order of the value corresponding to the string, the storage location of the value in the value partition, and store the value corresponding to the string into the value partition according to the storage location of the value in the value partition.

[0245] The processing module 1604 is configured to establish an index mapping table according to the storage location of each character in the string in the corresponding key-value bucket and the storage location of the value corresponding to the string in the value partition, wherein the index mapping table is used to represent the index mapping relationship between the storage location of the string and the value corresponding to the string.

[0246] In some embodiments, the first determining module 1602 is configured to determine, according to the size order of the string of each key-value pair data, the storage location of each character in the string in the corresponding key-value bucket, including:​

[0247] determining an i-th character of the string of the key-value pair data, i being greater than or equal to 1;

[0248] starting from i being equal to 1, searching, in the key-value bucket corresponding to the i-th character, whether there is a character identical to the i-th character, and if there is no character identical to the i-th character, determining a storage position of the i-th character in the corresponding key-value bucket according to a size order of the corresponding key-value bucket based on the i-th character;

[0249] if there is a character identical to the i-th character, increasing the value of i by 1, and repeating the step of searching whether there is a character identical to the i-th character.

[0250] In some embodiments, the key-value bucket corresponding to each character includes at least one bucket, and each bucket in the at least one bucket is used to store an ASCII value of the character and a continuous repetition number of the character.

[0251] The first determining module 1602 is further used to:

[0252] determining a target bucket in the key-value bucket corresponding to the i-th character, the target bucket indicating a bucket storing an ASCII value of a character identical to the i-th character;

[0253] increasing the continuous repetition number of the character in the target bucket by 1.

[0254] In some embodiments, the second determining module 1603 is further used to, before determining the storage position of the value corresponding to the string in the value partition according to the storage order of the value corresponding to the string:

[0255] determining the value partition storing the value corresponding to the string according to a type of the value corresponding to the string and / or an access frequency of the value corresponding to the string.

[0256] In some embodiments, the processing module 1604 is used to establish the index mapping table according to the storage position of each character in the string in the corresponding key-value bucket and the storage position of the value corresponding to the string in the value partition, including:

[0257] determining a size order position of the string according to the storage position of each character in the string in the corresponding key-value bucket;

[0258] determining a partition type code of the value partition storing the value corresponding to the string, and determining an internal code of the value corresponding to the string in the value partition according to the storage position of the value corresponding to the string in the value partition;

[0259] determine a first mapping relationship between the size ordering position of the string and the partition type code of the value partition corresponding to the string, and a second mapping relationship between the size ordering position of the string and the internal code of the value corresponding to the string in the value partition according to the size ordering position of the string, the partition type code of the value partition corresponding to the string, and the internal code of the value corresponding to the string in the value partition;

[0260] establish the index mapping table according to the first mapping relationship and the second mapping relationship.

[0261] In some embodiments, the apparatus further comprises a first reading module, configured to:

[0262] obtain a string of key-value pair data to be read;

[0263] determine a size ordering position of the string of key-value pair data to be read;

[0264] determine, in the index mapping table, a partition type code of a value corresponding to the string of key-value pair data to be read and an internal code of a partition of the value corresponding to the string of key-value pair data to be read according to the size ordering position of the string of key-value pair data to be read;

[0265] determine a storage position of the value corresponding to the string of key-value pair data to be read in the value partition according to the partition type code and the internal code of the partition;

[0266] read the value corresponding to the string of key-value pair data to be read according to the storage position of the value corresponding to the string of key-value pair data to be read in the value partition.

[0267] In some embodiments, the first reading module is configured to determine the size ordering position of the string of key-value pair data to be read, comprising:

[0268] determine a jth character of the string of key-value pair data to be read, j being greater than or equal to 1;

[0269] starting from j being equal to 1, accumulate the number of characters smaller than the jth character in the key-value bucket corresponding to the jth character, find whether there is a character identical to the jth character, and if there is no character identical to the jth character, determine the size ordering position of the string in the key-value bucket according to the number of characters smaller than the jth character;

[0270] If there is a character same as the jth character, let the value of j be added by 1, and repeat the steps of accumulating the number of characters less than the jth character, and searching whether there is a character same as the jth character in the key value bucket corresponding to the jth character, until there is no character same as the jth character.

[0271] In some embodiments, the apparatus further comprises a second reading module configured to:

[0272] obtain a string of key-value pair data to be read;

[0273] determine, according to the index mapping table, a storage location of a value corresponding to the string of key-value pair data to be read;

[0274] read the value corresponding to the string of key-value pair data to be read according to the storage location of the value corresponding to the string of key-value pair data to be read.

[0275] In practical applications, the obtaining module 1601, the first determining module 1602, the second determining module 1603, the processing module 1604, the first reading module and the second reading module can be implemented by a processor of an electronic device, and the processor can be at least one of an ASIC, a DSP, a DSPD, a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor. It can be understood that the electronic device implementing the above processor functions can also be other electronic devices, and the embodiments of the present application are not limited thereto.

[0276] It should be noted that the above description of the device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0277] It should be noted that in the embodiments of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a terminal, a server, etc.) to execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various storage media that can store program codes. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0278] Correspondingly, an embodiment of the present application further provides a computer program product, which comprises computer executable instructions for implementing any of the data processing methods provided by the embodiments of the present application.

[0279] Correspondingly, an embodiment of the present application further provides a computer storage medium, which stores computer executable instructions for implementing any of the data processing methods provided by the embodiments.

[0280] The embodiments of the present application further provide an electronic device, Figure 17 An optional component structure diagram of the electronic device provided by the embodiments of the present application is shown in FIG. 17. Figure 17 As shown in FIG. 17, the electronic device 1700 comprises:

[0281] a memory 1701 configured to store executable instructions;

[0282] a processor 1702 configured to execute the executable instructions stored in the memory 1701, so as to implement any of the data processing methods.

[0283] The processor 1702 can be at least one of an ASIC, a DSP, a DSPD, a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor.

[0284] The computer readable storage medium / memory can be a ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc. The computer readable storage medium / memory can also be various terminals including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0285] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0286] It should be understood that the "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiments are included in at least one embodiment of the present application. Therefore, "in some embodiments" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that the size of the sequence number of each process in various embodiments of the present application does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The sequence number of the above embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments.

[0287] It should be noted that in this paper, the term "including", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the sentence "including a…" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0288] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0289] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units; they can be located in one place or distributed on multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0290] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be separately taken as one unit, or two or more units can be integrated in one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software function unit.

[0291] Alternatively, the integrated unit of the present application, if realized in the form of software function module and sold or used as an independent product, can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing the equipment automatic test line to execute the methods described in each embodiment of the present application. The foregoing storage medium includes mobile storage equipment, ROM, magnetic disc or optical disc and various storage program codes.

[0292] The methods disclosed in the several method embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0293] The features disclosed in the several method or device embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0294] The above is only the preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain data to be stored, wherein the data to be stored includes at least two key-value pairs, and each key-value pair includes: a string and the value corresponding to the string; Determine any one of the at least two key-value pairs, sort the key-value pairs according to their string size, determine the storage position of each character in the string in the corresponding key-value bucket, and store each character of the string into the corresponding key-value bucket according to the storage position of each character in the corresponding key-value bucket. Based on the storage order of the values ​​corresponding to the string, determine the storage location of the value corresponding to the string in the value partition; based on the storage location of the value corresponding to the string in the value partition, store the value corresponding to the string in the value partition; An index mapping table is established based on the storage location of each character in the string in the corresponding key-value bucket and the storage location of the value corresponding to the string in the value partition. The index mapping table is used to represent the index mapping relationship between the storage locations of the string and the value corresponding to the string. The step of establishing the index mapping table based on the storage location of each character in the string in its corresponding key-value bucket and the storage location of the value corresponding to the string in the value partition includes: The size sorting position of the string is determined based on the storage position of each character in the string in the corresponding key-value bucket; Determine the partition type encoding of the value partition where the value corresponding to the string is stored; determine the internal encoding of the value corresponding to the string in the value partition based on the storage location of the value corresponding to the string in the value partition; Based on the size sorting position of the string, the partition type encoding of the value partition of the value corresponding to the string, and the internal encoding of the value corresponding to the string in the value partition, a first mapping relationship between the size sorting position of the string and the partition type encoding of the value partition of the value corresponding to the string, and a second mapping relationship between the size sorting position of the string and the internal encoding of the value corresponding to the string in the value partition are determined; Based on the first mapping relationship and the second mapping relationship, the index mapping table is established.

2. The method according to claim 1, characterized in that, The step of sorting the key-value pair data by string size and determining the storage location of each character in the string within the corresponding key-value bucket includes: Determine the i-th character of the string in the key-value pair data, where i is greater than or equal to 1; Starting from i equal to 1, in the key value bucket corresponding to the i-th character, check if there is a character that is the same as the i-th character. If there is no character that is the same as the i-th character, determine the storage location of the i-th character in the corresponding key value bucket according to the size of the i-th character in the corresponding key value bucket. If a character identical to the i-th character exists, increment the value of i by 1 and repeat the step of searching for a character identical to the i-th character.

3. The method according to claim 2, characterized in that, The key-value bucket corresponding to each character includes at least one bucket, and each of the at least one bucket is used to store the ASCII value of the character and the number of consecutive repetitions of the character; After the statement that if a character identical to the i-th character exists, the method further includes: In the key value bucket corresponding to the i-th character, a target bucket is determined, where the target bucket represents the bucket that stores the ASCII value of the character that is the same as the i-th character; Increment the number of consecutive repetitions of the characters in the target bucket by 1.

4. The method according to claim 1, characterized in that, Before determining the storage location of the value corresponding to the string in the value partition based on the storage order of the values ​​corresponding to the string, the method further includes: Based on the type of the value corresponding to the string, and / or the access frequency of the value corresponding to the string, determine the value partition for storing the value corresponding to the string.

5. The method according to claim 1, characterized in that, The method further includes: Get the string of the key-value pair data to be read; Determine the size and sorting position of the string in the key-value pair data to be read; Based on the size and sorting position of the string of the key-value pair data to be read, the partition type encoding of the value corresponding to the string of the key-value pair data to be read, and the internal encoding of the partition of the value corresponding to the string of the key-value pair data to be read are determined in the index mapping table; Based on the partition type encoding and the internal encoding of the partition, determine the storage location of the value corresponding to the string of the key-value pair data to be read in the value partition; Based on the storage location of the string corresponding to the key-value pair data to be read in the value partition, read the string corresponding to the key-value pair data to be read.

6. The method according to claim 5, characterized in that, Determining the size and sorting position of the string in the key-value pair data to be read includes: Determine the j-th character of the string containing the key-value pair data to be read, where j is greater than or equal to 1; Starting from j=1, in the key value bucket corresponding to the j-th character, accumulate the number of characters smaller than the j-th character, and check if there is a character that is the same as the j-th character. If there is no character that is the same as the j-th character, then determine the size sorting position of the string in the key value bucket according to the number of characters smaller than the j-th character. If a character identical to the j-th character exists, increment the value of j by 1, and repeat the steps of accumulating the number of characters smaller than the j-th character in the key-value bucket corresponding to the j-th character, and searching for a character identical to the j-th character, until no character identical to the j-th character exists.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Get the string of the key-value pair data to be read; Based on the index mapping table, determine the storage location of the value corresponding to the string of the key-value pair data to be read; The value corresponding to the string of the key-value pair data to be read is read according to the storage location of the string corresponding to the string of the key-value pair data to be read.

8. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire data to be stored, the data to be stored including at least two key-value pairs, each key-value pair including: a string and the value corresponding to the string; The first determining module is used to determine any one of the at least two key-value pairs, sort the key-value pairs according to their string size, determine the storage position of each character in the string in the corresponding key-value bucket, and store each character of the string into the corresponding key-value bucket according to the storage position of each character in the corresponding key-value bucket. The second determining module is used to determine the storage location of the value corresponding to the string in the value partition according to the storage order of the values ​​corresponding to the string; and to store the value corresponding to the string in the value partition according to the storage location of the value corresponding to the string in the value partition. The processing module is used to establish an index mapping table based on the storage location of each character in the string in the corresponding key-value bucket and the storage location of the value corresponding to the string in the value partition. The index mapping table is used to represent the index mapping relationship between the storage locations of the string and the value corresponding to the string. The processing module is configured to establish the index mapping table based on the storage location of each character in the string in its corresponding key-value bucket and the storage location of the value corresponding to the string in the value partition, including: The size sorting position of the string is determined based on the storage position of each character in the string in the corresponding key-value bucket; Determine the partition type encoding of the value partition where the value corresponding to the string is stored; determine the internal encoding of the value corresponding to the string in the value partition based on the storage location of the value corresponding to the string in the value partition; Based on the size sorting position of the string, the partition type encoding of the value partition of the value corresponding to the string, and the internal encoding of the value corresponding to the string in the value partition, a first mapping relationship between the size sorting position of the string and the partition type encoding of the value partition of the value corresponding to the string, and a second mapping relationship between the size sorting position of the string and the internal encoding of the value corresponding to the string in the value partition are determined; Based on the first mapping relationship and the second mapping relationship, the index mapping table is established.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data processing method according to any one of claims 1-7.

Images