Data storage method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data storage method and device, equipment and a storage medium, relates to the technical field of databases, and is suitable for storing key value pair data arranged in a lexicographical order by keys to a set storage area. The set storage area comprises a data block and a prefix index block; the prefix index block comprises a prefix key index area and a specific area; the prefix key index area stores a prefix key of any key-value pair data, the specific area stores a specific value of any key-value pair data, and the method comprises the following steps of: sequencing a plurality of key-value pair data by taking keys of the plurality of key-value pair data to be stored as a sequence, and determining the prefix key and the specific value corresponding to each key-value pair data; prefix keys of the multiple pieces of key value pair data are stored in a prefix key index area of a prefix index block in sequence, and specific values of the multiple pieces of key value pair data are stored in a specific area of the prefix index block in sequence; suffix keys and values of the plurality of key-value pair data are stored in a data block.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data storage method, device, equipment and storage medium. Background Art

[0002] In the field of storage of key-value data pairs, ordered data structures are widely used. The so-called key-value is a key plus value tuple. For example, in a table storing file information, there may be such a key-value pair {number: file name}. The number is the key used for indexing and must be globally unique, that is, there cannot be repeated file numbers, and its corresponding value is the file name.

[0003] Currently, indexing file names by number is usually divided into two processes: data indexing and data reading. First, the data storage location is determined by data indexing, and then the data value is determined based on the data storage location. However, in data indexing and data reading, data indexing often takes a long time, while data reading takes a short time.

[0004] In order to shorten the time, the data reading process is shortened, but the main problem of the long time is the data index, so shortening the data reading does not effectively shorten the time; the learning index algorithm can quickly read the data, but the learning index algorithm cannot quickly index string type keys. Summary of the invention

[0005] The embodiments of the present application provide a data storage method, apparatus, device and storage medium, which can store and query data more efficiently.

[0006] In a first aspect, an embodiment of the present application provides a data storage method, which is suitable for storing key-value pair data arranged in lexicographic order by keys in a set storage area; the set storage area includes: a data block and a prefix index block; the prefix index block includes: a prefix key index area and a special area; the prefix key index area stores a prefix key of any key-value pair data, and the special area stores a special value of any key-value pair data; the method includes:

[0007] After sorting the multiple key-value pair data to be stored in order of their keys, determine the prefix key and unique value corresponding to each key-value pair data; wherein the prefix key is the longest common part of the key-value pair data to be stored and the key of the previous key-value pair data starting from the first position; the unique value is the first different value in the key of the key-value pair data to be stored and the key of the previous key-value pair data;

[0008] The prefix keys of the plurality of key-value pairs are sequentially stored in the prefix key index area of ​​the prefix index block, and the unique values ​​of the plurality of key-value pairs are sequentially stored in the unique area of ​​the prefix index block;

[0009] The suffix keys and values ​​of the plurality of key-value pairs are stored in the data block; wherein the suffix key is the part of each key except the prefix key.

[0010] In the embodiment of the present application, the data structure for storing data is redesigned, the existing index block is changed into a prefix index block, and an index compression algorithm and a query algorithm are proposed accordingly, which can effectively reduce the size of the storage space required for the index block and the data block, thereby speeding up the speed of reading data into the memory. At the same time, the relative position obtained by the query algorithm is used as input, combined with the learning index, to overcome the limitation that the learning index is difficult to apply to string type data, and to make full use of its high indexing speed and low storage overhead, further improving the efficiency of key-value pair query.

[0011] Optionally, the prefix index block further includes a length area; and the sequentially storing the prefix keys of the plurality of key-value pairs of data in the prefix key index area of ​​the prefix index block includes:

[0012] The longest prefix key among the prefix keys of the plurality of key-value pair data is used as a compressed prefix key; the length values ​​corresponding to the common parts of the prefix keys of each key-value pair data in the compressed prefix keys are sequentially determined to obtain a length sequence;

[0013] The compressed prefix key is stored in the prefix key index area of ​​the prefix index block, and the length sequence is stored in the length area of ​​the prefix index block.

[0014] In the embodiment of the present application, by further compressing the prefix key, the storage space of the data can be further reduced; the length corresponding to the compressed prefix key and the prefix key of each key-value pair data in the compressed prefix key is stored as a length sequence, and the prefix key can be restored according to the compressed prefix key and the length sequence. This saves space and does not affect the integrity of the prefix key.

[0015] Optionally, it also includes:

[0016] Based on the relative positions and storage addresses of the multiple key-value pairs in the data block, a linear regression model for each interval is determined; wherein the linear regression model for each interval takes the relative positions of the key-value pairs in the interval as input and the storage addresses of the key-value pairs as output; the relative positions represent the serial numbers of the multiple key-value pairs in the data block in ascending order of the storage addresses; and the linear regression model for each interval is stored in a model index block.

[0017] In the embodiment of the present application, the linear regression model of each interval is determined according to the relative position. Multiple linear regression models are used to more accurately determine the storage location of the key-value pair.

[0018] Optionally, determining the linear regression model of each interval based on the relative positions and storage addresses of the plurality of key-value pairs in the data block includes:

[0019] A linear regression model is constructed with the relative position of the key-value pair data in the data block as the independent variable and the storage address of the key-value pair data in the data block as the dependent variable;

[0020] If the fitting error of the accumulated j-th key-value pair data in the linear regression model corresponding to the i-th key-value pair data does not exceed the set threshold, then the j-th key-value pair data is added to the linear regression model corresponding to the i-th key-value pair data, and the j-th key-value pair data is located one after the i-th key-value pair data;

[0021] If it exceeds the set threshold, the j-th key-value pair data is used as the starting point of the new interval, and a linear regression model corresponding to the j-th key-value pair data is constructed until all key-value pair data determine the linear regression model of the corresponding interval.

[0022] In an embodiment of the present application, the relative position of the key-value pair data in the data block is fitted with the relative position of the previous key-value pair data in the data block to determine the linear regression model in which the current key-value pair data is located. By setting a threshold so that the current key-value pair data is in a more appropriate linear regression model, the accuracy of the query on the current key-value pair data is increased.

[0023] In a second aspect, an embodiment of the present application provides a data query method, which is suitable for storing a key-value pair data storage format in which keys are arranged in lexicographic order in a set storage area, wherein the set storage area includes: a data block and a prefix index block; the prefix index block includes: a prefix key index area and a special area; the method includes:

[0024] Obtaining a query request, wherein the query request includes a key of a target value;

[0025] Based on the prefix keys stored in the prefix key index area of ​​the prefix index block in lexicographic order and the unique values ​​stored in the unique area of ​​the prefix index block, determine the prefix key that matches the key of the target value; wherein the prefix key is the longest common part of the key of any key-value pair data and the key of the previous key-value pair data; the unique value is the first different value between the key of any key-value pair data and the key of the previous key-value pair data;

[0026] Obtain the matching suffix key from the data block indicated by the matching prefix key; wherein the suffix key is the portion of each key excluding the prefix key;

[0027] After the matching prefix key is combined with the suffix key of the matching prefix key, the value of the target key is obtained from the data block.

[0028] In the embodiment of the present application, the data structure for storing data is redesigned, the existing index block is changed into a prefix index block, and an index compression algorithm and a query algorithm are proposed accordingly, which can effectively reduce the size of the storage space required for the index block and the data block, thereby speeding up the speed of reading data into the memory. At the same time, the relative position obtained by the query algorithm is used as input, combined with the learning index, to overcome the limitation that the learning index is difficult to apply to string type data, and to make full use of its high indexing speed and low storage overhead, further improving the efficiency of key-value pair query.

[0029] Optionally, determining a prefix key that matches a key of the target value based on prefix keys stored in lexicographic order in a prefix key index area of ​​a prefix index block and unique values ​​stored in a unique area of ​​the prefix index block includes:

[0030] For each prefix key sequentially stored in the prefix key index area based on the prefix index block, when the key of the target value matches the mth prefix key and the corresponding unique value, determining whether the key of the target value matches the m+1th prefix key and the corresponding unique value;

[0031] If there is no match, the prefix key that matches the key of the target value is determined to be the mth prefix key.

[0032] In the embodiment of the present application, by further compressing the prefix key, the storage space of the data can be further reduced; the length corresponding to the compressed prefix key and the prefix key of each key-value pair data in the compressed prefix key is stored as a length sequence, and the prefix key can be restored according to the compressed prefix key and the length sequence. This saves space and does not affect the integrity of the prefix key.

[0033] Optionally, each prefix key stored in the prefix key index area in lexicographic order is a compressed prefix key; the prefix key index area further includes a length sequence located in the length area; the compressed prefix key is the longest prefix key among the prefix keys, and the length sequence is the length value corresponding to the common part of each prefix key in the compressed prefix key;

[0034] Before each prefix key sequentially stored in the prefix key index area based on the prefix index block, it also includes:

[0035] For the nth prefix key, based on the length value k of the nth digit in the length sequence, the first k digits of the compressed prefix key are used as the nth prefix key.

[0036] In an embodiment of the present application, by storing the length sequence and the compressed prefix key together, the prefix key can be restored by both the length sequence and the compressed prefix key. All prefix keys can be obtained without storing all prefix keys, thus saving storage space.

[0037] Optionally, the model index block stores a linear regression model for each interval, and the linear regression model for any interval takes the relative position of the key-value pair data in the interval as input and the storage address of the key-value pair as output;

[0038] Based on the matched prefix key, obtaining a suffix key of the matched prefix key from a corresponding data block includes:

[0039] Determine the interval corresponding to the matching prefix key;

[0040] According to the linear regression model of the corresponding interval, based on the relative position of the matched prefix key in each prefix key, a target storage address of the key-value pair of the matched prefix key is obtained;

[0041] According to the target storage address, a suffix key of the matched prefix key is obtained from a corresponding data block.

[0042] In the embodiment of the present application, the linear regression model of each interval is determined according to the relative position. Multiple linear regression models are used to more accurately determine the storage location of the key-value pair.

[0043] In a third aspect, an embodiment of the present application provides a data storage device, which is suitable for storing a key-value pair in a key-ordered manner; the device comprises:

[0044] A determination module, used to sort the multiple key-value pair data to be stored in the order of their keys, and then determine the prefix key and unique value corresponding to each key-value pair data; wherein the prefix key is the longest common part of the key-value pair data to be stored and the key of the previous key-value pair data starting from the first position; the unique value is the first different value in the key of the key-value pair data to be stored and the key of the previous key-value pair data;

[0045] A storage module, used for sequentially storing the prefix keys of the plurality of key-value pair data into the prefix key index area of ​​the prefix index block, and sequentially storing the unique values ​​of the plurality of key-value pair data into the unique area of ​​the prefix index block;

[0046] The storage module is further used to store the suffix keys and values ​​of the multiple key-value pair data in the data block; wherein the suffix key is the part of each key except the prefix key.

[0047] Optionally, the storage module is specifically used for:

[0048] The longest prefix key among the prefix keys of the plurality of key-value pair data is used as a compressed prefix key; the length values ​​corresponding to the common parts of the prefix keys of each key-value pair data in the compressed prefix keys are sequentially determined to obtain a length sequence;

[0049] The compressed prefix key is stored in the prefix key index area of ​​the prefix index block, and the length sequence is stored in the length area of ​​the prefix index block.

[0050] Optionally, the determining module is specifically used to:

[0051] Based on the relative positions and storage addresses of the multiple key-value pairs in the data block, a linear regression model for each interval is determined; wherein the linear regression model for each interval takes the relative positions of the key-value pairs in the interval as input and the storage addresses of the key-value pairs as output; the relative positions represent the serial numbers of the multiple key-value pairs in the data block in ascending order of the storage addresses;

[0052] Store the linear regression model for each interval into the model index block.

[0053] Optionally, the determining module is further used for:

[0054] A linear regression model is constructed with the relative position of the key-value pair data in the data block as the independent variable and the storage address of the key-value pair data in the data block as the dependent variable;

[0055] If the fitting error of the accumulated j-th key-value pair data in the linear regression model corresponding to the i-th key-value pair data does not exceed the set threshold, then the j-th key-value pair data is added to the linear regression model corresponding to the i-th key-value pair data, and the j-th key-value pair data is located one after the i-th key-value pair data;

[0056] If it exceeds the set threshold, the j-th key-value pair data is used as the starting point of the new interval, and a linear regression model corresponding to the j-th key-value pair data is constructed until all key-value pair data determine the linear regression model of the corresponding interval.

[0057] Optionally, the prefix index block further includes a tail area, and the tail area is used to store meta information of the prefix index block.

[0058] In the embodiment of the present application, the data structure for storing data is redesigned, the existing index block is changed into a prefix index block, and an index compression algorithm and a query algorithm are proposed accordingly, which can effectively reduce the size of the storage space required for the index block and the data block, thereby speeding up the speed of reading data into the memory. At the same time, the relative position obtained by the query algorithm is used as input, combined with the learning index, to overcome the limitation that the learning index is difficult to apply to string type data, and to make full use of its high indexing speed and low storage overhead, further improving the efficiency of key-value pair query.

[0059] In a fourth aspect, an embodiment of the present application provides a data query device, which is suitable for storing a key-value pair data storage format in which keys are arranged in lexicographic order in a set storage area, wherein the set storage area includes: a data block and a prefix index block; the prefix index block includes: a prefix key index area and a special area; including:

[0060] An acquisition module, used for acquiring a query request, wherein the query request includes a key of a target value;

[0061] A determination module, configured to determine a prefix key that matches the key of the target value based on each prefix key stored in the prefix key index area of ​​the prefix index block in lexicographic order and each unique value stored in the unique area of ​​the prefix index block; wherein the prefix key is the longest common part of the key of any key-value pair data and the key of the previous key-value pair data; and the unique value is the first different value between the key of any key-value pair data and the key of the previous key-value pair data;

[0062] A matching module, which obtains the matched suffix key from the data block indicated by the matched prefix key; wherein the suffix key is the part of each key except the prefix key;

[0063] The matching module is further configured to obtain the value of the target key from the data block after the matched prefix key is combined with the suffix key of the matched prefix key.

[0064] Optionally, the determining module is further used for:

[0065] The method of determining a prefix key matching a key of the target value based on the prefix keys stored in the prefix key index area of ​​the prefix index block in lexicographic order and the unique values ​​stored in the unique area of ​​the prefix index block comprises:

[0066] For each prefix key sequentially stored in the prefix key index area based on the prefix index block, when the key of the target value matches the mth prefix key and the corresponding unique value, determining whether the key of the target value matches the m+1th prefix key and the corresponding unique value;

[0067] If there is no match, determining the prefix key that matches the key of the target value is the mth prefix key. Optionally, the matching module is specifically used to:

[0068] Determine the interval corresponding to the matching prefix key;

[0069] According to the linear regression model of the corresponding interval, based on the relative position of the matched prefix key in each prefix key, a target storage address of the key-value pair of the matched prefix key is obtained;

[0070] According to the target storage address, a suffix key of the matched prefix key is obtained from a corresponding data block.

[0071] In the embodiment of the present application, the data structure for storing data is redesigned, the existing index block is changed into a prefix index block, and an index compression algorithm and a query algorithm are proposed accordingly, which can effectively reduce the size of the storage space required for the index block and the data block, thereby speeding up the speed of reading data into the memory. At the same time, the relative position obtained by the query algorithm is used as input, combined with the learning index, to overcome the limitation that the learning index is difficult to apply to string type data, and to make full use of its high indexing speed and low storage overhead, further improving the efficiency of key-value pair query.

[0072] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned methods when executing the program.

[0073] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device, wherein when the program runs on the computer device, the computer device executes the steps of any of the above-described methods.

[0074] In a seventh aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the methods of the embodiments of the first or second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1a A schematic diagram of a process of data reading and data indexing;

[0077] Figure 1b It is a schematic diagram of the structure of a binary search;

[0078] Figure 2 A schematic diagram of the structure of an SStable provided in an embodiment of the present application;

[0079] Figure 3 A system architecture diagram provided for the implementation of this application;

[0080] Figure 4 A schematic diagram of a data storage process provided in an embodiment of the present application;

[0081] Figure 5A schematic diagram of a process for obtaining a compressed prefix key and a length sequence in a compressed prefix index area provided in an embodiment of the present application;

[0082] Figure 6 A flowchart of a method for obtaining a compressed prefix key provided in an embodiment of the present application;

[0083] Figure 7 A flowchart of a method for restoring a compressed prefix key provided in an embodiment of the present application;

[0084] Figure 8 A structural diagram of a matching method for a key to be queried provided in an embodiment of the present application;

[0085] Fig. 9 A schematic diagram of a flow chart of a linear regression model training method provided in an embodiment of the present application;

[0086] Fig.10 A schematic diagram of the structure of a data storage device provided in an embodiment of the present application;

[0087] Fig.11 A schematic diagram of the structure of a data query device provided in an embodiment of the present application;

[0088] Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0090] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0091] Sorted String Table (SSTable for short) is widely used in the field of key-value databases. The so-called key value is a key plus value tuple. For example, in a table storing file information, there may be a key-value pair {number: file name}. The number is the key used for indexing and must be globally unique, that is, there cannot be repeated file numbers, and its corresponding value is the file name. The salient feature of SSTable is that the key-value pairs in its internal data blocks are stored in order. Specifically, the SSTable file contains multiple data blocks, as shown in Table 1. The multiple key-value records in each data block are stored in the dictionary order of the key. The purpose of this design is to improve the efficiency of data search and retrieval.

[0092] Index Block key address aabaabaabb 00 abaaacedbb 17 abaaccafbh 37 abadegafbh 54

[0093] Table 1

[0094] Lexicographical order, a way of sorting strings, is similar to the order of words in a dictionary. That is, a < b < c < … < x < y < z. When comparing the lexicographical order of two strings, compare the characters one by one starting from the first character: If the characters at a certain position are different, the order of the characters in the alphabet determines the order of the strings. That is, the string with the smaller character comes first. For example, for the two keys "aa" and "ab", their first characters are both a, and then compare the sizes of the second characters. Since a < b, the lexicographical order of the first key is smaller, that is, "aa" < "ab". If all characters are the same, the shorter string comes first. For example, for the two keys "aa" and "aaa", their first two characters are the same, but the length of the first key "aa" is less than the length of the second key "aaa", so the lexicographical order of the first key is smaller, that is, "aa" < "aaa".

[0095] As shown in Table 1, there are 4 key-value pairs. Table 2 shows the organization form of a data block in the SSTable. This data block stores the above 4 key-value pairs, key_0 = aabaabaabb, key_1 = abaaacedbb, key_2 = abaaccafbh, key_3 = abadegafbh. According to the definition of lexicographical order, it is lexicographically increasing from key_0 to key_3. Because the second character of key_0 is less than the second character of key_1, the fifth character of key_1 is less than the fifth character of key_2, and the fourth character of key_2 is less than the fourth character of key_3.

[0096] Data Block address 00 Key 0 length 10 01 Key 0 aabaabaabb 11 Value 0 Length 5 12 Value 0 Alice 17 Key 1 Length 10 18 Key 1 abaaacedbb 28 Value 1 Length 8 29 Value 1 Benjamin 37 Key 2 Length 10 38 Key 2 abaaccafbh 48 Value 2 Length 5 49 Value 2 Cindy 54 Key 3 Length 10 55 Key 3 abadegafbh 65 Value 3 Length 5 66 Value 3 David

[0097] Table 2

[0098] The structure of the SSTable mainly includes a tail block, an index block, a filter block, and a data block, as shown in Table 3. Among them, the data block is used to store the key-value pair content, the filter block is used to quickly find whether a certain key exists in the SSTable, the index block records the meta-information related to the data block, including the position where a certain key exists in the data block, and the tail block is used to store the meta-information of the SSTable, including the position information of the index block, the filter block, and the data block in the file. The tail block is located at the end of the SSTable file.

[0099] Data Block 1 Data Block 2 …… Data Block N Filter Block Index Block Trailing Block

[0100] Table 3

[0101] The existing data reading and indexing have the following problems:

[0102] 1) Existing SSTable read performance optimization methods focus on improving data indexing speed, such as shortening the time to index key-value pairs by adding multi-level indexes. The introduction of multi-level indexes allows you to quickly locate the target data block by narrowing the search range layer by layer when searching for a specific key. Multi-level indexes increase the level of indexes, but the size of the index items themselves is not reduced. This is a typical space-for-time trade-off, but it increases the amount of data read. However, the space occupied by index blocks has not been reduced, and the time to read index blocks cannot be optimized. Existing methods optimize the data indexing time, which accounts for a lower proportion of the overall key-value pair reading time, while ignoring the optimization of the data reading time, which accounts for a higher proportion. Furthermore, under the standard configuration of RocksDB, the size of an index block can reach 265KB, while the size of a data block is only 4KB. The cost of reading index blocks is much higher than that of reading data blocks, resulting in performance bottlenecks.

[0103] 2) Traditional SSTables rely on index blocks to quickly locate key-value pairs in data blocks. However, as the amount of data increases, the size of index blocks and query complexity are also increasing, resulting in limited room for improvement in index speed and storage efficiency. Learning indexes only require some additional space, such as linear regression models that only need to record simple slopes and intercepts to predict the actual storage location of key-value data. Due to the immutability of SSTable, once built, its data distribution remains stable, which provides a fixed data set for learning indexes to train, enabling them to accurately learn and simulate the distribution characteristics of data in SSTables. This combination makes the application of learning indexes on SSTables both efficient and accurate, especially in environments that process large amounts of data, which can significantly improve the speed and efficiency of data retrieval. However, we cannot directly use string-type keys as training inputs for learning indexes, because string-type addition, subtraction, multiplication and division are much more complicated than integers, and require high-precision multiplication and matrix multiplication to calculate. The longer the string-type key, the greater the time and space overhead of calculation.

[0104] Figure 1a The process of reading data based on a key is described: 1. The user requests the value corresponding to the target key. 2. The user-state process reads the metadata of the tail block and obtains the location of the index block and the filter block. 3. The user-state process reads the filter block based on the location information of the filter block. Use the filter in the filter block, such as the Bloom filter, to find out whether the target key requested by the user exists. 3. If the target key exists, read the index block and perform a binary search in the index block to determine in which data block the key is specifically stored. If the target key does not exist, the data reading process ends. 4. Based on the location information of the target key, read the corresponding data block. 5. Read the target data in the data block. From Figure 1aAs can be seen, in the reading of a key-value pair, there are 4 operations to read the SSTable: reading the tail block, reading the filter block, reading the index block, and reading the data block. And 3 data index operations: the filter determines whether the data exists, the index block searches for the location of the data block where the key is located, and the data is searched in the data block.

[0105] Binary Search is an efficient algorithm for finding specific elements in an ordered array. Its basic principle is to quickly locate the target element by continuously reducing the search range by half. The premise of binary search is that the array must be ordered.

[0106] like Figure 1b As shown, the target key to be searched at the beginning is key_2=abaaccafbh. First, the left pointer left=0, the right pointer right=3 are initialized, and the middle position mid=(left+right) / 2=(0+3) / 2=1 is calculated.

[0107] In the first round of binary search, the key abaaacedbb at the middle position mid=1 is compared with the target key abaaccafbh. Since abaaacedbb is lexicographically smaller than abaaccafbh, we update the left pointer left=mid+1=2 and start the next round of search. The reason for updating the left pointer is that we have confirmed that the key abaaacedbb at the mid position is smaller than the target key abaaccafbh, so the target key abaaccafbh must be on the right side of mid. Therefore, we move the left pointer to mid+1 to narrow the search range and exclude the invalid area in the left half.

[0108] In the second round of binary search, the new left pointer left = 2, the right pointer right = 3, and the new middle position is:

[0109] mid=(left+right) / 2=(2+3) / 2=2. At this time, the key at the middle position mid=2 is abaaccafbh, which just matches the target key we are looking for, so the search is successful. In this way, binary search can efficiently find the target key in logarithmic time complexity. The ordered storage structure of SSTable makes this efficient search possible.

[0110] From the above method, it can be seen that data reading takes up most of the time, while data indexing takes up less time. Therefore, data reading becomes the bottleneck for reading a key-value pair. This application designs a new SStable structure, such as Figure 2As shown in the figure, the structure of the original index block is divided into a prefix index block and a model index block, and the other structures remain unchanged. The prefix index block includes: a prefix key index area, a special area, a length area, and a tail area. The prefix index area is used to store the prefix key of the key of the key-value pair data, such as prefix key 0, prefix key 1, prefix key..., prefix key n; the special area is used to store special values, such as Spec0, Spec1,..., Specn, and quickly locate the relative position of the key of the key-value pair data through the prefix key and the special value; the length area is used to store the length of the key, such as Len0, Len1,..., Lenn; the tail area is used to store the meta information of the prefix index block. The tail area is used to store the original information of the prefix index block, including the first address of the prefix key index area, the first address of the special area, and the first address of the length area; the model index block is used to store the linear regression model to predict the position of the key-value pair in the data block to improve the query efficiency. The key of the key-value pair data originally stored in the data block is changed to only store the suffix key, which reduces the space size of the data block. The tail block and the filter block remain unchanged.

[0111] Once the SSTable is determined, it will not change, and the index can be constructed based on this feature. The core of the algorithm is to determine a partition point. The partition point is a specific mark used to divide the string into two parts: the prefix key and the suffix key. Taking "abcde" as an example, if the partition point is set between "ab" and "cde", "ab" constitutes the prefix key of the original string, and "cde" becomes the suffix key. Each key is divided into two parts: the prefix key and the suffix key, and only the prefix key of the key is used to build the index. The core idea of ​​this method is that when performing a key size comparison operation, the characters are usually compared from left to right until the first different character appears. This comparison mechanism is highly consistent with the definition of the string prefix key. Therefore, by constructing an index based only on the prefix key part of the key, the amount of data that needs to be read during the index operation can be significantly reduced when reading the SSTable, thereby improving the overall processing efficiency.

[0112] refer to Figure 3 , is a system architecture diagram provided in an embodiment of the present application, the system includes: a terminal device 301, a server 302. The terminal device 301 is pre-installed with a business application for text matching, wherein the business application is a client application, a web application, a small program application, etc. The terminal device 301 can be a smart phone, a POS machine, a desktop notebook, a computer, etc., but is not limited thereto.

[0113] Server 302 is the background server of the business application. Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0114] Reference Figure 4 , is a flow chart of a data storage method provided in an embodiment of the present application, comprising the following steps:

[0115] Step 401, after sorting the multiple key-value pair data in order of the keys of the multiple key-value pair data to be stored, determine the prefix key and unique value corresponding to each key-value pair data; wherein the prefix key is the longest common part of the key of the key-value pair data to be stored and the key of the previous key-value pair data starting from the first position; the unique value is the first different value in the key of the key-value pair data to be stored and the key of the previous key-value pair data.

[0116] Specifically, the prefix key corresponding to the key-value pair data is a continuous non-empty substring starting from the first character to a certain end. Taking the string "abc" as an example, the set of prefix keys is {"a", "ab", "abc"}. As shown in Table 1 or 2, there are 4 keys of the key-value pair data, namely key_0 = aabaabaabb, key_1 = abaaacedbb, key_2 = abaaccafbh, key_3 = abadegafbh. Calculate the longest common substring (LCP) of each key and its previous key starting from the first position:

[0117]

[0118] LCP(key_0, key_1)="a";

[0119] LCP(key_1, key_2)="abaa";

[0120] LCP(key_2, key_3)="aba".

[0121] Three prefix keys are obtained, namely prefix_key_1 = "a", prefix_key_2 = "abaa", and prefix_key_3 = "aba".

[0122] The unique value is the first different value in the key of the previous key-value pair data. The unique value reveals the difference between the two keys. For example, the unique value of key_1 is "b", the unique value of key_2 is "c", and the unique value of key_1 is "d".

[0123] Step 402: store the prefix keys of the multiple key-value pair data in the prefix key index area of ​​the prefix index block in sequence, and store the unique values ​​of the multiple key-value pair data in the unique area of ​​the prefix index block in sequence; store the suffix keys and values ​​of the multiple key-value pair data in the data block; wherein the suffix key is the part of each key except the prefix key.

[0124] Specifically, after the prefix key is obtained, the prefix key is stored in the prefix key index area of ​​the prefix index block in lexicographic order, and the unique value is stored in the unique area of ​​the prefix index block in lexicographic order. After the prefix key is determined, the remaining part is the suffix key, the key of the key-value pair data = prefix key + suffix key, and the suffix key and value are stored in the data block.

[0125] For example, after the original index key is changed to a prefix key, the space occupied by the index block is reduced from 10*4=40 bytes to 1+4+3=8 bytes, which greatly reduces the space occupied, thereby reducing the time consumed in data reading. At the same time, the key in the data block is changed from a complete key to a suffix key, which also saves space.

[0126] In some embodiments, the prefix key index area may also include address information of the value corresponding to each prefix key in the data block.

[0127] In the prefix key index area, corresponding address information can be stored in each position corresponding to each prefix key. Through the index of the prefix key, the query range can be narrowed according to the address information corresponding to it. Similarly, the prefix key can also reduce the storage space. The embodiment of the present application is not limited, and whether the address information is stored in the prefix key index area can be determined according to the situation.

[0128] In the embodiment of the present application, the data structure for storing data is redesigned, the existing index block is changed into a prefix index block, and an index compression algorithm and a query algorithm are proposed accordingly, which can effectively reduce the size of the storage space required for the index block and the data block, thereby speeding up the speed of reading data into the memory. At the same time, the relative position obtained by the query algorithm is used as input, combined with the learning index, to overcome the limitation that the learning index is difficult to apply to string type data, and to make full use of its high indexing speed and low storage overhead, further improving the efficiency of key-value pair query.

[0129] In some embodiments, Figure 5 As shown, the prefix keys of the plurality of key-value pair data are sequentially stored in the prefix key index area of ​​the prefix index block, including:

[0130] Step 501: Use the longest prefix key among the prefix keys of multiple key-value pair data as the compressed prefix key; sequentially determine the length values ​​corresponding to the common parts of the prefix keys of each key-value pair data in the compressed prefix key to obtain a length sequence;

[0131] Specifically, the longest prefix key is obtained from the prefix keys of each key-value pair data as the compressed prefix key.

[0132] For example, Figure 6 It can be seen that the prefix key prefix_key_1 itself does not need to be compressed, so the entire prefix key prefix_key_1 is written into the compressed prefix key compress_prefix_keys, that is, compress_prefix_key = "a". At the same time, the length of the prefix key prefix_key_length[1] = 1 is recorded.

[0133] Let’s look at prefix_key_2. It can share the string “a” with prefix_key_1, so prefix_key_2 and prefix_key_1 can be combined and stored, compress_prefix_keys = “abaa”. At the same time, the length of the prefix key is recorded as prefix_key_length[2] = 4.

[0134] Let’s look at prefix_key_3. It can share the string "aba" with prefix_key_2. Similarly, prefix_key_2 and prefix_key_1 can be combined and stored, compress_prefix_keys = "aba". At the same time, the length of the prefix key is recorded as prefix_key_length[3] = 3.

[0135] Step 502: Store the compressed prefix key in the prefix key index area of ​​the prefix index block, and store the length sequence in the length area of ​​the prefix index block.

[0136] Specifically, after compression, we get Figure 6 The compressed prefix key and the length sequence of the prefix key are shown in the figure. The compressed prefix key part compress_prefix_keys occupies 4 bytes, and the largest number in the length sequence prefix_key_length is 4, which occupies 3 bits. Therefore, it occupies a maximum of 3*3=9bits=1.125bytes, a total of 4+1.125=5.125 bytes, which further compresses the index storage space compared to the original 8 bytes.

[0137] The prefix keys stored in the prefix key index area in lexicographic order are compressed prefix keys; the prefix key index area also includes a length sequence in the length area; the compressed prefix key is the longest prefix key among the prefix keys, and the length sequence is the length value corresponding to the common part of each prefix key in the compressed prefix key; before the prefix keys stored in sequence in the prefix key index area based on the prefix index block, it also includes: for the nth prefix key, based on the length value k of the nth bit in the length sequence, the first k bits in the compressed prefix key are used as the nth prefix key. Storing the length sequence can restore the prefix key prefix_key. For example, according to the first unit of the length sequence being 1, prefix_key_1=compress_prefix_keys[0:1]="a" can be restored. For another example, according to the second unit of the length sequence being 4, prefix_key_1=compress_prefix_keys[0:4]="abaa" can be restored. The compressed information can be restored, laying the foundation for query operations.

[0138] In particular, if there is a prefix key that cannot be merged and stored with the previous ones, then the prefix key needs to be stored separately as a group of compressed prefix keys. In the most extreme case, each prefix key cannot be merged and stored with the previous prefix key, so each group of compressed prefix keys can only store one prefix key at most. In special cases, the first prefix key is stored as a separate compressed prefix key because there is no prefix key before the first prefix key.

[0139] In the prefix index block, the obtained compressed prefix key and length sequence are stored. The compressed prefix key is saved in the prefix index area, and the length sequence is saved in the length area.

[0140] After the new SStable is built, you can use single-point query and range query to query the keys included in the query request and the stored multiple key-value pairs.

[0141] Single-point query is to query the corresponding value according to a single key. When you want to find the key aabaabaabb, you get Figure 5 The compressed prefix key is used to sequentially search for the required key in such a set of compressed prefix keys. The search process needs to first restore the prefix key based on the length sequence and the compressed prefix key, and obtain the unique value at the same time, such as Figure 7 shown.

[0142] Match the key and prefix key included in the query request. If the prefix key matches, continue to match the unique value corresponding to the prefix key. If the unique value does not match, return the position of the previous key of the current key, indicating that the relative position of the key to be queried in this SSTable is the position of the previous key. If both the prefix key and the unique value match, continue to match the next prefix key. If the prefix key does not match, directly match the next prefix key.

[0143] Based on the prefix keys stored in sequence in the prefix key index area of ​​the prefix index block, when the key of the target value matches the mth prefix key and the corresponding unique value, determine whether the key of the target value matches the m+1th prefix key and the corresponding unique value; if not, determine that the prefix key that matches the key of the target value is the mth prefix key.

[0144] For example, Figure 8 As shown, the query key aabaabaabb matches the prefix key 1, but does not match its unique value, indicating that the relative position of the key aabaabaabb is 0.

[0145] Another example Figure 8 As shown, the query key abaaccafbh matches the prefix key 1 and its unique value, and also matches the prefix key 2 and its unique value, but does not match the unique value of prefix 3, indicating that the relative position of the key abaaccafbh is 2.

[0146] For interval query, for example, to query [L, R], you only need to do a single-point query on the two keys at both ends of the interval. Do a single-point query on the left endpoint and the right endpoint as well.

[0147] In the embodiment of the present application, by further compressing the prefix key, the storage space of the data can be further reduced; the length corresponding to the compressed prefix key and the prefix key of each key-value pair data in the compressed prefix key is stored as a length sequence, and the prefix key can be restored according to the compressed prefix key and the length sequence. This saves space and does not affect the integrity of the prefix key.

[0148] In some embodiments, the set storage area also includes a model index block, including: determining the linear regression model of each interval based on the relative positions and storage addresses of multiple key-value pairs in the data block; wherein the linear regression model of each interval takes the relative position of the key-value pair data in the interval as input and the storage address of the key-value pair as output; the relative position represents the serial numbers of the multiple key-value pairs in the data block in ascending order of the storage address, and the linear regression model of each interval is stored in the model index block.

[0149] Specifically, the model index module is used to store linear regression models. The inconsistency of the length of the key-value pair data in the SSTable makes it difficult for a simple linear regression model to accurately predict the storage location of the key-value pair. Changes in length lead to fluctuations in the offset, which may lead to large errors for models that make predictions based on linear relationships. Therefore, the embodiment of the present application selects piecewise linear regression as the most appropriate model. Piecewise linear regression is particularly suitable for processing data sets that exhibit different linear characteristics in different intervals. The model subdivides the entire data set into multiple intervals and independently applies a linear regression model in each interval, so that it can more accurately capture and describe the complex changes in the data.

[0150] In some embodiments, Fig. 9 As shown, based on the relative positions and storage addresses of multiple key-value pairs in the data block, a linear regression model for each interval is determined, including the following steps:

[0151] Step 901: construct a linear regression model with the relative position of the key-value pair data in the data block as the independent variable and the storage address of the key-value pair data in the data block as the dependent variable.

[0152] The training of the linear regression model starts with the initialization of the model. At this time, the model only contains one reference point, that is, the offset position of the first key-value pair data is set to 0. As new key-value pairs are continuously written, the starting offset of each key-value pair is calculated and associated with its relative position in the data block to form a set of new data points. These new data points are added to the model immediately, triggering the refitting of the current linear relationship.

[0153] Although the piecewise linear regression model can capture this change, simple division alone is not enough to ensure accuracy. The dynamic programming method is used to optimize the segmentation and recursively solve the problem to minimize the overall fitting error, which ultimately reflects the complex change pattern of the data more accurately and ensures that the linear regression model in each interval can accurately predict the storage location of the key-value pair data.

[0154] In order to optimize the segmentation scheme of the piecewise linear regression model, the present invention uses the idea of ​​dynamic programming for optimization. The algorithm divides the data set into several segments, and each segment is fitted with a linear regression model, with the goal of minimizing the overall error. The following variables can be defined:

[0155] (1) n: The total number of data points, that is, the total number of keys in the key-value pair data.

[0156] (2) Fi: The linear regression equation with i as the root node, ΔFi(x) represents the storage position of the key numbered x and the fitting error of the linear regression model. Assume that the linear regression equation with i as the root node is Fi = kx + b, where k and b are fitting parameters, k is the slope of the fitting parameter, and b is the intercept of the fitting parameter. The storage position of the key numbered x is pos x , then ΔFi(x)=(Fi(x)-posx) 2 .

[0157] (3) DP[j]: minimum fitting error of the linear regression model for the jth bond ending.

[0158] (4) pre[i]: represents the previous point of i among the points that belong to the same linear regression model as i. For example, {0,2,7} belong to the same linear regression model, pre[7] = 2, pre[2] = 0, and pre[0] = 0.

[0159] (5) root(i): represents the point with the smallest position among the points that belong to the same linear regression model as i, which is also called the root node. For example, {0,2,7} belong to the same linear regression model, root(7) = root(2) = root(0) = 0. root(i) can be solved by the recursive formula (1), that is, root(7) = root(pre[7]) = root(2) = root(pre[2]) = root(0) = 0.

[0160]

[0161] Table 4 shows the relationship between the key and storage location in the key-value pair data:

[0162]

[0163] Table 4

[0164] First, initialize the state of the minimum fitting error (DP[j]) table. At the beginning, there is no fitting, each point is independent, so there is no error, DP[j] = 0, as shown in Table 5. At the same time, each point is not fitted with other points, so the leading node of each point is itself, and there is no fitting parameter at this time. The fitting parameters are slope and intercept.

[0165]

[0166] Table 5

[0167] Step 902: If the accumulated fitting error of the jth key-value pair data in the linear regression model corresponding to the i-th key-value pair data does not exceed the set threshold threshold, the jth key-value pair data is added to the linear regression model corresponding to the i-th key-value pair data, and the jth key-value pair data is located after the i-th key-value pair data.

[0168] Specifically, by solving formula (2), it is determined whether the fitting error of the j-th key-value pair data exceeds a set threshold.

[0169]

[0170] Among them, DP[j] represents the minimum fitting error of the linear regression model ending with j, min i<j Indicates enumerating all nodes i with numbers less than j. DP[i] indicates the minimum fitting error of the linear regression model ending with i. F root(i) (i) is the linear regression model where node i is located, and root(i) is because the fitting parameters of the model are recorded at the root node. (ΔF root(i) (j)) 2 It means to substitute the current node j into the above linear regression model and calculate the fitting error of point j. DP[i]+(ΔF root(i) (j)) 2 It represents the original fitting error plus the new error after point j is added to the model. If the total error of the two is less than the threshold, then point j can be added to the linear regression model.

[0171] Step 903: If the set threshold is exceeded, the j-th key-value pair data is used as the starting point of the new interval, and a linear regression model corresponding to the j-th key-value pair data is constructed until the linear regression models of the corresponding intervals are determined for all key-value pair data.

[0172] Specifically, if the fitting error of the jth key-value pair data accumulated in the linear regression model corresponding to the i-th key-value pair data exceeds the set threshold, let DP[j]=0, indicating that the regression model ending at point j only has itself, without adding any previous model.

[0173] The following is an explanation using a specific example, where threshold=100.

[0174] First, initialization: Let DP[0] = 0, which means that the key numbered 0 itself constitutes a linear regression model with an error of 0.

[0175] Second, calculate the minimum fitting error of the second node according to formula (2).

[0176]

[0177] The root node of node 0 is itself. From Table 6, we can see that there is no fitting parameter at number 0, so (ΔF root(0) (1)) 2 =0, which means that the node numbered 1 is added to the linear regression model to which 0 originally belonged, and the newly added error is 0. Therefore, the node numbered 1 can form a linear regression model together with 0, and Table 6 is updated: DP[1]=0, indicating that the minimum fitting error of the linear regression model ending with 1 is 0. At the same time, Table 7 is updated so that the leading node of node 1 is 0, and the fitting parameters (slope, intercept) at the root node 0 are updated to (120,0), as shown in Table 7-1.

[0178]

[0179] Table 6-1

[0180]

[0181] Table 7-1

[0182] Third, calculate the minimum fitting error of the third node according to formula (2).

[0183]

[0184] In the above recursive process, try to add the point numbered 2 to the linear regression model ending with 1 or 0, and the overall error after adding it needs to be less than threshold = 100. Because 0 and 1 belong to the same linear regression model, we only need to calculate DP[1]+(ΔF root(1) (1)) 2 , the calculation result is 100, which is not less than the threshold value 100, so the point numbered 2 can only form a linear regression model alone. Update Table 6 to obtain Table 6-2, DP[3] = 0.

[0185]

[0186] Table 6-2

[0187] Fourth, calculate the minimum fitting error of the fourth node according to formula (2).

[0188]

[0189] In the above recursive process, try to add the point numbered 3 to the linear regression model ending with 2 or 1, and the overall error after adding needs to be less than threshold = 100. The overall fitting error of the linear regression model ending with 2 is 0, while the overall fitting error of the linear regression model ending with 1 is 19600. Therefore, add 3 to {2}, and {2,3} constitutes a linear regression model. DP[3] = 0, update to Table 6, and get Table 6-3. At the same time, update the leading of 3 to 2, and record the fitting parameters (slope, intercept) = (250, -250) at the root node 2.

[0190]

[0191] Table 6-3

[0192]

[0193] Table 7-2

[0194] From Table 6-3, we can see that the best segmentation scheme is:

[0195] (1) Interval 1: The key is from 0 to 1, and the linear regression model is: offset = 120*key+0.

[0196] (2) Interval 2: The key is from 2 to 3, and the linear regression model is: offset = 250*key-250.

[0197] The corresponding model index block record is as follows:

[0198] The triplet of interval 1: {starting position of interval 1, slope, intercept} = {0, 120, 0}.

[0199] The triplet of interval 2: {starting position of interval 2, slope, intercept} = {250, 250, -250}.

[0200] In actual applications, after determining that the relative positions of the two key-value pairs key0 and key3 in the SSTable are 0 and 3 respectively, the line segment containing key0 (recorded as segment0) and the line segment containing key3 (recorded as segment1) are accurately locked. Finally, the system will extract the corresponding slope and intercept parameters from these two segments, which are the key parameters for the piecewise linear regression model to predict the actual position.

[0201] Based on the extracted slope and intercept, the learning index uses the formula of the piecewise linear regression model to calculate the physical offset positions of the key0 and key3 key-value pairs, and obtains offset0 = 120 × 0 = 0, and offset3 = 250 × 3 - 250 = 500. This calculation process not only accurately reflects the position of the key-value pairs in physical storage, but also fully considers the inherent rules of key-value pair sorting in SSTable and the increasing nature of the physical offset.

[0202] In the embodiment of the present application, the data structure for storing data is redesigned, the existing index block is changed to a prefix index block, and an index compression algorithm and a query algorithm are proposed accordingly, which can effectively reduce the size of the storage space required for the index block and the data block, thereby speeding up the speed of reading data into the memory. At the same time, the relative position obtained by the query algorithm is used as input, combined with the learned index, to overcome the limitation that the learned index is difficult to apply to string type data, and to make full use of its high indexing speed and low storage overhead, and further improve the efficiency of key-value pair queries. Based on the same technical concept, the embodiment of the present application provides a data storage device suitable for a key-value pair storage format arranged in order by keys, such as Fig.10 As shown, the device 1000 includes:

[0203] The determination module 1001 is used to sort the multiple key-value pair data to be stored in the order of their keys, and then determine the prefix key and unique value corresponding to each key-value pair data; wherein the prefix key is the longest common part of the key of the previous key-value pair data; and the unique value is the first different value in the key of the previous key-value pair data;

[0204] The storage module 1002 is used to sequentially store the prefix keys of the plurality of key-value pair data into the prefix key index area of ​​the prefix index block, and sequentially store the unique values ​​of the plurality of key-value pair data into the unique area of ​​the prefix index block; wherein the prefix key index area also includes the address information of the value corresponding to each prefix key in the data block;

[0205] The storage module 1002 is further configured to store the suffix keys and values ​​of the plurality of key-value pairs in a data block; wherein the suffix key is the portion of each key excluding the prefix key.

[0206] Optionally, the storage module 1002 is specifically used for:

[0207] Compress the prefix keys of the plurality of key-value pair data to obtain compressed prefix keys; determine the length corresponding to the prefix key of each key-value pair data in the compressed prefix key to obtain a length sequence; obtain a plurality of prefix key information based on the length sequence and the compressed prefix key;

[0208] The compressed prefix key is stored in the prefix key index area of ​​the prefix index block, and the length sequence is stored in the length area of ​​the prefix index block.

[0209] Optionally, the determining module 1001 is specifically configured to:

[0210] Determine a linear regression model for each interval based on the relative positions and storage addresses of the plurality of key-value pairs in the data block; wherein the linear regression model for each interval takes the relative positions of the key-value pairs in the interval as input and the storage addresses of the key-value pairs as output;

[0211] Store the linear regression model for each interval into the model index block.

[0212] Optionally, the determining module 1001 is further configured to:

[0213] A linear regression model is constructed with the relative position of the key-value pair data in the data block as the independent variable and the storage address of the key-value pair data in the data block as the dependent variable;

[0214] If the fitting error of the accumulated j-th key-value pair data in the linear regression model corresponding to the i-th key-value pair data does not exceed the set threshold, the j-th key-value pair data is added to the linear regression model corresponding to the i-th key-value pair data, and the j-th key-value pair data is located after the i-th key-value pair data;

[0215] If it exceeds the set threshold, the j-th key-value pair data is used as the starting point of the new interval, and a linear regression model corresponding to the j-th key-value pair data is constructed until all key-value pair data determine the linear regression model of the corresponding interval.

[0216] Optionally, the prefix index block further includes a tail area, and the tail area is used to store meta information of the prefix index block.

[0217] Based on the same technical concept, the embodiment of the present application provides a data query device suitable for a key-value pair storage format arranged in order by keys, such as Fig.11 As shown, the device 1100 includes:

[0218] An acquisition module 1101 is used to acquire a query request, where the query request includes a key of a target value;

[0219] The determination module 1102 is used to determine a prefix key that matches the key of the target value based on each prefix key sequentially stored in the prefix key index area of ​​the prefix index block and each unique value stored in the unique area of ​​the prefix index block; wherein the prefix key is the longest common part of the key of the previous key-value pair data; and the unique value is the first different value in the key of the previous key-value pair data;

[0220] A matching module 1103 is used to obtain a suffix key of the matched prefix key from a corresponding data block based on the matched prefix key; wherein the suffix key is the part of each key except the prefix key;

[0221] The matching module 1103 is further configured to obtain the value of the target key from the data block after the matched prefix key and the suffix key of the matched prefix key are combined to be consistent with the target key.

[0222] Optionally, each prefix key sequentially stored in the prefix key index area is a compressed prefix key; the prefix key index area further includes a length sequence located in the length area, and the length sequence is the length corresponding to the prefix key of each key-value pair data in the compressed prefix key;

[0223] The determining module 1102 is further configured to:

[0224] Based on the compressed prefix key and the length sequence, each prefix key stored in sequence in the prefix key index area is obtained.

[0225] Optionally, the model index block stores a linear regression model for each interval, and the linear regression model for any interval takes the relative position of the key-value pair data in the interval as input and the storage address of the key-value pair as output;

[0226] The matching module 1103 is specifically used for:

[0227] Determine the interval corresponding to the matching prefix key;

[0228] According to the linear regression model of the corresponding interval, based on the relative position of the matched prefix key in each prefix key, a target storage address of the key-value pair of the matched prefix key is obtained;

[0229] According to the target storage address, a suffix key of the matched prefix key is obtained from a corresponding data block.

[0230] Based on the same technical concept, the embodiment of the present application provides a computer device, which may be a terminal or a server, such as Fig.12 As shown, it includes at least one processor 1201 and a memory 1202 connected to the at least one processor. The specific connection medium between the processor 1201 and the memory 1202 is not limited in the embodiment of the present application. Fig.12 For example, the processor 1201 and the memory 1202 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0231] In an embodiment of the present application, the memory 1202 stores instructions that can be executed by at least one processor 1201. The at least one processor 1201 can execute the steps included in the above-mentioned cosmic ray removal method by executing the instructions stored in the memory 1202.

[0232] Among them, the processor 1201 is the control center of the computer device, and can use various interfaces and lines to connect various parts of the computer device, by running or executing instructions stored in the memory 1202 and calling data stored in the memory 1202. Optionally, the processor 1201 may include one or more processing units, and the processor 1201 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1201. In some embodiments, the processor 1201 and the memory 1202 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0233] Processor 1201 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0234] Memory 1202, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1202 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. Memory 1202 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1202 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0235] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned cosmic ray removal method.

[0236] Based on the same inventive concept, an embodiment of the present application provides a computer program product, characterized in that the computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned cosmic ray removal method.

[0237] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0238] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0239] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0240] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0241] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data storage method, characterized in that: Applicable to storing key-value pairs arranged in lexicographical order into the specified storage area; The set storage area includes: a data block and a prefix index block; the prefix index block includes: a prefix key index area and a special area; the prefix key index area stores a prefix key of any key-value pair data, and the special area stores a special value of any key-value pair data, and the method includes: After sorting the multiple key-value pair data to be stored in order of their keys, determine the prefix key and unique value corresponding to each key-value pair data; wherein the prefix key is the longest common part of the key-value pair data to be stored and the key of the previous key-value pair data starting from the first position; the unique value is the first different value in the key of the key-value pair data to be stored and the key of the previous key-value pair data; The prefix keys of the plurality of key-value pairs are sequentially stored in the prefix key index area of ​​the prefix index block, and the unique values ​​of the plurality of key-value pairs are sequentially stored in the unique area of ​​the prefix index block; The suffix keys and values ​​of the plurality of key-value pairs are stored in the data block; wherein the suffix key is the part of each key except the prefix key.

2. The method according to claim 1, characterized in that The prefix index block also includes a length area; the sequentially storing the prefix keys of the plurality of key-value pairs of data in the prefix key index area of ​​the prefix index block includes: The longest prefix key among the prefix keys of the plurality of key-value pair data is used as a compressed prefix key; the length values ​​corresponding to the common parts of the prefix keys of each key-value pair data in the compressed prefix keys are sequentially determined to obtain a length sequence; The compressed prefix key is stored in the prefix key index area of ​​the prefix index block, and the length sequence is stored in the length area of ​​the prefix index block.

3. The method according to claim 1, characterized in that The set storage area also includes a model index block, and the method further includes: Based on the relative positions and storage addresses of the multiple key-value pairs in the data block, a linear regression model for each interval is determined; wherein the linear regression model for each interval takes the relative positions of the key-value pairs in the interval as input and the storage addresses of the key-value pairs as output; the relative positions represent the serial numbers of the multiple key-value pairs in the data block in ascending order of the storage addresses; The linear regression model of each interval is stored in the model index block.

4. The method according to claim 3, characterized in that The step of determining a linear regression model for each interval based on the relative positions and storage addresses of the plurality of key-value pair data in the data block comprises: A linear regression model is constructed with the relative position of the key-value pair data in the data block as the independent variable and the storage address of the key-value pair data in the data block as the dependent variable; If the fitting error of the accumulated j-th key-value pair data in the linear regression model ending with the i-th key-value pair data does not exceed the set threshold, then the j-th key-value pair data is added to the linear regression model corresponding to the i-th key-value pair data, and the j-th key-value pair data is located one after the i-th key-value pair data; If it exceeds the set threshold, the j-th key-value pair data is used as the starting point of the new interval, and a linear regression model corresponding to the j-th key-value pair data is constructed until all key-value pair data determine the linear regression model of the corresponding interval.

5. A data query method, characterized in that: The method is suitable for storing key-value pair data in a set storage area in which the keys are arranged in lexicographic order, wherein the set storage area includes: a data block and a prefix index block; the prefix index block includes: a prefix key index area and a special area; the method includes: Obtaining a query request, wherein the query request includes a key of a target value; Based on the prefix keys stored in the prefix key index area of ​​the prefix index block in lexicographic order and the unique values ​​stored in the unique area of ​​the prefix index block, determine the prefix key that matches the key of the target value; wherein the prefix key is the longest common part of the key of any key-value pair data and the key of the previous key-value pair data starting from the first position; the unique value is the first different value between the key of any key-value pair data and the key of the previous key-value pair data; Obtain the suffix key of the matching prefix key from the data block indicated by the matching prefix key; wherein the suffix key is the part of each key except the prefix key; After the matching prefix key is combined with the suffix key of the matching prefix key, the value of the target key is obtained from the data block.

6. The method according to claim 5, characterized in that The method of determining a prefix key matching a key of the target value based on the prefix keys stored in the prefix key index area of ​​the prefix index block in lexicographic order and the unique values ​​stored in the unique area of ​​the prefix index block comprises: For each prefix key sequentially stored in the prefix key index area based on the prefix index block, when the key of the target value matches the mth prefix key and the corresponding unique value, determining whether the key of the target value matches the m+1th prefix key and the corresponding unique value; If there is no match, the prefix key that matches the key of the target value is determined to be the mth prefix key.

7. The method according to claim 6, characterized in that Each prefix key stored in the prefix key index area in lexicographic order is a compressed prefix key; the prefix key index area also includes a length sequence located in the length area; the compressed prefix key is the longest prefix key among the prefix keys, and the length sequence is the length value corresponding to the common part of each prefix key in the compressed prefix key; Before each prefix key sequentially stored in the prefix key index area based on the prefix index block, it also includes: For the nth prefix key, based on the length value k of the nth digit in the length sequence, the first k digits of the compressed prefix key are used as the nth prefix key.

8. The method according to claim 6 or 7, characterized in that The linear regression model of each interval is stored in the model index block. The linear regression model of any interval takes the relative position of the key-value pair data in the interval as input and the storage address of the key-value pair as output; The acquiring, based on the matched prefix key, the suffix key of the matched prefix key from the corresponding data block comprises: Determine the interval corresponding to the matching prefix key; According to the linear regression model of the corresponding interval, based on the relative position of the matched prefix key in each prefix key, a target storage address of the key-value pair of the matched prefix key is obtained; According to the target storage address, a suffix key of the matched prefix key is obtained from a corresponding data block.

9. A data storage device, characterized in that: A key-value pair storage format suitable for key-ordered storage; the device comprises: A determination module, configured to determine a prefix key and a unique value corresponding to each key-value pair data after sorting the multiple key-value pair data in the order of the keys of the multiple key-value pair data to be stored; wherein the prefix key is the longest common part of the key of the previous key-value pair data; and the unique value is the first different value in the key of the previous key-value pair data; A storage module, used to sequentially store the prefix keys of the plurality of key-value pair data into a prefix key index area of ​​a prefix index block, and sequentially store the unique values ​​of the plurality of key-value pair data into a unique area of ​​the prefix index block; wherein the prefix key index area also includes address information of the value corresponding to each prefix key in the data block; The storage module is also used to store the suffix keys and values ​​of the multiple key-value pairs in a data block; wherein the suffix key is the part of each key except the prefix key.

10. A data query device, characterized in that: Applicable to key-value pair storage formats arranged in order by key, including: An acquisition module, used for acquiring a query request, wherein the query request includes a key of a target value; A determination module, configured to determine a prefix key matching the key of the target value based on each prefix key sequentially stored in the prefix key index area of ​​the prefix index block and each unique value stored in the unique area of ​​the prefix index block; wherein the prefix key is the longest common part of the key of the previous key-value pair data; and the unique value is the first different value in the key of the previous key-value pair data; A matching module, configured to obtain a suffix key of the matched prefix key from a corresponding data block based on the matched prefix key; wherein the suffix key is a portion of each key excluding the prefix key; The matching module is further configured to obtain the value of the target key from the data block after the matched prefix key and the suffix key of the matched prefix key are combined to be consistent with the target key.

11. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method according to any one of claims 1 to 8 are implemented when the processor executes the program.

12. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device, and when the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 8.

Images