File data storage and reading method and device for database storage

Abstract

The present invention discloses a file data storage and reading method applied to database storage, including file data storage, wherein the file data storage comprises: writing key-value data to a data area of ​​a file, recording an offset and size S[offset+size] in the data area; calculating a hash value hash(key) of a key in the key-value data, and updating a bit value of the hashID position corresponding to the hash value hash(key) in a bitmap; the hashID, key, and S[offset+size] are used as a hash index record when writing key-value data, and the hash index record is written in a file. The file data storage and reading method applied to database storage of the present invention adopts a new file format through a hash index when performing data storage, and the time complexity can reach O(1) when performing key value retrieval, which greatly improves the retrieval efficiency of key-value data.

Description

Technical Field

[0001] The present invention relates to the technical field of database storage, and in particular to a method and device for storing and reading file data applied to database storage. Background Art

[0002] KV database refers to Key-value database, which is a kind of database that stores data in key-value pairs. In order to improve the query speed of the database, it is necessary to establish a database index for the database. There are three common database index data structures for KV database: hash table, ordered array and search tree.

[0003] The hash table stores key-value pairs. If you use a hash table to store indexes, you must first use a hash function to convert the key into a corresponding index position in the array, and then put the value at this position. If multiple keys get the same index after conversion, then add a linked list here to store key-value objects with the same hash value. If you are given a specified key and asked to query the corresponding value, you must first convert the index value of the array according to the hash function. If there are multiple values ​​at this position, then traverse the linked list. The disadvantages of the hash table are obvious. Since the hash table is unordered, the speed of interval query using the hash index is very slow.

[0004] The performance of ordered arrays in equal value queries and range queries is excellent. The key-value array is stored in order. At this time, if you want to find the value corresponding to the key, you can use the binary search method to quickly get it, and the time complexity is O(log(N)). Although the binary search method can greatly improve the efficiency of queries, the time complexity of O(log(N)) is still inefficient for database queries. In addition, the key-value array is stored in order, which is troublesome when updating data. To insert a record in the middle, all the subsequent records must be moved, which is too costly.

[0005] Search tree, a data structure, it is a set of n (n>=0) finite nodes with a hierarchical relationship. It is called a "tree" because it looks like an upside-down tree, that is, its root is facing up and its leaves are facing down. Search tree query generally uses binary search, and the time complexity is O(log(N)).

[0006] In view of this, the technical problem to be solved by the present invention is: how to improve the query efficiency of the KV database, and even the time complexity of the KV database query can reach O(1), where O(1) means that the target element can be directly obtained in one operation. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a file data storage and reading method and device applied to database storage. The file data storage method applied to database storage obtains a new file format, which can be indexed by hash, and the time complexity can basically reach O(1). The specific technical solution is as follows:

[0008] The file data storage and reading method applied to database storage includes file data storage, and the file data storage includes:

[0009] Write the key-value data to the data area of ​​the file, and record the offset and size S[offset+size] in the data area;

[0010] Calculate the hash value hash(key) of the key in the key-value data, and update the bit value of the hashIDth bit corresponding to the hash value hash(key) in the bitmap;

[0011] The hashID, key, S[offset+size] are used as hash index records when key-value data is written, and the hash index records are written in the file.

[0012] As an optional implementation manner of the present invention, in the file data storage and reading method applied to database storage, recording the hash index record in a file includes:

[0013] Sort the hash index records by hashID, write the offset and size of all hashIDs into the hash area of ​​the file in a fixed-length manner, and record the offset and size H[offset+size] of the hash area;

[0014] Serialize the bitmap recorded by the hash index, write it into the file as the bitmap area, and record the offset and size B[offset+size] of the bitmap area;

[0015] The offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area are stored as the footer area of ​​the file.

[0016] As an optional implementation manner of the present invention, in the file data storage and reading method applied to database storage, before sorting the hash index records by hashID, the method further includes:

[0017] According to the hash index record, the key with the same hashID and the offset and size S[offset+size] corresponding to the key in the data area are packaged into a conflict block, and written into the conflict area of ​​the file in sequence;

[0018] Save the offset and size C[offset+size] of the conflicting block of the key with the same hashID in the conflicting area, and write back C[offset+size] as the unique associated offset and size of the key with the same hashID.

[0019] As an optional embodiment of the present invention, the file data storage and reading method applied to database storage includes: in the process of writing the key-value data to the data area of ​​the file, calculating the check code of the key-value data, storing the check code in the check area of ​​the file, and recording the offset and size J[offset+size] of the check code of each key in the check area; saving the offset and size J[offset+size] in the check area to the footer area of ​​the file.

[0020] As an optional implementation manner of the present invention, the file data storage and reading method applied to database storage includes file data reading, and the file data reading includes file opening, and the file opening process includes:

[0021] Read the offset and size S[offset+size] of the data area in the footer area of ​​the file, the offset and size J[offset+size] of the check area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area;

[0022] According to the offset and size B[offset+size] of the bitmap area, the data in the bitmap area is deserialized and stored in the memory;

[0023] According to the offset and size H[offset+size] of the hash area, all hashIDs and corresponding offset+size are loaded into the memory.

[0024] As an optional implementation manner of the present invention, in the file data storage and reading method applied to database storage, the file data reading includes a data reading process, and the data reading process includes:

[0025] Perform hash calculation on the query key to obtain hash(key) of the query key, and further obtain hashID=hash(key) of the query key;

[0026] Check whether the key exists in the bitmap area according to the hashID of the query key. If it does not exist, return an empty result and the query ends. If it does exist, proceed to the next step.

[0027] According to the rank of the query key’s hashID in the bitmap area, calculate the offset and size H[offset+size] of the hash area record;

[0028] According to the offset and size H[offset+size] of the hash area, the query data corresponding to the query key is read in the data area or the conflict area.

[0029] As an optional implementation of the present invention, in the file data storage and reading method applied to database storage, reading the query data corresponding to the query key in the data area or the conflict area according to the offset and size H[offset+size] of the hash area includes:

[0030] Determine whether it falls into the conflict area according to the offset and size H[offset+size] of the hash area;

[0031] If it is determined that it does not fall in the conflict area, the data is read directly from the data area according to the offset and size S[offset+size]. If it is determined that it falls in the conflict area, the conflict block is determined according to the offset and size C[offset+size] of the corresponding conflict area, and the offset and size S[offset+size] of the query key in the data area are replaced by binary search in the conflict block, and the data is read from the data area.

[0032] As an optional implementation of the present invention, in the file data storage and reading method applied to database storage, the calculation of the offset and size H[offset+size] of the hash area record according to the sorting rank of the hashID of the query key in the bitmap area includes:

[0033] The fixed length of the hash area when writing each H[offset+size] is width, and the writing is in order, so the offset and size H[offset+size] of the query key recorded in the hash area are calculated by rank*width.

[0034] The present invention also provides a file data storage and reading device for database storage, including a file data storage module, wherein the file data storage module includes:

[0035] The data writing unit writes the key-value data to the data area of ​​the file and records the offset and size S[offset+size] in the data area;

[0036] Hash calculation unit, calculates the hash value hash(key) of the key in the key-value data;

[0037] Bitmap unit, updates the bit value of the hashIDth bit corresponding to the hash value hash(key) in the bitmap;

[0038] And a hash index record unit, which records the hashID, key, S[offset+size] when writing key-value data as a hash index record and writes it in the file.

[0039] The present invention also provides a storage medium storing a computer executable program, characterized in that when the computer executable program is executed, the file data storage and reading method applied to database storage as described in any one of the above items is implemented.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The file data storage and reading method applied to database storage of the present invention updates the bit value on the corresponding bit position of the bitmap according to the hash value hash(key) of the key in the key-value data when performing data storage, and uses it as the hash index record when the key-value data is written; in this way, when performing a key query, the data existence is judged through the bitmap area. If the data does not exist, it can be returned with only one comparison calculation overhead. If the data exists, the data is directly indexed according to the hash index record to obtain the key-value data.

[0042] Therefore, the file data storage and reading method applied to database storage of the present invention adopts a new file format through hash index when storing data, and the time complexity can reach O(1) when performing key value retrieval, which greatly improves the retrieval efficiency of key-value data.

[0043] The file data storage and reading method for database storage of the present invention forms a new file data structure when storing key-value data. The meaning and function of each partition of the specific file data structure are as follows:

[0044] Data area: stores compressed key-value data;

[0045] Verification area: Use cyclic verification code to verify the data area;

[0046] Conflict zone: keys with the same hash value and their offset and data size in the data zone will be placed in the conflict zone;

[0047] Hash area: a fixed-length area that stores the offset and size of the data area where the key is located or the conflict area;

[0048] Bitmap area: Use bitmap to store whether the bit corresponding to the hash of the key has a value;

[0049] Footer area: fixed-length storage data area, check area, conflict area, hash area, offset and size of the bitmap area. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Process of the file data storage and reading method applied to database storage according to the embodiment of the present invention Figure 1 ;

[0051] Figure 2 Schematic diagram of marking the hashID bit in a bitmap in a file data storage and reading method applied to database storage according to an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of processing the same hashID in a file data storage and reading method applied to database storage according to an embodiment of the present invention;

[0053] Figure 4 A processing flow chart of a file data storage and reading method applied to database storage according to an embodiment of the present invention for the same hashID;

[0054] Figure 5 A file data structure applicable to the file data storage and reading method applied to database storage according to an embodiment of the present invention;

[0055] Figure 6 Process of the file data storage and reading method applied to database storage according to the embodiment of the present invention Figure 2 ;

[0056] Figure 7 Process of the file data storage and reading method applied to database storage according to the embodiment of the present invention Figure 3 . DETAILED DESCRIPTION

[0057] To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

[0058] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0060] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0061] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0062] See also Figure 1 and Figure 2 As shown, this embodiment provides a file data storage and reading method applied to database storage, including file data storage, wherein the file data storage includes:

[0063] Write the key-value data to the data area of ​​the file, and record the offset and size S[offset+size] in the data area;

[0064] Calculate the hash value hash(key) of the key in the key-value data, and update the bit value of the hashIDth bit corresponding to the hash value hash(key) in the bitmap;

[0065] The hashID, key, S[offset+size] are used as hash index records when key-value data is written, and the hash index records are written in the file.

[0066] In this embodiment, hash(key), where key represents the key value of the element, the value of hash(key) represents the hash value calculated by the hash function.

[0067] The so-called bitmap uses a bit to mark the value corresponding to an element, and the key is the element. Since the data is stored in bits, the storage space can be greatly saved.

[0068] The file data storage and reading method applied to database storage in this embodiment updates the bit value on the corresponding bit of the bitmap according to the hash value hash(key) of the key in the key-value data when performing data storage, and uses it as the hash index record when writing the key-value data; in this way, when performing a key query, the data existence is judged through the bitmap area. If the data does not exist, it can be returned with only one comparison calculation overhead. If the data exists, the key-value data is directly indexed to the data according to the hash index record to obtain the key-value data. Therefore, the file data storage and reading method applied to database storage in this embodiment adopts a new file format through hash indexing when performing data storage, and the time complexity can reach O(1) when performing key value retrieval, which greatly improves the retrieval efficiency of key-value data.

[0069] As an optional implementation of this embodiment, in the file data storage and reading method applied to database storage described in this embodiment, recording the hash index record in a file includes:

[0070] Sort the hash index records by hashID, write the offset and size of all hashIDs into the hash area of ​​the file in a fixed-length manner, and record the offset and size H[offset+size] of the hash area;

[0071] Serialize the bitmap recorded by the hash index, write it into the file as the bitmap area, and record the offset and size B[offset+size] of the bitmap area;

[0072] The offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area are stored as the footer area of ​​the file.

[0073] After sorting by hashID, this embodiment stores the offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area as the footer area of ​​the file. When querying the key value, if the data existence in the bitmap area is judged as the key value, the data area can be gradually indexed to obtain the key-value data according to the offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area.

[0074] An ideal hash function would achieve the effect that if key1≠key2, then hash(key1)≠hash(key2). However, in reality, it is almost impossible to find a hash function that has different hash values ​​for different keys. In fact, no matter how good a hash function is, hash collisions are unavoidable. For a hash function, no matter how large the storage area (n) is, when the data to be stored is greater than n, there will inevitably be situations where the hash values ​​are the same. This is called a hash collision.

[0075] Since this embodiment performs hash calculation on key-value data, in principle, assuming that the hash algorithm is uniform, the hash values ​​hash(key) obtained by hash calculation of key-value data are different from each other. However, if there are two or more key-value data that have the same hash value hash(key) after hash calculation, a group of identical hashIDs will appear, which will affect the hashID sorting. Therefore, see Figure 3 and Figure 4 As shown, as an optional implementation of this embodiment, this embodiment further includes, before sorting the hash index records by hashID:

[0076] According to the hash index record, the key with the same hashID and the offset and size S[offset+size] corresponding to the key in the data area are packaged into a conflict block, and written into the conflict area of ​​the file in sequence;

[0077] Save the offset and size C[offset+size] of the conflicting block of the key with the same hashID in the conflicting area, and write back C[offset+size] as the unique associated offset and size of the key with the same hashID.

[0078] This embodiment packages key-value data with the same hash value obtained through hash calculation into a conflict block, stores the conflict block in the conflict area of ​​the file, sorts the entire conflict block sharing the same hashID, and uses the offset and size C[offset+size] of the conflict block stored in the conflict area of ​​the file as the offset and size of the key-value data storage corresponding to the hashID.

[0079] In the file data storage and reading method applied to database storage of this embodiment, when performing file data storage, assuming that the hash algorithm is uniform, a large proportion of data will not fall into the conflict area. For data that does not fall into the conflict area, only one io in the hash area is required to obtain the offset and size H[offset+size] of the hash area of ​​the data, and one io in the data area is required to obtain the data through the offset and size S[offset+size] of the data area. Even if the queried key falls into the conflict area, a binary search is performed on the key in the same conflict area, which only requires a very small io cost, and the overall average query times are constant.

[0080] See also Figure 4 As shown, as an optional implementation of the present embodiment, the file data storage and reading method applied to database storage described in the present embodiment, in the process of writing the key-value data to the data area of ​​the file, the check code of the key-value data is calculated, the check code is stored in the check area of ​​the file, and the offset and size J[offset+size] of the check code of each key in the check area are recorded; the offset and size J[offset+size] in the check area are saved to the footer area of ​​the file.

[0081] The verification area of ​​this embodiment is used to perform data verification on the data area when opening a file. The footer area of ​​this embodiment can add custom data as a file format verification when reading, and the entire footer area is a fixed size.

[0082] In summary, the file data storage and reading method for database storage described in this embodiment forms a new file data structure when performing key-value data storage. Figure 5 As shown, the meaning and function of each partition of the specific file data structure are:

[0083] Data area: stores compressed key-value data;

[0084] Verification area: Use cyclic verification code to verify the data area;

[0085] Conflict zone: keys with the same hash value and their offset and data size in the data zone will be placed in the conflict zone;

[0086] Hash area: a fixed-length area that stores the offset and size of the data area where the key is located or the conflict area;

[0087] Bitmap area: Use bitmap to store whether the bit corresponding to the hash of the key has a value;

[0088] Footer area: fixed-length storage data area, check area, conflict area, hash area, offset and size of the bitmap area.

[0089] At the same time, the file data storage and reading method applied to database storage described in this embodiment includes file data reading, and the file data reading includes file opening, and the file opening process includes:

[0090] Open the file, read the content of the footer area according to the fixed size, and perform file format verification on the footer area;

[0091] Read the offset and size S[offset+size] of the data area in the footer area of ​​the file, the offset and size J[offset+size] of the check area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area;

[0092] According to the offset and size B[offset+size] of the bitmap area, the data in the bitmap area is deserialized and stored in the memory;

[0093] According to the offset and size H[offset+size] of the hash area, all hashIDs and corresponding offset+size are loaded into the memory.

[0094] Further, see Figure 6 and Figure 7 As shown, the file data storage and reading method applied to database storage described in this embodiment, the file data reading includes a data reading process, and the data reading process includes:

[0095] Perform hash calculation on the query key to obtain hash(key) of the query key, and further obtain hashID=hash(key) of the query key;

[0096] Check whether the key exists in the bitmap area according to the hashID of the query key. If it does not exist, return an empty result and the query ends. If it does exist, proceed to the next step.

[0097] According to the rank of the query key’s hashID in the bitmap area, calculate the offset and size H[offset+size] of the hash area record;

[0098] According to the offset and size H[offset+size] of the hash area, the query data corresponding to the query key is read in the data area or the conflict area.

[0099] Furthermore, the file data storage and reading method applied to database storage described in this embodiment, reading the query data corresponding to the query key in the data area or the conflict area according to the offset and size H[offset+size] of the hash area includes:

[0100] Determine whether it falls into the conflict area according to the offset and size H[offset+size] of the hash area;

[0101] If it is determined that it does not fall in the conflict area, the data is read directly from the data area according to the offset and size S[offset+size]. If it is determined that it falls in the conflict area, the conflict block is determined according to the offset and size C[offset+size] of the corresponding conflict area, and the offset and size S[offset+size] of the query key in the data area are replaced by binary search in the conflict block, and the data is read from the data area.

[0102] As an optional implementation of this embodiment, see Figure 7 As shown, the file data storage and reading method applied to database storage described in this embodiment, the offset and size H[offset+size] of the hash area record are calculated according to the sort rank of the hashID of the query key in the bitmap area, including:

[0103] The fixed length of the hash area when writing each H[offset+size] is width, and the writing is in order, so the offset and size H[offset+size] of the query key recorded in the hash area are calculated by rank*width.

[0104] As an optional implementation of this embodiment, the data verification area can adopt a variety of different verification algorithms; the hsah algorithm can select different specific numbers in the footer area and can select different values; the offset+size information of each area in the footer area can be stored separately in the meta area, and the footer area only stores the offset+size of the meta area.

[0105] This embodiment also provides a file data storage and reading device for database storage, including a file data storage module, wherein the file data storage module includes:

[0106] The data writing unit writes the key-value data to the data area of ​​the file and records the offset and size S[offset+size] in the data area;

[0107] Hash calculation unit, calculates the hash value hash(key) of the key in the key-value data;

[0108] Bitmap unit, updates the bit value of the hashIDth bit corresponding to the hash value hash(key) in the bitmap;

[0109] And a hash index record unit, which records the hashID, key, S[offset+size] when writing key-value data as a hash index record and writes it in the file.

[0110] The file data storage and reading device applied to database storage in this embodiment, when performing data storage, the file data storage module updates the bit value on the corresponding bit of the bitmap according to the hash value hash(key) of the key in the key-value data, and uses it as the hash index record when the key-value data is written; in this way, when performing a key query, the data existence is judged through the bitmap area. If the data does not exist, it can be returned with only one comparison calculation overhead. If the data exists, it is directly indexed to the data according to the hash index record to obtain the key-value data. Therefore, the file data storage and reading device applied to database storage in this embodiment adopts a new file format through hash index when performing data storage, and the time complexity can reach O(1) when performing key value retrieval, which greatly improves the retrieval efficiency of key-value data.

[0111] As an optional implementation manner of this embodiment, in the file data storage and reading device for database storage described in this embodiment, the hash index recording unit records the hash index record in a file including:

[0112] The hash index record unit sorts the hash index records by hashID, writes the offset and size of all hashIDs into the hash area of ​​the file in a fixed-length manner, and records the offset and size H[offset+size] of the hash area;

[0113] The hash index record unit serializes the bitmap of the hash index record, writes it into the file as a bitmap area, and records the offset and size B[offset+size] of the bitmap area;

[0114] The hash index record unit stores the offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area as the footer area of ​​the file.

[0115] After the hash index record units of this embodiment are sorted by hashID, the offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area are stored as the footer area of ​​the file. When querying the key value, if the data existence in the bitmap area is judged as the key value, the data area can be gradually indexed to obtain the key-value data according to the offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area.

[0116] Since this embodiment performs hash calculation on key-value data, in principle, assuming that the hash algorithm is uniform, the hash values ​​hash(key) obtained by hash calculation of key-value data are different from each other. However, if there are two or more key-value data that obtain the same hash value hash(key) by hash calculation, a group of identical hashIDs will appear, which will affect the sorting of hashIDs. Therefore, as an optional implementation of this embodiment, the file data storage and reading device applied to database storage in this embodiment also includes:

[0117] The hash conflict processing unit packs the keys of the same hashID and the offset and size S[offset+size] corresponding to the keys in the data area into a conflict block according to the hash index record, and writes the blocks into the conflict area of ​​the file in sequence;

[0118] Save the offset and size C[offset+size] of the conflicting block of the key with the same hashID in the conflicting area, and write back C[offset+size] as the unique associated offset and size of the key with the same hashID.

[0119] The hash conflict processing unit of this embodiment packages the key-value data with the same hash value obtained through hash calculation into a conflict block, stores the conflict block in the conflict area of ​​the file, sorts the entire conflict block sharing the same hashID, and uses the offset and size C[offset+size] of the conflict block stored in the conflict area of ​​the file as the offset and size of the key-value data storage corresponding to the hashID.

[0120] In the file data storage and reading device applied to database storage of this embodiment, when performing file data storage, assuming that the hash algorithm is uniform, a large proportion of data will not fall into the conflict area. For the data that does not fall into the conflict area, only one io in the hash area is needed to obtain the offset and size H[offset+size] of the hash area of ​​the data, and one io in the data area is needed to obtain the data through the offset and size S[offset+size] of the data area. Even if the queried key falls into the conflict area, a binary search is performed on the key in the same conflict area, which only requires a very small io cost, and the overall average query times are constant.

[0121] As an optional implementation of this embodiment, the file data storage and reading device applied to database storage described in this embodiment includes a verification unit. In the process of writing the key-value data to the data area of ​​the file, the verification unit calculates the verification code of the key-value data, stores the verification code in the verification area of ​​the file, and records the offset and size J[offset+size] of the verification code of each key in the verification area; the offset and size J[offset+size] in the verification area are saved to the footer area of ​​the file.

[0122] The verification area of ​​this embodiment is used to perform data verification on the data area when opening a file. The footer area of ​​this embodiment can add custom data as a file format verification when reading, and the entire footer area is a fixed size.

[0123] In summary, the file data storage and reading device for database storage described in this embodiment forms a new file data structure when performing key-value data storage. The meaning and function of each partition of the specific file data structure are as follows:

[0124] Data area: stores compressed key-value data;

[0125] Verification area: Use cyclic verification code to verify the data area;

[0126] Conflict zone: keys with the same hash value and their offset and data size in the data zone will be placed in the conflict zone;

[0127] Hash area: a fixed-length area that stores the offset and size of the data area where the key is located or the conflict area;

[0128] Bitmap area: Use bitmap to store whether the bit corresponding to the hash of the key has a value;

[0129] Footer area: fixed-length storage data area, check area, conflict area, hash area, offset and size of the bitmap area.

[0130] At the same time, the file data storage and reading device for database storage described in this embodiment includes a file data reading module, the file data reading module includes a culture opening unit, and the file loading unit:

[0131] Open the file, read the content of the footer area according to the fixed size, and perform file format verification on the footer area;

[0132] Read the offset and size S[offset+size] of the data area in the footer area of ​​the file, the offset and size J[offset+size] of the check area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area;

[0133] According to the offset and size B[offset+size] of the bitmap area, the data in the bitmap area is deserialized and stored in the memory;

[0134] According to the offset and size H[offset+size] of the hash area, all hashIDs and corresponding offset+size are loaded into the memory.

[0135] Furthermore, in the file data storage and reading device for database storage described in this embodiment, the file data reading module includes a query unit, and the query unit:

[0136] Perform hash calculation on the query key to obtain hash(key) of the query key, and further obtain hashID=hash(key) of the query key;

[0137] Check whether the key exists in the bitmap area according to the hashID of the query key. If it does not exist, return an empty result and the query ends. If it does exist, proceed to the next step.

[0138] According to the rank of the query key’s hashID in the bitmap area, calculate the offset and size H[offset+size] of the hash area record;

[0139] According to the offset and size H[offset+size] of the hash area, the query data corresponding to the query key is read in the data area or the conflict area.

[0140] Furthermore, in the file data storage and reading device for database storage described in this embodiment, the query unit reads the query data corresponding to the query key in the data area or the conflict area according to the offset and size H[offset+size] of the hash area, including:

[0141] Determine whether it falls into the conflict area according to the offset and size H[offset+size] of the hash area;

[0142] If it is determined that it does not fall in the conflict area, the data is read directly from the data area according to the offset and size S[offset+size]. If it is determined that it falls in the conflict area, the conflict block is determined according to the offset and size C[offset+size] of the corresponding conflict area, and the offset and size S[offset+size] of the query key in the data area are replaced by binary search in the conflict block, and the data is read from the data area.

[0143] As an optional implementation of this embodiment, the file data storage and reading device for database storage described in this embodiment, the query unit calculates the offset and size H[offset+size] of the hash area record according to the sort rank of the hashID of the query key in the bitmap area, including:

[0144] The fixed length of the hash area when writing each H[offset+size] is width, and the writing is in order, so the offset and size H[offset+size] of the query key recorded in the hash area are calculated by rank*width.

[0145] As an optional implementation of this embodiment, the data verification area can adopt a variety of different verification algorithms; the hsah algorithm can select different specific numbers in the footer area and can select different values; the offset+size information of each area in the footer area can be stored separately in the meta area, and the footer area only stores the offset+size of the meta area.

[0146] This embodiment provides a storage medium storing a computer executable program. When the computer executable program is executed, any one of the file data storage and reading methods for database storage is implemented.

[0147] The storage medium described in the present embodiment may include a data signal propagated in a baseband or as a part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0148] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the file data storage and reading method applied to database storage.

[0149] The electronic device is presented in the form of a general-purpose computing device. The processor may be one or more than one and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0150] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0151] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0152] It should be understood that the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0153] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and can also be distributed and stored on the network, as long as it can enable the electronic device to execute the method according to the present invention.

[0154] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.

Claims

1. A file data storage and reading method applied to database storage, characterized in that: Includes file data storage, the file data storage includes: Write the key-value data to the data area of ​​the file, and record the offset and size S[offset+size] in the data area; Calculate the hash value hash(key) of the key in the key-value data, and update the bit value of the hashIDth bit corresponding to the hash value hash(key) in the bitmap; The hashID, key, S[offset+size] are used as hash index records when key-value data is written, and the hash index records are written into the file; The recording of the hash index record in a file includes: Sort the hash index records by hashID, write the offset and size of all hashIDs into the hash area of ​​the file in a fixed-length manner, and record the offset and size H[offset+size] of the hash area; Serialize the bitmap recorded by the hash index, write it into the file as the bitmap area, and record the offset and size B[offset+size] of the bitmap area; The offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area are stored as the footer area of ​​the file; Before sorting the hash index records by hashID, also include: According to the hash index record, the key with the same hashID and the offset and size S[offset+size] corresponding to the key in the data area are packaged into a conflict block, and written into the conflict area of ​​the file in sequence; Save the offset and size C[offset+size] of the conflicting block of the key with the same hashID in the conflicting area, and write back C[offset+size] as the unique associated offset and size of the key with the same hashID; In the process of writing the key-value data to the data area of ​​the file, the check code of the key-value data is calculated, the check code is stored in the check area of ​​the file, and the offset and size J[offset+size] of the check code of each key in the check area are recorded; the offset and size J[offset+size] in the check area are saved in the footer area of ​​the file; The file data storage and reading method applied to database storage includes file data reading, and the file data reading includes file opening, and the file opening process includes: Read the offset and size S[offset+size] of the data area in the footer area of ​​the file, the offset and size J[offset+size] of the check area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area; According to the offset and size B[offset+size] of the bitmap area, the data in the bitmap area is deserialized and stored in the memory; According to the offset and size H[offset+size] of the hash area, all hashIDs and corresponding offset+size are loaded into the memory; The file data reading includes a data reading process, and the data reading process includes: Perform hash calculation on the query key to obtain the query key hash(key), and further obtain the query key hashID = hash(key); Check whether the key exists in the bitmap area according to the hashID of the query key. If it does not exist, return an empty result and the query ends. If it does exist, proceed to the next step. According to the rank of the query key’s hashID in the bitmap area, calculate the offset and size H[offset+size] of the hash area record; According to the offset and size H[offset+size] of the hash area, the query data corresponding to the query key is read in the data area or the conflict area.

2. The file data storage and reading method for database storage according to claim 1, characterized in that: The step of reading the query data corresponding to the query key in the data area or the conflict area according to the offset and size H[offset+size] of the hash area includes: Determine whether it falls into the conflict area according to the offset and size H[offset+size] of the hash area; If it is determined that it does not fall in the conflict area, the data is read directly from the data area according to the offset and size S[offset+size]. If it is determined that it falls in the conflict area, the conflict block is determined according to the offset and size C[offset+size] of the corresponding conflict area, and the offset and size S[offset+size] of the query key in the data area are replaced by binary search in the conflict block, and the data is read from the data area.

3. The file data storage and reading method for database storage according to claim 1, characterized in that: The calculation of the offset and size H[offset+size] of the hash area record based on the rank of the query key hashID in the bitmap area includes: The fixed length of the hash area when writing each H[offset+size] is width, and the writing is in order, then the offset and size H[offset+size] of the query key recorded in the hash area are calculated by rank*width.

4. A file data storage and reading device for database storage, characterized in that: It includes a file data storage module, and the file data storage module includes: The data writing unit writes the key-value data to the data area of ​​the file and records the offset and size S[offset+size] in the data area; Hash calculation unit, calculates the hash value hash(key) of the key in the key-value data; Bitmap unit, updates the bit value of the hashIDth bit corresponding to the hash value hash(key) in the bitmap; and a hash index recording unit, which records the hashID, key, S[offset+size] when writing key-value data as a hash index record and writes it into the file; Recording the hash index record in a file includes: Sort the hash index records by hashID, write the offset and size of all hashIDs into the hash area of ​​the file in a fixed-length manner, and record the offset and size H[offset+size] of the hash area; Serialize the bitmap recorded by the hash index, write it into the file as the bitmap area, and record the offset and size B[offset+size] of the bitmap area; The offset and size S[offset+size] of the data area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area are stored as the footer area of ​​the file; Before sorting the hash index records by hashID, also include: According to the hash index record, the key with the same hashID and the offset and size S[offset+size] corresponding to the key in the data area are packaged into a conflict block, and written into the conflict area of ​​the file in sequence; Save the offset and size C[offset+size] of the conflicting block of the key with the same hashID in the conflicting area, and write back C[offset+size] as the unique associated offset and size of the key with the same hashID; In the process of writing the key-value data to the data area of ​​the file, the check code of the key-value data is calculated, the check code is stored in the check area of ​​the file, and the offset and size J[offset+size] of the check code of each key in the check area are recorded; the offset and size J[offset+size] in the check area are saved in the footer area of ​​the file; The file data storage and reading device applied to database storage includes a file data reading module, and the file data reading module includes a file opening unit. The file opening process includes: Open the file, read the content of the footer area according to the fixed size, and perform file format verification on the footer area; Read the offset and size S[offset+size] of the data area in the footer area of ​​the file, the offset and size J[offset+size] of the check area, the offset and size H[offset+size] of the hash area, and the offset and size B[offset+size] of the bitmap area; According to the offset and size B[offset+size] of the bitmap area, the data in the bitmap area is deserialized and stored in the memory; According to the offset and size H[offset+size] of the hash area, all hashIDs and corresponding offset+size are loaded into the memory; The file data reading module includes a query unit, wherein: Perform hash calculation on the query key to obtain the query key hash(key), and further obtain the query key hashID = hash(key); Check whether the key exists in the bitmap area according to the hashID of the query key. If it does not exist, return an empty result and the query ends. If it does exist, proceed to the next step. According to the rank of the query key’s hashID in the bitmap area, calculate the offset and size H[offset+size] of the hash area record; According to the offset and size H[offset+size] of the hash area, the query data corresponding to the query key is read in the data area or the conflict area.

5. A storage medium storing a computer executable program, characterized in that: When the computer executable program is executed, the file data storage and reading method applied to database storage as described in any one of claims 1 to 3 is implemented.

Images