A space recycling method of a key-value pair database

Abstract

The application discloses a space recycling method of a key-value pair database, comprising the following steps: traversing all key-value pair entries in a starting sector; judging whether the key-value pair entries are valid; if yes, creating a migrated key-value pair entry in other sectors except the starting sector, copying data of the key-value pair entry to the migrated key-value pair entry, updating an address field of a leaf node of an index string matching the key-value pair entry to a storage address of the migrated key-value pair entry, and marking a state of the key-value pair entry as a deletion state; if not, judging whether the key-value pair entries in the starting sector are traversed completely; if not, starting to traverse a next key-value pair entry in the starting sector; if yes, erasing the starting sector and taking a next sector of the starting sector as a new starting sector. The space recycling method provided by the application can recycle the space of all invalid key-value pair entries in the initial sector of the key-value pair database.

Description

[0001] This case is a divisional application based on the invention patent with application date of April 11, 2024, application number: 202410432123.6, and name: "A storage method for a key-value database". Technical Field

[0002] The present invention relates to the technical field of databases, and in particular to a space recovery method for a key-value pair database. Background Art

[0003] Flash storage is currently widely used in electronic devices such as mobile phones, tablets, and SSDs. Flash databases store logs in sectors within the flash database to achieve efficient log storage. Existing Flash storage methods primarily use structured records to store parameters, configuration information, dynamic data, and other data. This requires allocating one or more sectors to each module, storing different types of data in each sector. During operation, if a parameter in a sector changes, the data in that sector must first be read from RAM (random access memory), erased, and then all the parameters in RAM are written back to the sector after the change.

[0004] Existing database storage methods need to consider whether changing the order of parameters or increasing the number of parameters will affect previously stored parameter data. However, key-value databases use a dictionary tree to index key-value pair entries in the database. In practical applications, only as many sectors as possible need to be allocated to the database to provide a larger storage area. During the use of Flash, only key-value pair entries need to be added or deleted, without having to erase the entire sector. When some key-value pair entries in a sector are deleted during the operation of the key-value database, they occupy the sector's storage space, resulting in a waste of space resources in the key-value database, and the key-value database may fail to write data due to insufficient remaining space. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a space recovery method for a key-value pair database, so as to solve the problem of space resource waste in the existing key-value pair database.

[0006] In order to solve the above technical problems, the present invention adopts a technical solution: a space recovery method for a key-value pair database, comprising the steps of:

[0007] Traverse all key-value pair entries in the starting sector;

[0008] Determine whether the key-value pair entry is valid;

[0009] If so, a migration key-value pair entry is created in a sector other than the starting sector, the data of the key-value pair entry is copied to the migration key-value pair entry, the address field of the leaf node of the index string matching the key-value pair entry is updated to point to the storage address of the migration key-value pair entry, and the status of the key-value pair entry is marked as deleted;

[0010] If not, determine whether the traversal of the key-value pair entry of the starting sector is completed; if not, start traversing the next key-value pair entry in the starting sector; if so, erase the starting sector and use the next sector of the starting sector as the new starting sector.

[0011] Furthermore, the step before step S1 includes the following steps: determining whether the remaining space is less than a preset threshold; if so, entering step S1; if not, not reclaiming the starting sector.

[0012] Furthermore, the preset threshold includes a first preset threshold and a second preset threshold, and the first preset threshold is smaller than the second preset threshold; wherein the first preset threshold is used to determine whether to trigger the recovery of the starting sector when creating a new key-value pair entry, and the second preset threshold is used to determine whether to trigger the recovery of the starting sector value when idle.

[0013] Furthermore, the key-value pair entry includes: a second flag bit, a second magic number information, total data length information, a cyclic redundancy check code, a string length, a data length, a key name and key value data.

[0014] Furthermore, the step S1 includes the following steps:

[0015] Construct a dictionary tree, traverse the key-value pair entries, construct index strings according to the key names of each key-value pair entry, insert the index strings into the dictionary tree, and write the address field pointing to the storage address corresponding to the key-value pair entry into the leaf node of the index string.

[0016] Furthermore, a dictionary tree is constructed, key-value pair entries are traversed, index strings are constructed according to the key names of the key-value pair entries, the index strings are inserted into the dictionary tree, and an address field pointing to the storage address corresponding to the key-value pair entry is written into the leaf node of the index string, including:

[0017] S101: Determine whether the first sector is formatted; if not, proceed to step S102; if so, proceed to step S103;

[0018] S102: Format all sectors;

[0019] S103: Construct a dictionary tree;

[0020] S104: Starting from the first key-value pair entry of the first sector, traverse all key-value pair entries of all sectors in sequence;

[0021] S105: Determine whether the key-value pair entry is valid; if so, proceed to step S106; if not, proceed to step S107;

[0022] S106: construct an index string according to the key name of the key-value pair entry, insert the index string into the dictionary tree, and write an address field pointing to the storage address of the key-value pair entry into the leaf node of the index string;

[0023] S107: Determine whether all key-value pair entries of all sectors have been traversed; if so, start traversing the next untraversed key-value pair entry; if not, complete initialization.

[0024] Furthermore, formatting all sectors includes: writing corresponding sector information in the headers of all sectors respectively, and obtaining a starting sector; wherein the sector information includes first magic number information for checking whether the sector is formatted and a first flag bit for checking the status of the sector.

[0025] Furthermore, the nodes of the dictionary tree are stored in a variable array, so that the dictionary tree dynamically adjusts the length of the variable array according to the number of child nodes of the node.

[0026] The beneficial effect of the present invention is that the space recovery method of the key-value database provided by the present invention can reclaim the space of all invalid key-value pair entries in the initial sector of the key-value database. The space recovery method provided by the present invention can ensure the reliability of the key-value database and avoid the failure of Flash to write data due to insufficient remaining space in the key-value database. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic flow chart of a space recovery method for a key-value database according to one embodiment of the present invention;

[0028] Figure 2 This is a flowchart of a storage method for a key-value pair database according to one embodiment of the present invention;

[0029] Figure 3 for Figure 1 Schematic diagram of the process of step S1;

[0030] Figure 4 for Figure 1 Schematic diagram of the process of step S2;

[0031] Figure 5A schematic diagram of a process for reading key-value pair entries in the key-value pair database according to one embodiment of the present invention;

[0032] Figure 6 The figure schematically shows the structure of the dictionary tree in the key-value pair database of the present invention;

[0033] Figure 7 The figure schematically shows the principle of inserting a newly added index string into the dictionary tree in the key-value pair database according to the present invention. DETAILED DESCRIPTION

[0034] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following description is given in conjunction with the embodiments and accompanying drawings.

[0035] Please refer to Figures 1 to 7 The present invention provides a space recovery method for a key-value pair database, comprising the steps of:

[0036] Traverse all key-value pair entries in the starting sector;

[0037] Determine whether the key-value pair entry is valid;

[0038] If so, a migration key-value pair entry is created in a sector other than the starting sector, the data of the key-value pair entry is copied to the migration key-value pair entry, the address field of the leaf node of the index string matching the key-value pair entry is updated to point to the storage address of the migration key-value pair entry, and the status of the key-value pair entry is marked as deleted;

[0039] If not, determine whether the traversal of the key-value pair entry of the starting sector is completed; if not, start traversing the next key-value pair entry in the starting sector; if so, erase the starting sector and use the next sector of the starting sector as the new starting sector.

[0040] As can be seen from the above description, the beneficial effect of the present invention is that the space recovery method of the key-value database provided by the present invention can reclaim the space of all invalid key-value pair entries in the initial sector of the key-value database. The space recovery method provided by the present invention can ensure the reliability of the key-value database and avoid the failure of Flash to write data due to insufficient remaining space in the key-value database.

[0041] Furthermore, the step before step S1 includes the following steps: determining whether the remaining space is less than a preset threshold; if so, entering step S1; if not, not reclaiming the starting sector.

[0042] Furthermore, the preset threshold includes a first preset threshold and a second preset threshold, and the first preset threshold is smaller than the second preset threshold; wherein the first preset threshold is used to determine whether to trigger the recovery of the starting sector when creating a new key-value pair entry, and the second preset threshold is used to determine whether to trigger the recovery of the starting sector value when idle.

[0043] From the above description, it can be seen that the space recovery method includes two situations that can trigger the recovery of the starting sector: the first situation is to determine whether the remaining space is lower than the first preset threshold when writing a new key-value pair entry, which can ensure the reliability of the key-value pair database when writing data, and avoid the Flash from failing to write data due to insufficient remaining space in the key-value pair database; the second situation is as a user-optional space recovery function. When the user has extra resources and has high timeliness requirements for writing key-value pair entries, the idle space recovery function can be used to reclaim the starting sector by calling the user's idle thread, which will not affect other module tasks.

[0044] Furthermore, the key-value pair entry includes: a second flag bit, a second magic number information, total data length information, a cyclic redundancy check code, a string length, a data length, a key name and key value data.

[0045] From the above description, it can be seen that the second magic number information is used to determine whether the key-value pair entry is valid; the second flag is used to determine the status of the key-value pair entry; the total data length information includes the total data length in the key-value pair entry; the cyclic redundancy check code is used to check the key-value pair entry to ensure data accuracy; the string length is used to record the string length of the key name of the key-value pair entry; the data length is used to record the data length of the key-value data of the key-value pair entry; the key name is the name of the corresponding storage of the key-value pair; the key-value data is the parameter data corresponding to the storage of the key-value pair.

[0046] Furthermore, the step S1 includes the following steps:

[0047] Construct a dictionary tree, traverse the key-value pair entries, construct index strings according to the key names of each key-value pair entry, insert the index strings into the dictionary tree, and write the address field pointing to the storage address corresponding to the key-value pair entry into the leaf node of the index string.

[0048] From the above description, it can be seen that the space recovery method described in the present invention realizes space recovery based on a key-value pair database. Specifically, the key-value pair database uses a dictionary tree to index the key-value pair entries in the key-value pair database. In actual application, it is only necessary to allocate as many sectors as possible to the key-value pair database so that the key-value pair database has a larger storage area. In the process of using Flash, it is only necessary to add or delete key-value pair entries without erasing the entire sector. This storage method is not only beneficial to reducing the difficulty of maintaining the key-value pair database, but also can avoid the key-value pair database from repeatedly erasing sectors and reducing the service life of the sectors.

[0049] Furthermore, a dictionary tree is constructed, key-value pair entries are traversed, index strings are constructed according to the key names of the key-value pair entries, the index strings are inserted into the dictionary tree, and an address field pointing to the storage address corresponding to the key-value pair entry is written into the leaf node of the index string, including:

[0050] S101: Determine whether the first sector is formatted; if not, proceed to step S102; if so, proceed to step S103;

[0051] S102: Format all sectors;

[0052] S103: Construct a dictionary tree;

[0053] S104: Starting from the first key-value pair entry of the first sector, traverse all key-value pair entries of all sectors in sequence;

[0054] S105: Determine whether the key-value pair entry is valid; if so, proceed to step S106; if not, proceed to step S107;

[0055] S106: construct an index string according to the key name of the key-value pair entry, insert the index string into the dictionary tree, and write an address field pointing to the storage address of the key-value pair entry into the leaf node of the index string;

[0056] S107: Determine whether all key-value pair entries of all sectors have been traversed; if so, start traversing the next untraversed key-value pair entry; if not, complete initialization.

[0057] As can be seen from the above description, step S1 initializes the key-value database to create a trie based on the existing data in the database. During the initialization process, this storage method traverses all key-value pair entries in the database, constructs an index string based on the key name of the key-value pair entry, and inserts the index string into the trie. Therefore, the number of nodes in the trie is related to the number of existing key-value pairs and the length of the names in the key-value pairs.

[0058] In addition, this storage method indexes all key-value pair entries in all sectors by creating a dictionary tree, so that the key-value pair database does not need to be divided into sectors according to project or parameter type. All sectors are combined to form a larger storage space for the key-value pair database driver, and parameters can be added and deleted in this storage space at will. This eliminates the need for technicians to plan the allocation of storage sectors, which can reduce the difficulty of technicians in writing code and reduce their workload.

[0059] Furthermore, formatting all sectors includes: writing corresponding sector information in the headers of all sectors respectively, and obtaining a starting sector; wherein the sector information includes first magic number information for checking whether the sector is formatted and a first flag bit for checking the status of the sector.

[0060] As can be seen from the above description, by writing sector information at the header of a sector, the status of the sector can be checked based on the sector information. Specifically, identifying the first magic number information can check whether the sector has been formatted. The sector information also sets a first flag bit based on the characteristic that each byte can be written independently once when erasing the sector. The first flag bit is used to determine the current status of the sector, for example: whether there is free space in the sector to write a new key-value pair entry, or whether there is invalid data in the sector. The starting sector is the first non-empty sector, because Flash is used cyclically, which is easy to understand: when there are 10 sectors in Flash and each sector is numbered in sequence, the following situation may occur: the first and second sectors are empty, the third to ninth sectors are full, and the tenth sector has partial space written. In this case, the third sector is the starting sector.

[0061] Furthermore, the nodes of the dictionary tree are stored in a variable array, so that the dictionary tree dynamically adjusts the length of the variable array according to the number of child nodes of the node.

[0062] From the above description, it can be seen that when a child node is added to one of the nodes of the dictionary tree, the dictionary tree dynamically reallocates the node, which can effectively reduce the RAM memory space occupied by the dictionary tree.

[0063] Example 1

[0064] Please refer to Figures 1 to 7 In a first embodiment of the present invention, a method for storing a key-value pair database comprises the following steps:

[0065] S1: Build a dictionary tree, traverse the key-value pair entries, build index strings according to the key names of the key-value pair entries, insert the index strings into the dictionary tree, and write the address field pointing to the storage address of the corresponding key-value pair entry into the leaf node of the index string;

[0066] S2: query the dictionary tree based on the key name of the key-value pair to be written to see if there is a matching index string; if so, proceed to step S3; if not, proceed to step S4;

[0067] S3: creating a new key-value pair entry according to the key-value pair to be written, and updating the address field of the leaf node of the index string obtained by querying the key name of the key-value pair to be written to point to the storage address of the new key-value pair entry;

[0068] S4: Construct a new index string according to the key name of the key-value pair to be written, insert the new index string into the dictionary tree, create a new key-value pair entry according to the key-value pair to be written, and write the leaf node of the new index string into the address field pointing to the storage address of the new key-value pair entry.

[0069] In the first embodiment of the present invention, the root node of the dictionary tree does not store any characters. Except for the root node, each node only stores one character. The character string formed on the path from the root node to a leaf node is the corresponding index string, and the address field pointing to the storage address of the key-value pair entry corresponding to the index string is written in the leaf node.

[0070] Specifically, the code for constructing the dictionary tree in this embodiment is described as follows:

[0071] typedef struct TrieNode

[0072] {

[0073] bool isEnd; / / marked as the last character of the word

[0074] Char c; / / The character stored in this node

[0075] char cnt; / / How many child nodes exist after this node

[0076] uint32_t addr; / / This node carries the flash address (first address) of the matching key-value pair entry. If it is 0x00000000, it means that it does not carry

[0077] struct TrieNode*next[]; / / The pointer to the child node after this node

[0078] }*Trie,TrieNode;

[0079] In this first embodiment, a dictionary tree is constructed, an index string inserted into the dictionary tree is constructed according to the key name of the key-value pair entry, and the dictionary tree is also transformed; specifically, in the dictionary tree constructed in this first embodiment, the address field pointing to the storage address of the key-value pair entry that matches it is written in the "uint32_t addr" parameter of the leaf node of the index string.

[0080] In addition, in the first embodiment, the nodes of the dictionary tree are stored in a variable array, that is, the parameter structTrieNode*next[] is a variable array, so that the dictionary tree dynamically adjusts the length of the variable array according to the number of child nodes of the node.

[0081] The nodes of a traditional dictionary tree are stored in an immutable array, which requires pre-allocation of storage space for letters and numbers that may be used by pointers. The sub-node structure of a traditional dictionary tree is as follows:

[0082] typedef struct TrieNode

[0083] {

[0084] bool isEnd; / / marked as the last character of the word

[0085] char cnt; / / How many child nodes exist after this node

[0086] struct TrieNode*next

[62] / / The child node pointer after this node

[0087] }*Trie,TrieNode,

[0088] Easy to understand, with Figure 6 The dictionary tree structure shown is illustrated by the traditional dictionary tree storage node method: 26 uppercase letters, 26 lowercase letters and 10 numbers "0-9" may be applied, that is, the 62 character pointers required to store a child node each need to occupy 4 bytes, the parameter of the last character of the path needs to occupy 4 bytes and the parameter of the number of child nodes after this node needs to occupy 4 bytes, so a child node needs to occupy (62+1+1)*4=256 bytes, so the traditional dictionary tree storage is as follows: Figure 6 The dictionary tree structure shown takes up a total of 11*256=2816 bytes.

[0089] In the storage method provided in the first embodiment, the nodes of the dictionary tree are stored in a variable array, wherein there are 3 nodes carrying 2 child nodes and 8 nodes carrying 1 child node. In the node definition of the dictionary tree of the storage method, the last character parameter of the path occupies 1 byte, the character parameter stored in this node occupies 1 byte, the number of child nodes after this node occupies 1 byte, the address field parameter occupies 4 bytes, and a child node pointer parameter of this node occupies 4 bytes. Therefore, a node with two child nodes occupies 1+1+1+4+4*2=16 bytes, and a node with one child node occupies 1+1+1+4+4*1=11 bytes. In the storage method provided in the first embodiment, a variable array is used to store such Figure 6 The dictionary tree shown takes up a total of 16*3+11*8=136 bytes, which can save 95% of storage space compared to the traditional dictionary tree.

[0090] Since the nodes of the dictionary tree of the storage method described in the first embodiment are stored in a variable array, when a new index string is inserted into the dictionary tree, the existing nodes need to be reallocated.

[0091] Easy to understand, Figure 7 This shows the principle of inserting the index string hi into the dictionary tree when the dictionary tree stores the index string hell, as shown in Figure 7 As shown in (a), the index string "hell" has been stored in the dictionary tree. The index strings "hell" and "hi" both start with the character "h", but Figure 7 (a) The node storing the character "h" only contains one child node pointer. Figure 7 As shown in (b), before inserting the index string "hi", a new node of the character "h" including two child node pointers needs to be created. Figure 7 As shown in (c), a new "h" character node including two child node pointers replaces the original "h" character node including one child node pointer, wherein one of the node pointers of the new "h" character node points to the child node of the original "h" character, and the original "h" character node is deleted after being replaced to release space. Figure 7 As shown in (d), a child node is added to another node pointer of the new "h" character node, that is, the "i" character node is added to complete the creation of the index string "hi".

[0092] In embedded scenarios, because the types of parameters, configuration information, and dynamic data stored in Flash are relatively fixed, and each parameter, configuration information, dynamic data, and other data are stored separately in key-value pair entries, and the types of parameters, configuration information, dynamic data, and other data are distinguished by the key names of the key-value pair entries, and the key-value pair entries and their key names are also relatively fixed, so it is mainly necessary to create a dictionary tree and insert an index string into the dictionary tree during the initialization process. During the operation process, new key-value pair entries are rarely added and new index strings are rarely created and inserted into the dictionary tree.

[0093] Therefore, when applying the storage method described in the first embodiment, during operation, time is spent on comparing characters when reading, while very little time is spent on inserting new index strings. The search efficiency of this storage method will not be reduced too much, but it can effectively reduce RAM memory overhead.

[0094] like Figure 3 As shown, in the first embodiment, step S1 initializes the key-value pair database. Specifically, step S1 includes the following steps:

[0095] S101: Determine whether the first sector is formatted; if not, proceed to step S102; if so, proceed to step S103;

[0096] S102: Format all sectors;

[0097] S103: Construct a dictionary tree;

[0098] S104: Starting from the first key-value pair entry of the first sector, traverse all key-value pair entries of all sectors in sequence;

[0099] S105: Determine whether the key-value pair entry is valid; if so, proceed to step S106; if not, proceed to step S107;

[0100] S106: construct an index string according to the key name of the key-value pair entry, insert the index string into the dictionary tree, and write an address field pointing to the storage address of the key-value pair entry into the leaf node of the index string;

[0101] S107: Determine whether all key-value pair entries of all sectors have been traversed; if so, start traversing the next untraversed key-value pair entry; if not, complete initialization.

[0102] Specifically, in the first embodiment, formatting all sectors includes: writing corresponding sector information in the headers of all sectors respectively, and obtaining a starting sector; wherein the sector information includes first magic number information for checking whether the sector is formatted and a first flag bit for checking the status of the sector.

[0103] It is easy to understand that the main purpose of the initialization process of step S1 is to read out the existing data in the Flash to build a dictionary tree. The number of nodes in the dictionary tree is related to the number of existing key-value pair entries and the length of the name in the key-value pair. The starting sector will be recorded in the initialization process. The starting sector is the first sector in the Flash that has recorded data and is not empty. For example, there are ten sectors numbered in sequence for the Flash key-value pair database driver, among which the first sector to the second sector are empty, the third sector to the ninth sector are full, and the tenth sector has partial space written. At this time, the third sector is the starting sector. Among them, when the device is powered on for the first time or the status of the first sector has not been formatted, the sector formatting will be triggered, and the first sector will be used as the starting sector.

[0104] The sector information in the first embodiment may be specifically as follows:

[0105] typedef struct

[0106] {

[0107] uint8_t status_flag[MYFLASH_SECTOR_FLAG_MAX]; / / sector information first flag bit information

[0108] uint32_t magic; / / First magic number information ('M', 'Y', 'F', 'S') used to check whether the sector has been formatted

[0109] }myflash_sectorInfoType;

[0110] In detail, this embodiment 1 defines a first flag bit based on the characteristic that each byte can be written independently once during the whole block erase and write process of the sector. The key-value pair database can determine the current state of the sector (for example, whether there is free space for writing in the sector or whether there is invalid data) by reading the first flag bit, and thus make corresponding processing. Specifically, because each byte space can only be written once if the Flash is not erased, the sector reserves space for the first flag bit. When the sector is in its current described state, 0x00 is written to the space of the first flag bit. Specifically, the sector can include at least the following states:

[0111] typedef enum{

[0112] MYFLASH_SECTOR_FLAG_FORMAL = 0; / / Formatted but not used

[0113] MYFLASH_SECTOR_FLAG_USING; / / Already used, but not fully written

[0114] MYFLASH_SECTOR_FLAG_FULL; / / full

[0115] MYFLASH_SECTOR_FLAG_INVALID_DATA; / / The sector contains invalid data

[0116] MYFLASH_SECTOR_FLAG_MAX;

[0117] }myflash_sectorStatusFlagEnum;

[0118] This storage method sets a first flag bit in the sector, which can easily check and determine what state the sector is in. The first flag bit can be used as a basis for determining the sector status after each device restart.

[0119] Specifically, the database will determine the sector status during initialization, power-on restart, and storage space recovery. The database can quickly identify whether the sector has been initialized and whether the sector is valid based on the first magic number information, so the database is highly efficient.

[0120] Correspondingly, in the first embodiment, the key-value pair entry includes: a second flag bit, second magic number information, total data length information, a cyclic redundancy check code, string length, data length, key name, and key value data.

[0121] Specifically, in the first embodiment, the key-value pair entry may adopt the following structure:

[0122] typedef struct

[0123] {

[0124] uint8_t status_flag[MYFLASH ITEM_FLAG MAX]; / / The second flag bit of the key-value pair entry

[0125] uint32_t magic; / / The second magic number information ('M', 'Y', 'F', 'I') used to determine whether the key-value pair entry is valid

[0126] uint16_t len; / / Total length of data (header+name+'value)

[0127] uint16_t crc16; / / crc16(key_len+data_len+name+value)* /

[0128] uint16_t key_len; / / key length

[0129] uint16_t value_len; / / value length

[0130] }myflashitemInfoType;

[0131] In detail, the second magic number information is used to determine whether the key-value pair entry is valid. Specifically, during database initialization, power-on restart, space recovery, reading and writing, etc., the entry status can be checked according to the second magic number information to confirm whether the located key-value pair entry is valid, and then subsequent operations can be performed; the total data length information is used to indicate the total length of the key-value pair entry; the cyclic redundancy check code can specifically adopt a 16-bit CRC check algorithm to check the key-value pair entry to prevent data from being abnormally tampered with and ensure data accuracy and reliability; in addition, the key-value pair entry also includes a key name (key) and a key value (value) that are not displayed in the structure but actually exist, wherein the key name is the name of the storage corresponding to the key-value pair entry, and the key value is the data stored corresponding to the key-value pair entry.

[0132] The second flag of the key-value pair entry is used to determine the status of the key-value pair entry. When the key-value pair entry meets the current described status, 0x00 is written after the second flag to facilitate subsequent addition, deletion, modification, and removal. Specifically, the key-value pair entry can include at least the following statuses:

[0133] typedef enum{

[0134] MYFLASH_ITEM_FLAG_PRE_WRITE = 0; / / Entry has been partially written

[0135] MYFLASH_ITEM_FLAG_WRITE; / / Entry is completely written

[0136] MYFLASH_ITEM_FLAG_PRE_DELETE; / / Entry is ready to be deleted

[0137] MYFLASH_ITEM_FLAG_DELETED; / / Entry is deleted

[0138] MYFLASH_ITEM_FLAG ERROR; / / Entry abnormal (verification failed)

[0139] MYFLASH_ITEM_FLAG MAX;

[0140] }myflash_itemStatusFlagEnum;

[0141] Among them, the "entry has been partially written" state and the "entry is completely written" state of the key-value pair entry are to avoid conflicts when multiple threads access the same data; the "entry is ready to be deleted" state of the key-value pair entry is to avoid other threads preempting reading when updating data, so that the old key-value pair entry remains accessible before the new key-value pair entry is completed. Until the new key-value pair entry is completed and the new key-value pair entry becomes the "entry is completely written" state, the old key-value pair entry is set to the "entry is deleted" state.

[0142] Please refer to Figure 4 It is not difficult to see from the flowchart of the storage method provided in the first embodiment of the present invention that step S3 includes the following steps during the operation of writing a key-value pair entry in the key-value pair database:

[0143] The data of the key value of the key-value pair to be written is compared with the data of the key value of the matching key-value pair entry to determine whether the length and content of the two are consistent; if so, the leaf node of the index string obtained by querying the key name of the key-value pair to be written is not updated; if not, a new key-value pair entry is created based on the key-value pair to be written, and the address field of the leaf node of the index string obtained by querying the key name of the key-value pair to be written is updated to point to the storage address of the new key-value pair entry, and the status of the matching key-value pair entry is marked as deleted.

[0144] In an embedded scenario, when updating the parameters, configuration information, and dynamic data stored in the Flash, it is only necessary to create a new key-value pair entry in the key-value pair database, where the key name of the new key-value pair entry is the same as the key name of the matching key-value pair entry, the key value data of the new key-value pair entry is different from the key value data of the matching key-value pair entry, and the address field carried by the leaf byte of the index string in the dictionary tree is updated to point to the storage address of the new key-value pair entry, and the data update in the key-value pair database can be completed.

[0145] Therefore, the storage method provided in the first embodiment does not need to erase the entire sector when updating data. Instead, a new key-value pair entry is directly created and the address field in the leaf node of the index string in the dictionary tree is updated, so that when the key-value pair database reads data, the new key-value pair entry is directly read through the index string. This storage method can effectively reduce the number of times the sector is erased and written, thereby achieving the purpose of extending the service life of the sector.

[0146] In detail, in step S3, the status marking of the above-mentioned matching key-value pair entries includes the steps of: when the new key-value pair entry is not completed, marking the status of the matching key-value pair entry as the "entry ready to be deleted" state; when the new key-value pair entry is completed, marking the status of the matching key-value pair entry as the "entry completely deleted" state.

[0147] In the storage method provided in the first embodiment, in order to further improve the writing speed and make the key-value database more suitable for periodic dynamic data storage, the key-value database can also automatically reclaim storage space. Figure 1 As shown, the storage method further includes step S5: determining whether the remaining space is less than a preset threshold; if so, reclaiming the starting sector; if not, not reclaiming the starting sector.

[0148] The process of reclaiming the starting sector includes the following steps:

[0149] S501: Traverse all key-value pair entries in the starting sector;

[0150] S502: Determine whether the key-value pair entry is valid; if so, proceed to step S503; if not, proceed to step S504;

[0151] S503: Creating a migration key-value pair entry in a sector other than the starting sector, copying the data of the key-value pair entry to the migration key-value pair entry, updating the address field of the leaf node of the index string matching the key-value pair entry to point to the storage address of the migration key-value pair entry, and marking the status of the key-value pair entry as deleted;

[0152] S504: Determine whether the traversal of the key-value pair entry of the starting sector is completed; if not, start traversing the next key-value pair entry in the starting sector; if so, erase the starting sector and use the next sector of the starting sector as the new starting sector.

[0153] Among them, marking the status of the key-value pair entry as the deletion status includes: when the migration key-value pair entry is not completed, marking the status of the key-value pair entry as the "entry ready to be deleted" status; when the migration key-value pair entry is completed, marking the status of the key-value pair entry as the "entry completely deleted" status.

[0154] In the first embodiment of the present invention, the above-mentioned preset threshold includes a first preset threshold and a second preset threshold, and the first preset threshold is smaller than the second preset threshold; wherein, the first preset threshold is used to determine whether to trigger the recovery of the starting sector when creating a new key-value pair entry, and the second preset threshold is used to determine whether to trigger the recovery of the starting sector value when idle.

[0155] Therefore, the storage method provided in the first embodiment may include at least two situations in which space reclamation is triggered:

[0156] The first case is that space reclamation is triggered when a new key-value pair entry is created. Specifically, each time a new key-value pair entry is created, it is determined whether the remaining space in the key-value pair database is less than a first preset threshold; if so, the starting sector is reclaimed; if not, the starting sector is not reclaimed;

[0157] The second scenario triggers space reclamation when the key-value database is idle. The second preset threshold is greater than the first preset threshold, enabling automatic space reclamation when the key-value database is idle. Specifically, when the key-value database enters the idle thread, a determination is made as to whether the remaining space in the key-value database is less than the second preset threshold. If so, the starting sector is reclaimed; otherwise, the starting sector is not reclaimed.

[0158] Among them, triggering space reclamation when the key-value database is idle is a user-selectable space reclamation method. When the key-value database has high timeliness requirements and the key-value database has idle thread resources, the idle thread can be used to reclaim the key-value database space resources. Since this reclamation function is performed in the user's idle thread, it will not affect other module tasks. In addition, because multiple threads may create new key-value entries at the same time, in this case, if the previous thread has triggered the reclamation mechanism, the next thread can simply write the key-value pair, avoiding triggering the space reclamation function at the same time.

[0159] In the first embodiment, when the key-value pair database needs to perform a key-value pair entry read operation, the following steps S6 may be performed:

[0160] According to the key name of the key-value pair to be read requested by the application layer, check whether there is a matching index string in the dictionary tree; if so, read the matching key-value pair entry from the address field stored in the leaf node of the index string obtained by querying the key name of the key-value pair to be read, and send the matching key-value pair entry to the application layer; if not, return the information that there is no key-value pair entry to the application layer.

[0161] To sum up, the storage method described in the present invention uses a dictionary tree to index the key-value pair entries in the key-value pair database. In actual application, it is only necessary to allocate as many sectors as possible to the key-value pair database so that the key-value pair database has a larger storage area. In the process of using Flash, it is only necessary to add or delete the key-value pair entries without erasing the entire sector. This storage method is not only beneficial to reducing the difficulty of maintaining the key-value pair database, but also can avoid the key-value pair database from repeatedly erasing sectors and reducing the service life of the sectors.

[0162] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A space recovery method for a key-value pair database, comprising the following steps: S1: constructing a dictionary tree, traversing key-value pair entries, constructing index strings according to the key names of the key-value pair entries, inserting the index strings into the dictionary tree, and writing an address field pointing to the storage address of the corresponding key-value pair entry in the leaf node of the index string; S2: querying the dictionary tree based on the key name of the key-value pair to be written whether there is a matching index string; if so, proceeding to step S3; if not, proceeding to step S4; S3: creating a new key-value pair entry based on the key-value pair to be written, and updating the address field of the leaf node of the index string obtained by querying the key name of the key-value pair to be written to point to the storage address of the new key-value pair entry; S4: construct a new index string according to the key name of the key-value pair to be written, insert the new index string into the dictionary tree, create a new key-value pair entry according to the key-value pair to be written, and write the address field pointing to the storage address of the new key-value pair entry into the leaf node of the new index string; S5: determine whether the remaining space is less than a preset threshold; if so, reclaim the starting sector; if not, do not reclaim the starting sector; it is characterized in that Reclaiming the starting sector includes the following steps: Traverse all key-value pair entries in the starting sector; Determine whether the key-value pair entry is valid; If so, a migration key-value pair entry is created in a sector other than the starting sector, the data of the key-value pair entry is copied to the migration key-value pair entry, the address field of the leaf node of the index string matching the key-value pair entry is updated to point to the storage address of the migration key-value pair entry, and the status of the key-value pair entry is marked as deleted; If not, determine whether the traversal of the key-value pair entry of the starting sector is completed; if not, start traversing the next key-value pair entry in the starting sector; if so, erase the starting sector and use the next sector of the starting sector as the new starting sector; The marking of the state of the key-value pair entry as a deletion state includes: when the migration key-value pair entry is not completely created, marking the state of the key-value pair entry as an "entry ready for deletion" state; When the migration key-value pair entry is created, the state of the key-value pair entry is marked as "entry completely deleted".

2. The space recovery method according to claim 1, characterized in that: The preset threshold includes a first preset threshold and a second preset threshold, and the first preset threshold is less than the second preset threshold; wherein the first preset threshold is used to determine whether to trigger the recovery of the starting sector when creating a new key-value pair entry, and the second preset threshold is used to determine whether to trigger the recovery of the starting sector value when idle.

3. The space reclamation method according to claim 1, wherein: The key-value pair entry includes: a second flag bit, a second magic number information, total data length information, a cyclic redundancy check code, a string length, a data length, a key name and key value data.

4. The space reclamation method according to claim 1, wherein: Constructing a dictionary tree, traversing the key-value pair entries, constructing index strings according to the key names of the key-value pair entries, inserting the index strings into the dictionary tree, and writing an address field pointing to the storage address of the corresponding key-value pair entry into the leaf node of the index string, including: S101: Determine whether the first sector is formatted; if not, proceed to step S102; if so, proceed to step S103; S102: Format all sectors; S103: Construct a dictionary tree; S104: Starting from the first key-value pair entry of the first sector, traverse all key-value pair entries of all sectors in sequence; S105: Determine whether the key-value pair entry is valid; if so, proceed to step S106; if not, proceed to step S107; S106: construct an index string according to the key name of the key-value pair entry, insert the index string into the dictionary tree, and write an address field pointing to the storage address of the key-value pair entry into the leaf node of the index string; S107: Determine whether all key-value pair entries of all sectors have been traversed; if so, start traversing the next untraversed key-value pair entry; if not, complete initialization.

5. The space reclamation method according to claim 4, characterized in that: Formatting all sectors includes: writing corresponding sector information in the headers of all sectors respectively, and obtaining a starting sector; wherein the sector information includes first magic number information for checking whether the sector is formatted and a first flag bit for checking the status of the sector.

6. The space reclamation method according to claim 1, characterized in that: The nodes of the dictionary tree are stored in a variable array, so that the dictionary tree dynamically adjusts the length of the variable array according to the number of child nodes of the node.

Images