A database storage method

Abstract

The present application discloses a database storage method, which achieves a high degree of optimization of storage space by finely classifying key-value pairs and storing them in different areas. The recycling frequency is dynamically adjusted, and only the key-value pairs marked for deletion are erased, which reduces storage fragmentation and the number of erasures, and improves storage efficiency and the life of storage devices. When sectors are recycled, adjacent idle and invalid sectors are merged to improve storage space utilization and access efficiency. When new key-value pairs are stored, they are preferentially allocated from the merged sector pool, which simplifies the storage process and improves response speed. Combining data access frequency and sector erase life, the merged sectors are finely grouped, resource allocation is optimized, and data access efficiency is improved. At the same time, the direct local modification technology is used to reduce data movement and rewriting, and improve the efficiency of update operations.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to a database storage method. Background Art

[0002] With the widespread adoption of electronic devices, Flash storage technology has been widely adopted in numerous fields, including mobile phones, tablets, and SSDs. In Flash database architectures, log data is cleverly stored within sectors to ensure fast data read and write speeds and long-term data preservation. However, current Flash storage solutions present a series of challenges when processing frequently changing data, such as parameters, configuration information, and dynamic data.

[0003] Traditional Flash storage strategies rely on a structured recording model, requiring each functional module to be divided into one or more sectors, with each type of data stored separately within these sectors. Whenever a parameter in a sector changes, the entire sector's data must first be loaded into RAM (random access memory), then the original data in the sector must be erased, and finally the updated parameter data must be written back to the sector. This process is not only time-consuming and labor-intensive, but the frequent erase and write operations significantly shorten the sector's lifespan.

[0004] Furthermore, to reduce the time it takes to erase and write sectors, existing storage methods merge sectors based on static rules or simple algorithms. These methods lack the ability to predict future data access patterns and storage device status, resulting in potentially suboptimal merge operations. Furthermore, existing Flash storage solutions are cumbersome in terms of parameter management. When adjusting parameter order or adding new parameters, technicians must carefully consider the potential impact of these changes on existing parameter data to prevent data corruption or loss. This complex parameter management mechanism undoubtedly increases the difficulty and operational costs of database maintenance.

[0005] Therefore, a new database storage strategy is urgently needed to optimize the performance of Flash storage methods, improve data access speed, extend the life of storage devices, and simplify database maintenance and management. Summary of the Invention

[0006] This application addresses the above-mentioned issues by providing a database storage method, a storage optimization strategy based on grouping and merging sectors, local modification optimization, and attribute inheritance. This reduces storage fragmentation, improves data access speed, extends the life of storage devices, and optimizes the allocation and utilization of storage resources.

[0007] This application provides a database storage method, including:

[0008] S101, constructing a dictionary tree structure according to the type and length of the database key-value pairs and storing the key-value pairs in a classified manner;

[0009] S102, when writing a new key-value pair, first search the dictionary tree for the index string based on the key name. If it exists, proceed to step S103a to process the duplicate key. If it does not exist, set a new index string and proceed to step S103b;

[0010] S103a, creating a new key-value pair entry and storing it, and updating the index string leaf node address to point to the new entry;

[0011] S103b, construct an index string, create a key-value pair entry, and update the leaf node address to point to the new entry;

[0012] S104: Determine whether the remaining space is less than a preset threshold based on the trigger condition; dynamically adjust the recycling frequency based on the load of each storage area; if recycling is required, proceed to step S105; otherwise, do not recycle the starting sector;

[0013] S105, traverse all key-value pair entries in the starting sector and determine the validity of the key-value pair entries; if valid, proceed to step S106a; if invalid, proceed to step S106b;

[0014] S106a, select the last element in the sequence after the sector load is in descending order as the low storage area, allocate space for the migration key-value pair in the area and create a migration entry, copy the original data to the new entry, update the address of the leaf node of the relevant index string to point to the new entry, and mark the original entry as deleted;

[0015] S106b: If the key-value pair entries of the starting sector have not been traversed, the next key-value pair entry is traversed. If the traversal is complete, the load is low, and the number of invalid entries exceeds 30% of the sector capacity, the process proceeds to step S107. If the load is high and the number of invalid entries does not exceed 30% of the sector capacity, the sector reclamation is delayed and the process returns to normal operation to wait for the next evaluation.

[0016] S107 , only the space occupied by the key-value pair entries marked as deleted in the starting sector is erased to release the storage space, and the sector next to the starting sector is used as a new starting sector.

[0017] Preferably, the S101, classifying and storing key-value pairs includes:

[0018] The classification standard of key-value pairs divides key-value pairs into different types according to their purpose, nature and data type;

[0019] For each type, the type format is further broken down;

[0020] Based on the type classification, it is further subdivided according to the length of the key name and key value.

[0021] Preferably, the step S104 of dynamically adjusting the recycling frequency includes:

[0022] The recycling frequency is inversely proportional to the load of the storage area. Dynamic adjustment of the recycling frequency means that when the load is below 30%, the recycling frequency is increased by 30% of the base frequency. When the load is above 70%, the recycling frequency is reduced by 20% of the base frequency.

[0023] Preferably, the step S107 includes:

[0024] S201, initializing the storage device, establishing a sector mapping table, and setting a preset threshold for storage space usage;

[0025] S202, when the remaining space of a sector is lower than a preset threshold, triggering a sector recycling process;

[0026] S203, traverse these sectors and their adjacent sectors, and mark sectors with empty status and invalid data;

[0027] S204, merging adjacent sectors marked as empty and invalid data to form a continuous storage area, and updating the sector mapping table;

[0028] S205, erasing invalid data in the merged sector to release storage space, and recording the merged sector information in the management information of the storage device;

[0029] S206, when a new key-value pair needs to be stored, first check whether there is an available sector in the merged sector pool;

[0030] S207, if there are available sectors, allocate sectors matching the key-value pair requirements from the merged sector pool for storage;

[0031] S208: Associate the newly allocated sector with the new key-value pair and record its storage information.

[0032] Preferably, the sector mapping table in S201 includes:

[0033] The sector mapping table stores the location, status, and storage classification rules of each sector in the device;

[0034] The set lower limit of storage space usage. When the remaining space in a sector falls below this value, the sector reclamation process is triggered. The preset threshold is set to 10% of the total sector capacity.

[0035] Preferably, the step S203 includes:

[0036] Starting from the sector to be reclaimed, traverse its adjacent sectors;

[0037] Check the status of each sector. If the status is empty or invalid data, mark the sector as mergeable.

[0038] During the traversal process, the physical continuity of adjacent sectors is consistent.

[0039] Preferably, the state is empty or invalid data, including:

[0040] The status of Empty represents unused or completely erased sectors, and Invalid Data represents sectors with deleted or corrupted data.

[0041] Preferably, the step S204 of merging adjacent sectors includes:

[0042] Perform the merge operation based on the marked mergeable sectors;

[0043] The merge operation involves the reallocation of physical addresses of multiple sectors;

[0044] During the merging process, the storage classification rules of the original sector are retained, and the merged sector inherits the storage classification rules of the original sector

[0045] Update the sector mapping table to record the merged sector information;

[0046] The size of the sector after merging is equal to the sum of the sizes of all sectors before merging.

[0047] Preferably, the step S204 further includes:

[0048] S301, establishing a grouping standard for data access frequency and sector erase life;

[0049] S302, grouping the merged sectors according to a predetermined grouping standard during the sector merging process;

[0050] S303, upon receiving a key-value pair update request, locating the sector group in the sector mapping table according to the identification information of the key-value pair, and further locating the specific sector and position within the sector within the sector group;

[0051] S304: Perform a local modification operation on the target key-value pair in the located sector.

[0052] Preferably, the local modification in S304 includes:

[0053] After locating the sector where the target key-value pair is located, perform a read operation on the sector to extract the target key-value pair;

[0054] Only update and modify key-value pairs;

[0055] After the modification is completed, the updated sector data is written back to the storage device.

[0056] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0057] 1. By categorizing key-value pairs by type and length, and implementing a zoned storage strategy within the sectors of Flash storage devices, we achieve highly optimized and refined management of storage space. This categorized storage mechanism not only accelerates data retrieval and access, but also effectively avoids the problem of idle storage space caused by differences in data type or length. Furthermore, we dynamically adjust the recycling frequency based on the real-time load of each storage area, abandoning the traditional inefficient one-size-fits-all recycling approach for the entire database storage device. This significantly improves the operating efficiency and resource utilization of storage devices, while also reducing operation and maintenance costs, providing strong support for the efficient operation of the database.

[0058] 2. When faced with the need to reclaim storage space, this database storage method innovatively adopts a strategy of erasing only key-value pairs marked as deleted, avoiding a full wipe of entire storage sectors. This effectively reduces storage fragmentation and maintains the continuity and high availability of storage space. This optimized storage and reclamation mechanism not only significantly improves database read and write performance but also further extends the lifespan of storage devices, providing a more efficient, stable, and reliable solution for database data storage and management, meeting the dual requirements of high performance and low cost for modern database systems.

[0059] 3. When reclaiming sectors that exceed the threshold, adjacent sectors with "empty" or "invalid data" status are merged into a larger sector. This technical strategy significantly improves storage space utilization and access efficiency. By merging scattered free and invalid sectors, storage fragmentation is reduced, allowing storage devices to maintain more contiguous storage space. This not only optimizes storage layout but also reduces addressing and allocation overhead during subsequent data writes, thereby accelerating data read and write speeds and improving overall storage performance.

[0060] 4. When new key-value pairs require storage, sectors are allocated preferentially from the merged sector pool. This strategy improves efficiency and flexibility in storage allocation. As a collection of optimized storage space, the merged sector pool can quickly locate contiguous storage space that meets requirements, reducing search and decision-making time during storage allocation. Furthermore, because the merged sectors have been pre-processed, their storage classification rules are clear and can be directly matched with new key-value pairs, further simplifying the storage process. This priority allocation mechanism not only improves the responsiveness of storage operations, but also makes storage space management more flexible and intelligent, better adapting to the storage needs of different application scenarios.

[0061] 5. By combining data access frequency with sector erase lifespan, we innovatively achieve fine-grained grouping of merged sectors. This approach not only optimizes storage resource allocation but also significantly improves data access efficiency and storage device lifespan. Secondly, for key-value pair updates within grouped merged sectors, we employ direct local modification technology, reducing data movement and rewriting, thereby significantly improving the efficiency and speed of update operations.

[0062] 6. Grouped sector merging inherits the physical and logical attributes of the original sectors while effectively integrating sectors with different data access patterns and lifespans, achieving efficient utilization of storage resources. This inheritance mechanism ensures the stability and compatibility of the storage database, avoiding performance degradation or data loss caused by inconsistent sector attributes. In addition, local modifications of key-value pairs directly within the grouped and merged sectors further improve the storage database's responsiveness and processing capabilities, providing users with a smoother and more efficient data storage and access experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of a database storage method according to an embodiment of the present invention;

[0064] Figure 2 This is the grouping standard for data access frequency and sector erase life in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] To facilitate understanding of the present invention, the present application will be described more comprehensively below with reference to the relevant drawings; the drawings show preferred embodiments of the present invention, but the present invention can be implemented in many different forms and is not limited to the embodiments described herein; on the contrary, the purpose of providing these embodiments is to enable a more thorough and comprehensive understanding of the disclosed content of the present invention.

[0066] It should be noted that the terms “vertical”, “horizontal”, “up”, “down”, “left”, “right” and similar expressions used in this document are for illustrative purposes only and do not represent the only implementation method.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains; the terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention; the term "and / or" used herein includes any and all combinations of one or more of the associated listed items.

[0068] Figure 1 The figure is a flowchart of a database storage method according to an embodiment of the present invention.

[0069] like Figure 1 As shown, a database storage method includes the following steps:

[0070] Example 1:

[0071] S101, constructing a dictionary tree structure according to the type and length of the database key-value pairs and storing the key-value pairs in a classified manner.

[0072] Specifically, it traverses the key-value pair entries in the database and inserts each key-value pair entry into the corresponding position of the dictionary tree according to the character order of the key name. During the insertion process, an index string (which can be the key name or its encoding / hash value) is constructed for each key-value pair and used to uniquely identify the key-value pair in the dictionary tree;

[0073] Within the sectors of a Flash storage device, different storage areas are allocated based on the type, length, or other characteristics of key-value pairs, specifically for storing specific categories of key-value pairs. Key-value pair entries are written to the corresponding storage areas, ensuring a compact and efficient storage format to maximize storage space utilization. The leaf nodes of the dictionary tree contain an address field pointing to the storage address of the corresponding key-value pair entry, as well as information about the category and storage area to which the key-value pair belongs.

[0074] The classification criteria for key-value pairs can be divided into different types based on their purpose, nature, or data type. For example, key-value pairs can be divided into string types, integer types, floating-point types, date and time types, etc. Each type can be further subdivided. For example, string types can be subdivided based on character sets (such as ASCII, UTF-8) or special formats (such as IP addresses, email addresses). Based on the type classification, they can be further subdivided based on the length of the key name or key value. For example, string key-value pairs can be divided into categories such as short key names (such as less than or equal to 8 characters), medium key names (such as 9-16 characters), and long key names (such as greater than 16 characters) based on the length of the key name.

[0075] For example, consider the key-value pairs {"name":"Alice","age":30} and {"id":12345,"score":95.5}. Based on their type and length, these two key-value pairs are classified as strings and numbers, respectively, and stored in the corresponding storage areas. Then, the index strings "name" and "id" are constructed, inserted into the trie, and the storage address pointing to the corresponding key-value pair entry is written into the leaf node.

[0076] S102, when writing a new key-value pair, first search the dictionary tree for the index string based on the key name. If it exists, go to step S103a to process the duplicate key. If it does not exist, set a new index string and go to step S103b.

[0077] For example, suppose you want to write the key-value pair {"name":"Bob"}, first search the dictionary tree for the index string "name". If it is found, it means that a key-value pair with "name" as the key already exists, and the process proceeds to S103a; if it is not found, it means that this is a new key name, and the process proceeds to S103b.

[0078] S103a, construct a new index string according to the key name of the key-value pair to be written, create a key-value pair entry storage, and update the leaf node address field to point to the new entry.

[0079] Specifically, a new key-value pair entry is created based on the key-value pair to be written and stored in the corresponding storage area; the address field of the leaf node of the index string obtained by querying the key name of the key-value pair to be written is updated to point to the storage address of the new key-value pair entry.

[0080] Among them, if the key name of the key-value pair to be written is the same as the existing key name but the key value is different, choose to overwrite the original key value or retain the original key value and add a new version based on application requirements.

[0081] For example, continuing the above case, if the index string "name" is found, a new key-value pair entry {"name":"Bob"} is created, and the address field of the leaf node of the "name" index string in the dictionary tree is updated to the storage address of the new key-value pair entry.

[0082] S103b, construct an index string, create a key-value pair entry, and update the leaf node address to point to the new entry.

[0083] For example, if the index string "newKey" is not found, a new index string "newKey" is constructed and inserted into the dictionary tree. Then, a new key-value pair entry {"newKey":"value"} is created and written to the address field of the leaf node of the "newKey" index string.

[0084] Specifically, the data of the key value of the key-value pair to be written is compared with the data of the key value of the matching key-value pair entry to determine whether the length and content of the two are consistent; if so, the leaf node of the index string obtained by querying the key name of the key-value pair to be written is not updated; if not, a new key-value pair entry is created based on the key-value pair to be written, and the address field of the leaf node of the index string obtained by querying the key name of the key-value pair to be written is updated to point to the storage address of the new key-value pair entry, and the status of the matching key-value pair entry is marked as deleted.

[0085] S104, judging whether the remaining space is less than a preset threshold according to the triggering condition; dynamically adjusting the recycling frequency according to the load of each storage area; if recycling is required, proceeding to step S105; otherwise, not recycling the starting sector.

[0086] Dynamically adjusting the recycling frequency means increasing the recycling frequency by 30% of the base frequency in low-load areas and reducing it by 20% of the base frequency in high-load areas. The recycling frequency is inversely proportional to the load of the storage area, increasing it in low-load areas and decreasing it in high-load areas. The recycling frequency increases when the load is below 30% and decreases when the load is above 70%. The threshold setting can be modified based on the performance and space usage of the storage device.

[0087] S105, traverse all key-value pair entries in the starting sector and determine the validity of the key-value pair entries; if valid, proceed to step S106a; if invalid, proceed to step S106b.

[0088] Invalid represents a key-value pair entry that has been marked for deletion or expired.

[0089] For example, suppose there are 100 key-value pair entries in the starting sector. When these entries are traversed and their validity is checked, it is found that 5 of them have been marked as deleted or expired, and the remaining 95 entries are valid.

[0090] S106a, select the last element in the sequence after the sector load is in descending order as the low storage area, allocate space for the migration key-value pair in this area and create a migration entry, copy the original data to the new entry, update the address of the leaf node of the relevant index string to point to the new entry, and mark the original entry as deleted.

[0091] For example, continuing with the previous example, for the 95 valid key-value pairs, sort them in descending order by load. Select the storage area with the lowest load for the last element in the descending order and create a new migration entry for it. Copy the data of the original key-value pair entry to the new migration entry, and update the address field of the leaf node of the corresponding index string in the dictionary trie. Then, mark the status of the original key-value pair entry as deleted.

[0092] S106b, if the key-value pair entry of the starting sector has not been traversed, continue to traverse the next key-value pair entry; if the traversal is completed, the load is low and there are many invalid entries in the sector, go to step S107; if the load is high or there are few invalid entries, the sector recovery can be delayed, return to the normal operation process or wait for the next evaluation.

[0093] For example, continuing with the above case, if the key-value pair entries of the starting sector have been traversed and it is found that there are many invalid entries (more than 50%), then enter S107 for sector recovery; if there are fewer invalid entries or the load is high, then delay recovery and return to the normal operation process.

[0094] S107 , only the space occupied by the key-value pair entries marked as deleted in the starting sector is erased to release the storage space, and the sector next to the starting sector is used as a new starting sector.

[0095] For example, continuing with the above example, the space occupied by the five key-value pairs marked for deletion in the starting sector is erased to free up storage space. The next sector is then used as the new starting sector for subsequent operations. If 85% of the current sector is reclaimed, a few sectors can be skipped to find the next underused sector as the starting sector.

[0096] It should be noted that the database storage method employed in the present invention eliminates the need for meticulously planning sector allocation based on the project attribution or parameter classification of the key-value database. Instead, a more concise approach is employed, uniformly allocating multiple sectors to the key-value database. These sectors collectively constitute a complete and large-capacity storage area for the database. Within this area, key-value pair entries consist of a key name and a key value. The key value is highly flexible, allowing for storage of any project and any type of data.

[0097] When the device is first powered on or reset, the database formats all sectors to completely remove any invalid data. Key-value pairs are then written to the database in an orderly fashion, and a trie structure is constructed. By traversing each key-value pair entry in the database, the database generates index strings based on the key names and cleverly inserts these strings into the trie. At the leaf nodes of the trie, the database records the storage address of the corresponding key-value pair entry, forming a precise index.

[0098] When a key-value database using this storage method needs to update the data of a key-value pair entry, it does not need to perform a tedious erase and write operation on the entire sector. Instead, it simply creates a new key-value pair entry and updates the address field in the corresponding index string leaf node to point to the new data storage location. This efficient update mechanism greatly improves the efficiency of the key-value database when handling parameter data changes, making the database operation faster and more flexible.

[0099] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages:

[0100] 1. By categorizing key-value pairs by type and length, and implementing a zoned storage strategy within the sectors of Flash storage devices, we achieve highly optimized and refined management of storage space. This categorized storage mechanism not only accelerates data retrieval and access, but also effectively avoids the problem of idle storage space caused by differences in data type or length. Furthermore, we dynamically adjust the recycling frequency based on the real-time load of each storage area, abandoning the traditional inefficient one-size-fits-all recycling approach for the entire database storage device. This significantly improves the operating efficiency and resource utilization of storage devices, while also reducing operation and maintenance costs, providing strong support for the efficient operation of the database.

[0101] 2. When faced with the need to reclaim storage space, this database storage method innovatively adopts a strategy of erasing only key-value pairs marked as deleted, avoiding a full wipe of entire storage sectors. This effectively reduces storage fragmentation and maintains the continuity and high availability of storage space. This optimized storage and reclamation mechanism not only significantly improves database read and write performance but also further extends the lifespan of storage devices, providing a more efficient, stable, and reliable solution for database data storage and management, meeting the dual requirements of high performance and low cost for modern database systems.

[0102] Example 2:

[0103] The above-mentioned embodiment 1 improves the utilization of storage space, reduces fragmentation, extends the life of Flash storage devices, and enhances overall storage performance through strategies such as key-value pair classification storage, regional load adjustment of recycling frequency, and small-scale erasure. In order to further solve the problem of storage fragmentation and improve the utilization and performance of storage space, embodiment 1 is now further improved by introducing a merged sector pool and new key-value pair storage. When sectors exceeding the threshold range are recycled, adjacent sectors with a status of "empty" or "invalid data" are merged into a larger sector. When a new key-value pair needs to be stored, sectors are preferentially allocated from the merged sector pool.

[0104] Now, based on step S107 of the second embodiment, further improvements are made as follows:

[0105] S201, initializing a storage device, creating a sector mapping table, and setting a preset threshold for storage space usage.

[0106] The sector mapping table is a data structure used to store the location, status (empty, valid data, invalid data) and other attributes (storage classification rules) of each sector in the storage device;

[0107] The set lower limit of storage space usage. When the remaining space in a sector falls below this value, the sector reclamation process is triggered. The preset threshold is set to 10% of the total sector capacity. In actual application, the threshold can be set based on the storage device type and performance.

[0108] Specifically, create a data structure that stores detailed information about each sector in the device;

[0109] The mapping table includes the sector's physical location (such as start address, end address), current status (empty, valid data, invalid data), storage classification rules (such as data type, access rights, etc.), and other attributes that may be required;

[0110] The preset threshold is used to determine whether a sector needs to be reclaimed. It is set to 10% of the total sector capacity. That is, when the remaining space in a sector is less than 10% of its total capacity, the sector reclamation process is triggered.

[0111] In actual applications, the threshold can be dynamically adjusted based on the storage device type (such as SSD, HDD), performance (such as read and write speed, IOPS), and application scenario (such as database, file storage).

[0112] S202: When the remaining space of a sector is lower than a preset threshold, a sector recycling process is triggered.

[0113] Specifically, the remaining space of each sector is checked in real time. When the remaining space of a sector is found to be lower than a preset threshold, the sector is marked as pending for recycling and the sector recycling process is triggered.

[0114] S203: traverse these sectors and their adjacent sectors, and mark sectors that are empty or have invalid data.

[0115] Among them, the status of Empty represents unused or completely erased sectors, and Invalid Data represents sectors with deleted or damaged data.

[0116] Specifically, starting from the sector to be reclaimed, traverse its adjacent sectors; check the status of each sector, and if the status is empty (unused or completely erased) or invalid data (deleted or damaged data), mark the sector as mergeable; during the traversal process, it is necessary to ensure the physical continuity of adjacent sectors for subsequent merging operations.

[0117] S204: Merge adjacent sectors marked as empty and invalid data to form a continuous storage area, and update the sector mapping table.

[0118] The merged sectors must inherit the storage classification rules of the original sectors.

[0119] Specifically, a merge operation is performed based on the marked mergeable sectors;

[0120] The merge operation may involve the reallocation of the physical addresses of multiple sectors to ensure that the merged sectors are continuous. During the merge process, the storage classification rules of the original sectors need to be retained so that the new sectors can inherit these rules; the sector mapping table is updated to record the merged sector information, including the new physical location, status, storage classification rules, etc.

[0121] The size of the sector after merging is equal to the sum of the sizes of all sectors before merging.

[0122] S205 , erasing invalid data in the merged sector to release storage space, and recording the merged sector information in the management information of the storage device.

[0123] Specifically, an erasing operation is performed on invalid data in the merged sectors to release storage space;

[0124] The erase operation may involve physical data clearing, such as the TRIM command in an SSD or head erase in an HDD; the merged sector information is recorded in the storage device's management information (metadata, log) for subsequent management and use; the released storage space can be reallocated to new key-value pairs or used for other storage needs.

[0125] S206: When a new key-value pair needs to be stored, first check whether there is an available sector in the merged sector pool.

[0126] Specifically, when a new key-value pair needs to be stored, the database first checks the merged sector pool. The merged sector pool is a collection of merged sectors that is used to optimize storage space allocation.

[0127] The database traverses the merged sector pool and searches for sectors with an "empty" status and a size that meets the requirements of the new key-value pair.

[0128] S207: If there are available sectors, sectors matching the key-value pair requirements are allocated from the merged sector pool for storage.

[0129] S208: Associate the newly allocated sector with the new key-value pair and record its storage information.

[0130] For example, assume that there are three adjacent sectors A, B, and C in sector S. A and C are empty, and B contains invalid data. After executing S204, A, B, and C are merged into a larger sector D, which inherits the storage classification rules of A and C (assuming it is "normal data").

[0131] Erase and merge invalid data in sector D (i.e., data in sector B) to free up storage space. Record detailed information about sector D in the storage device's management information.

[0132] When a new key-value pair needs to be stored, first check whether there is an available sector in the merged sector pool;

[0133] If there is an available sector (such as D), D sectors are allocated from the merged sector pool for storage based on the size and demand of the key-value pair;

[0134] Associate the D sector with the new key-value pair and record its storage location, size and other detailed information.

[0135] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages:

[0136] 1. When reclaiming sectors that exceed the threshold, adjacent sectors with "empty" or "invalid data" status are merged into a larger sector. This technical strategy significantly improves storage space utilization and access efficiency. By merging scattered free and invalid sectors, storage fragmentation is reduced, allowing storage devices to maintain more contiguous storage space. This not only optimizes storage layout but also reduces addressing and allocation overhead during subsequent data writes, thereby accelerating data read and write speeds and improving overall storage performance.

[0137] 2. When new key-value pairs require storage, sectors are allocated preferentially from the merged sector pool. This strategy improves efficiency and flexibility in storage allocation. As a collection of optimized storage space, the merged sector pool can quickly locate contiguous storage space that meets requirements, reducing search and decision-making time during storage allocation. Furthermore, because the merged sectors have been pre-processed, their storage classification rules are clear and can be directly matched with new key-value pairs, further simplifying the storage process. This priority allocation mechanism not only improves the responsiveness of storage operations, but also makes storage space management more flexible and intelligent, better adapting to the storage needs of different application scenarios.

[0138] Example 3:

[0139] The above-mentioned second embodiment forms a larger continuous storage space by merging adjacent empty or invalid data sectors, and preferentially allocates sectors from the merged sector pool to new key-value pairs, thereby reducing storage fragmentation and improving storage space utilization. In order to solve the problems existing in the existing Flash storage method, by grouping the merged sectors, when a key-value pair needs to be updated in the grouped merged sectors, the key-value pair is locally modified directly in the grouped merged sectors, thereby improving storage efficiency, reducing maintenance difficulty, and extending the service life of the Flash.

[0140] Now, based on step S204 of the second embodiment, further improvements are made, specifically:

[0141] S301, establishing a grouping standard for data access frequency and sector erase lifespan.

[0142] The grouping criteria can be referred to Figure 2 Grouping criteria for data access frequency and sector erase life.

[0143] S302: Grouping the merged sectors according to a predetermined grouping standard during the sector merging process.

[0144] Among them, a group identification field is added to the sector mapping table to record the group information to which each sector belongs;

[0145] When performing a sector merging operation, the physical and logical attribute information of each sector before the merging is recorded;

[0146] These attributes include sector size, physical layout, numbering and addressing methods, specific partitions, and checksum mechanisms;

[0147] Determine the rules for inheriting physical and logical properties of merged sectors based on the characteristics of the storage device and application requirements;

[0148] Applying corresponding physical and logical attributes according to the determined inheritance rules when forming new grouped merged sectors;

[0149] Record the attribute information of the new sector in the sector mapping table.

[0150] S303: When a key-value pair update request is received, the sector group to which the key-value pair belongs is located in the sector mapping table according to the identification information of the key-value pair, and the specific sector and the position within the sector are further located within the sector group.

[0151] Specifically, the corresponding sector group is searched in the sector mapping table using the key-value pair identification information. If the mapping table uses a hash table structure, the hash value can be used to locate the corresponding sector group. If it uses a tree structure, the node search must be traversed. After locating the sector group, the specific sector and position within the sector can be further located based on the data structure within the sector.

[0152] S304: Perform a local modification operation on the target key-value pair in the located sector.

[0153] Specifically, after locating the sector containing the target key-value pair, a read operation is performed on that sector to extract the target key-value pair. Then, only the key-value pair is updated or modified. Once the modification is complete, the updated sector data is written back to the storage device. This involves both reading and writing sector data, ensuring data consistency and integrity.

[0154] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages:

[0155] 1. By combining data access frequency with sector erase lifespan, we innovatively achieve fine-grained grouping of merged sectors. This approach not only optimizes storage resource allocation but also significantly improves data access efficiency and storage device lifespan. Secondly, for key-value pair updates within grouped merged sectors, we employ direct local modification technology, reducing data movement and rewriting, thereby significantly improving the efficiency and speed of update operations.

[0156] 2. Grouped sector merging inherits the physical and logical attributes of the original sectors while effectively integrating sectors with different data access patterns and lifespans, achieving efficient utilization of storage resources. This inheritance mechanism ensures the stability and compatibility of the storage database, avoiding performance degradation or data loss caused by inconsistent sector attributes. In addition, local modifications of key-value pairs directly within the grouped and merged sectors further improve the storage database's responsiveness and processing capabilities, providing users with a smoother and more efficient data storage and access experience.

[0157] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A database storage method, characterized in that: include: S101, constructing a dictionary tree structure according to the type and length of the database key-value pairs and storing the key-value pairs in a classified manner; S102, when writing a new key-value pair, first search the dictionary tree for the index string based on the key name. If it exists, proceed to step S103a to process the duplicate key. If it does not exist, set a new index string and proceed to step S103b; S103a, creating a new key-value pair entry and storing it, and updating the index string leaf node address to point to the new entry; S103b, construct an index string, create a key-value pair entry, and update the leaf node address to point to the new entry; S104: Determine whether the remaining space is less than a preset threshold based on the trigger condition; dynamically adjust the recycling frequency based on the load of each storage area; if recycling is required, proceed to step S105; otherwise, do not recycle the starting sector; S105, traverse all key-value pair entries in the starting sector and determine the validity of the key-value pair entries; if valid, proceed to step S106a; if invalid, proceed to step S106b; S106a, select the last element in the sequence after the sector load is in descending order as the low storage area, allocate space for the migration key-value pair in the area and create a migration entry, copy the original data to the new entry, update the address of the leaf node of the relevant index string to point to the new entry, and mark the original entry as deleted; S106b: If the key-value pair entries of the starting sector have not been traversed, the next key-value pair entry is traversed. If the traversal is complete, the load is low, and the number of invalid entries exceeds 30% of the sector capacity, the process proceeds to step S107. If the load is high and the number of invalid entries does not exceed 30% of the sector capacity, the sector reclamation is delayed and the process returns to normal operation to wait for the next evaluation. S107, erasing only the space occupied by the key-value pair entries marked as deleted in the starting sector, releasing the storage space, and using the sector next to the starting sector as the new starting sector; S201, initializing the storage device, establishing a sector mapping table, and setting a preset threshold for storage space usage; S202, when the remaining space of a sector is lower than a preset threshold, triggering a sector recycling process; S203, traverse these sectors and their adjacent sectors, and mark sectors with empty status and invalid data; S204, merging adjacent sectors marked as empty and invalid data to form a continuous storage area, and updating the sector mapping table; S205, erasing invalid data in the merged sector to release storage space, and recording the merged sector information in the management information of the storage device; S206, when a new key-value pair needs to be stored, first check whether there is an available sector in the merged sector pool; S207, if there are available sectors, allocate sectors matching the key-value pair requirements from the merged sector pool for storage; S208: Associate the newly allocated sector with the new key-value pair and record its storage information.

2. A database storage method according to claim 1, characterized in that: The step S101 of classifying and storing key-value pairs includes: The classification standard of key-value pairs divides key-value pairs into different types according to their purpose, nature and data type; For each type, the type format is further broken down; Based on the type classification, it is further subdivided according to the length of the key name and key value.

3. A database storage method according to claim 1, characterized in that: The step S104, dynamically adjusting the recycling frequency, includes: The recycling frequency is inversely proportional to the load of the storage area. Dynamic adjustment of the recycling frequency means that when the load is below 30%, the recycling frequency is increased by 30% of the base frequency. When the load is above 70%, the recycling frequency is reduced by 20% of the base frequency.

4. A database storage method according to claim 1, characterized in that: The sector mapping table in step S201 includes: The sector mapping table stores the location, status, and storage classification rules of each sector in the device; The set lower limit of storage space usage. When the remaining space in a sector falls below this value, the sector reclamation process is triggered. The preset threshold is set to 10% of the total sector capacity.

5. A database storage method according to claim 1, characterized in that: The S203 includes: Starting from the sector to be reclaimed, traverse its adjacent sectors; Check the status of each sector. If the status is empty or invalid data, mark the sector as mergeable. During the traversal process, the physical continuity of adjacent sectors is consistent.

6. A database storage method as claimed in claim 5, characterized in that: The state is empty and invalid data, including: The status of Empty represents unused or completely erased sectors, and Invalid Data represents sectors with deleted or corrupted data.

7. A database storage method according to claim 1, characterized in that: The step S204 of merging adjacent sectors includes: Perform the merge operation based on the marked mergeable sectors; The merge operation involves the reallocation of physical addresses of multiple sectors; During the merging process, the storage classification rules of the original sector are retained, and the merged sector inherits the storage classification rules of the original sector Update the sector mapping table to record the merged sector information; The size of the sector after merging is equal to the sum of the sizes of all sectors before merging.

8. A database storage method according to claim 1, characterized in that: The S204 further includes: S301, establishing a grouping standard for data access frequency and sector erase life; S302, grouping the merged sectors according to a predetermined grouping standard during the sector merging process; S303, upon receiving a key-value pair update request, locating the sector group in the sector mapping table according to the identification information of the key-value pair, and further locating the specific sector and position within the sector within the sector group; S304: Perform a local modification operation on the target key-value pair in the located sector.

9. A database storage method as claimed in claim 8, characterized in that: The S304 is partially modified to include: After locating the sector where the target key-value pair is located, perform a read operation on the sector to extract the target key-value pair; Only update and modify key-value pairs; After the modification is completed, the updated sector data is written back to the storage device.

Images