Database storage method

Abstract

The invention discloses a database storage method, which realizes high optimization of a storage space by finely classifying key value pairs and storing the key value pairs in different regions. The recovery frequency is dynamically adjusted, only the key value pairs marked as deleted are erased, storage fragments and the erasure frequency are reduced, the storage efficiency is improved, and the service life of storage equipment is prolonged. When the sectors are recycled, adjacent idle and invalid sectors are combined, so that the storage space utilization rate and the access efficiency are improved. When the new key value pairs are stored, the new key value pairs are preferentially distributed from the combined sector pool, the storage process is simplified, and the response speed is increased. The data access frequency and the sector erasure life are combined, the merged sectors are finely grouped, resource allocation is optimized, and the data access efficiency is improved. And meanwhile, a direct local modification technology is adopted, so that data migration and rewriting are reduced, and the updating operation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to a database storage method. Background Art

[0002] With the widespread popularity of electronic devices, Flash storage technology has been widely used in many fields such as mobile phones, tablets, and SSDs. In the architecture of Flash databases, log data is cleverly stored inside sectors to ensure fast reading and writing of data and long-term preservation. However, the current Flash storage solution presents a series of challenges when processing frequently changing data such as parameters, configuration information, and dynamic data.

[0003] The traditional Flash storage strategy relies on the structure recording mode, which requires that each functional module be divided into one or more sectors, and each type of data is stored in these sectors. Once a parameter in a sector changes, the data of the entire sector must first be loaded into the RAM (random access memory), then the original data in the sector must be erased, and finally the updated parameter data must be written back to the sector. This process is not only time-consuming and labor-intensive, but frequent erase and write operations will greatly shorten the service life of the sector.

[0004] Furthermore, in order to reduce the time of erasing sectors, existing storage methods merge sectors based on static rules or simple algorithms, which lack the ability to predict future data access patterns and storage device status, resulting in the merge operation may not be the optimal solution. In addition, the existing Flash storage solution is also quite cumbersome in terms of parameter management. When it is necessary to adjust the parameter order or add new parameters, technicians must carefully consider the potential impact of these changes on existing parameter data to prevent data from being damaged or lost. This complex parameter management mechanism undoubtedly increases the difficulty of database maintenance and operating costs.

[0005] Therefore, a new database storage strategy is urgently needed to optimize the performance of Flash storage methods, improve data access speed, extend the life of storage devices, and simplify database maintenance and management. Summary of the invention

[0006] This application provides a database storage method to address the above problems, a storage optimization strategy based on grouping and merging sectors, local modification optimization, and attribute inheritance, which can reduce storage fragmentation, improve data access speed, extend the life of storage devices, and optimize the allocation and utilization of storage resources.

[0007] The present application provides a database storage method, including: S101, constructing a dictionary tree structure according to the type and length of the database key-value pairs and storing the key-value pairs in categories; S102, when writing a new key-value pair, first search the index string in the dictionary tree according to the key name, if it exists, go to step S103a to process the duplicate key, if it does not exist, set a new index string and go to step S103b; S103a, creating a new key-value pair entry and storing it, and updating the index string leaf node address to point to the new entry; S103b, construct an index string, create a key-value pair entry, and update the leaf node address to point to the new entry; S104, judging whether the remaining space is less than a preset threshold according to the trigger condition; dynamically adjusting the recycling frequency according to the load of each storage area; if recycling is required, proceeding to step S105; otherwise, not recycling the starting sector; S105, traverse all key-value pair entries in the starting sector and determine the validity of the key-value pair entries; if valid, proceed to step S106a; if invalid, proceed to step S106b; S106a, selecting the last element in the sequence after the sector load is in descending order as the low storage area, allocating space for the migration key-value pair in the area and creating a migration entry, copying the original data to the new entry, updating the address of the leaf node of the relevant index string to point to the new entry, and marking the original entry as deleted; S106b, if the key-value pair entry of the starting sector has not been traversed, continue to traverse the next key-value pair entry; if the traversal is completed, the load is low and the invalid entries exceed 30% of the sector capacity, then enter step S107; if the load is high and the invalid entries do not exceed 30% of the sector capacity, delay sector recovery and return to the normal operation process to wait for the next evaluation; S107, only the space occupied by the key-value pair entries marked as deleted in the starting sector is erased to release the storage space, and the next sector of the starting sector is used as a new starting sector.

[0008] Preferably, the S101, classifying and storing key-value pairs includes: Classification criteria for key-value pairs Key-value pairs are classified into different types according to their purpose, nature and data type; For each type, the type format is further broken down; Based on the type classification, it is further subdivided according to the length of the key name and key value.

[0009] Preferably, the step S104, dynamically adjusting the recycling frequency, includes: The recycling frequency is inversely proportional to the load of the storage area. Dynamic adjustment of the recycling frequency means that when the load is lower than 30%, the recycling frequency is increased by 30% of the base frequency, and when the load is higher than 70%, the recycling frequency is reduced by 20% of the base frequency.

[0010] Preferably, the S107 includes: S201, initializing the storage device, establishing a sector mapping table, and setting a preset threshold for storage space usage; S202, when the remaining space of a sector is lower than a preset threshold, a sector recycling process is triggered; S203, traverse these sectors and their adjacent sectors, and mark sectors with empty status and invalid data; S204, merging adjacent sectors marked as empty and invalid data to form a continuous storage area, and updating the sector mapping table; S205, erasing invalid data in the merged sector, releasing storage space, and recording the merged sector information in the management information of the storage device; S206, when a new key-value pair needs to be stored, first check whether there is an available sector in the merged sector pool; S207, if there are available sectors, allocate sectors matching the key-value pair requirements from the merged sector pool for storage; S208, associating the newly allocated sector with the new key-value pair, and recording its storage information.

[0011] Preferably, the S201, sector mapping table, includes: The sector mapping table stores the location, status and data structure of each sector in the storage device and the storage classification rules; The set storage space usage lower limit. When the remaining space in the sector is lower than this value, the sector recovery process is triggered. The preset threshold is set to 10% of the total sector capacity.

[0012] Preferably, the S203 includes: Starting from the sector to be reclaimed, traverse its adjacent sectors; Check the status of each sector. If the status is empty or contains invalid data, mark the sector as mergeable. During the traversal process, the physical continuity of adjacent sectors is consistent.

[0013] Preferably, the state is empty, invalid data, including: The status of Empty represents sectors that are not used or have been completely erased, and Invalid Data represents sectors with deleted or corrupted data.

[0014] Preferably, the step S204, merging adjacent sectors, includes: Perform a merge operation according to the marked mergeable sectors; The merge operation involves the reallocation of physical addresses of multiple sectors; During the merging process, the storage classification rules of the original sectors are retained, and the merged sectors inherit the storage classification rules of the original sectors. Update the sector mapping table to record the merged sector information; The size of the sector after merging is equal to the sum of the sizes of all sectors before merging.

[0015] Preferably, the S204 further includes: S301, formulating a grouping standard for data access frequency and sector erase life; S302, grouping the merged sectors according to a set grouping standard during the sector merging process; S303, when receiving a key-value pair update request, locate the sector group to which the key-value pair belongs in the sector mapping table according to the identification information of the key-value pair, and further locate the specific sector and the position within the sector in the sector group; S304: Perform a local modification operation on the target key-value pair in the located sector.

[0016] Preferably, the local modification in S304 includes: After locating the sector where the target key-value pair is located, a read operation is performed on the sector to extract the target key-value pair; Only update and modify key-value pairs; After the modification is completed, the updated sector data is written back to the storage device.

[0017] One or more technical solutions provided in this application have at least the following technical effects or advantages: 1. By finely classifying key-value pairs by type and length, and implementing a regional storage strategy within the sectors of the Flash storage device, we have achieved highly optimized and refined management of storage space. This classified storage mechanism not only speeds up the data retrieval and access process, but also effectively avoids the problem of idle storage space caused by differences in data type or length. In addition, we dynamically adjust the recycling frequency according to the real-time load status of each storage area, abandoning the traditional inefficient practice of one-size-fits-all recycling of the entire database storage device, significantly improving the operating efficiency and resource utilization of storage devices, while also reducing operation and maintenance costs, providing strong support for the efficient operation of the database.

[0018] 2. When faced with the need to recycle storage space, this database storage method innovatively adopts a strategy of erasing only key-value pairs marked as deleted, avoiding the full erasure of the entire storage sector, thereby effectively reducing the generation of storage fragmentation and maintaining the continuity and high availability of storage space. This optimized storage and recycling mechanism not only greatly improves the read and write performance of the database, but also further extends the service life of the storage device, providing a more efficient, stable and reliable solution for database data storage and management, and meeting the dual needs of modern database systems for high performance and low cost.

[0019] 3. When reclaiming sectors that exceed the threshold range, adjacent sectors with "empty" or "invalid data" status are merged into a larger sector. This technical strategy significantly improves the utilization and access efficiency of storage space. By merging scattered idle and invalid sectors, storage fragmentation is reduced, allowing storage devices to maintain more continuous storage space. This not only optimizes the storage layout, but also reduces the addressing and allocation overhead when writing subsequent data, thereby speeding up data reading and writing and improving overall storage performance.

[0020] 4. When new key-value pairs need to be stored, sectors are allocated from the merged sector pool first. This strategy brings high efficiency and flexibility to storage allocation. As a collection of optimized storage space, the merged sector pool can quickly locate continuous storage space that meets the needs, reducing the search and decision-making time during storage allocation. At the same time, since the merged sectors have been pre-processed, their storage classification rules are clear and can be directly matched with new key-value pairs, further simplifying the storage process. This priority allocation mechanism not only improves the response speed of storage operations, but also makes the management of storage space more flexible and intelligent, and can better adapt to the storage needs of different application scenarios.

[0021] 5. By combining data access frequency and sector erase life, we innovatively achieve fine grouping of merged sectors. This method not only optimizes the allocation of storage resources, but also significantly improves data access efficiency and the service life of storage devices. Secondly, for the update of key-value pairs within the grouped merged sectors, we use direct local modification technology to reduce data movement and rewriting, thereby greatly improving the efficiency and speed of update operations.

[0022] 6. While inheriting the physical and logical properties of the original sectors, the grouped merged sectors effectively integrate sectors with different data access modes and sector lifespans, achieving efficient use of storage resources. This inheritance mechanism ensures the stability and compatibility of the storage database, avoiding performance degradation or data loss caused by inconsistent sector properties. In addition, local modification of key-value pairs directly within the grouped merged sectors further improves the response speed and processing power of the storage database, providing users with a smoother and more efficient data storage and access experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of a database storage method according to an embodiment of the present invention; Figure 2 This is a grouping standard for data access frequency and sector erase life in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] To facilitate the understanding of the present invention, the present application will be described more comprehensively below with reference to the relevant drawings; the drawings show preferred embodiments of the present invention, but the present invention can be implemented in many different forms and is not limited to the embodiments described herein; on the contrary, the purpose of providing these embodiments is to enable a more thorough and comprehensive understanding of the disclosed content of the present invention.

[0025] It should be noted that the terms “vertical”, “horizontal”, “up”, “down”, “left”, “right” and similar expressions used in this document are only for illustrative purposes and do not represent the only implementation method.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs; the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the term "and / or" used herein includes any and all combinations of one or more related listed items.

[0027] Figure 1 It is a flowchart of a database storage method according to an embodiment of the present invention.

[0028] like Figure 1 As shown, a database storage method includes the following steps: Embodiment 1: S101, constructing a dictionary tree structure according to the type and length of the database key-value pairs and storing the key-value pairs in a classified manner.

[0029] Specifically, the key-value pair entries in the database are traversed, and each key-value pair entry is inserted into the corresponding position of the dictionary tree according to the character order of the key name. During the insertion process, an index string (which can be the key name or its encoding / hash value) is constructed for each key-value pair, and it is used to uniquely identify the key-value pair in the dictionary tree; In the sectors of the Flash storage device, different storage areas are divided according to the type, length or other characteristics of the key-value pair, which are specially used to store key-value pairs of specific categories. The key-value pair entries are written to the corresponding storage area, and the storage format is ensured to be compact and efficient to maximize the storage space utilization. In the leaf node of the dictionary tree, the address field pointing to the storage address of the corresponding key-value pair entry, as well as the category and storage area information of the key-value pair are recorded.

[0030] Among them, the classification standard of key-value pairs can classify key-value pairs into different types according to the purpose, nature or data type of the key-value pairs; for example, key-value pairs can be divided into string type, integer type, floating point type, date and time type, etc.; each type can be further subdivided. For example, the string type can be subdivided according to the character set (such as ASCII, UTF-8) or special format (such as IP address, email address); on the basis of type classification, it can be further subdivided according to the length of the key name or key value. For example, string type key-value pairs can be divided into short key names (such as less than or equal to 8 characters), medium key names (such as 9-16 characters) and long key names (such as more than 16 characters) according to the length of the key name.

[0031] For example, suppose there are key-value pairs {"name":"Alice","age":30} and {"id":12345,"score":95.5}. According to the type and length, these two key-value pairs are classified as string type and numeric type respectively, and stored in the corresponding storage area. Then, the index strings "name" and "id" are constructed, inserted into the dictionary tree, and the storage address pointing to the corresponding key-value pair entry is written in the leaf node.

[0032] S102, when writing a new key-value pair, first search the index string in the dictionary tree according to the key name, if it exists, go to step S103a to process the duplicate key, if it does not exist, set a new index string and go to step S103b.

[0033] For example, suppose you want to write a key-value pair {"name":"Bob"}, first search for the index string "name" in the dictionary tree. If found, it means that there is already a key-value pair with "name" as the key, and go to S103a; if not found, it means that this is a new key name, and go to S103b.

[0034] S103a, construct a new index string according to the key name of the key-value pair to be written, create a key-value pair entry storage, and update the leaf node address field to point to the new entry.

[0035] Specifically, a new key-value pair entry is created according to the key-value pair to be written and stored in the corresponding storage area; the address field of the leaf node of the index string obtained by querying the key name of the key-value pair to be written is updated to point to the storage address of the new key-value pair entry.

[0036] Among them, if the key name of the key-value pair to be written is the same as the existing key name but the key value is different, choose to overwrite the original key value or keep the original key value and add a new version according to application requirements.

[0037] For example, continuing the above case, if the index string "name" is found, a new key-value pair entry {"name":"Bob"} is created, and the address field of the leaf node of the "name" index string in the dictionary tree is updated to the storage address of the new key-value pair entry.

[0038] S103b, construct an index string, create a key-value pair entry, and update the leaf node address to point to the new entry.

[0039] For example, if the index string "newKey" is not found, a new index string "newKey" is constructed and inserted into the dictionary tree. Then, a new key-value pair entry {"newKey":"value"} is created and written into the address field of the leaf node of the "newKey" index string.

[0040] Specifically, the data of the key value of the key-value pair to be written is compared with the data of the key value of the matching key-value pair entry to determine whether the length and content of the two are consistent; if so, the leaf node of the index string obtained by querying the key name of the key-value pair to be written is not updated; if not, a new key-value pair entry is created based on the key-value pair to be written, and the address field of the leaf node of the index string obtained by querying the key name of the key-value pair to be written is updated to point to the storage address of the new key-value pair entry, and the status of the matching key-value pair entry is marked as deleted.

[0041] S104, judging whether the remaining space is less than a preset threshold according to the triggering condition; dynamically adjusting the recycling frequency according to the load of each storage area; if recycling is required, proceeding to step S105; otherwise, not recycling the starting sector.

[0042] Among them, dynamically adjusting the recycling frequency means increasing the recycling frequency by 30% of the base frequency in low-load areas, and reducing the recycling frequency by 20% of the base frequency in high-load areas; the recycling frequency is inversely proportional to the load of the storage area, increasing the recycling frequency in low-load areas and reducing the recycling frequency in high-load areas; the recycling frequency is increased when the load is below 30% and reduced when the load is above 70%; the threshold setting can be modified according to the performance and space usage of the storage device.

[0043] S105, traverse all key-value pair entries in the starting sector and determine the validity of the key-value pair entries; if valid, proceed to step S106a; if invalid, proceed to step S106b.

[0044] Invalid represents a key-value pair entry that has been marked for deletion or expired.

[0045] For example, suppose there are 100 key-value pair entries in the starting sector, and these entries are traversed and checked for validity. It is found that 5 of them have been marked as deleted or expired, and the remaining 95 entries are valid.

[0046] S106a, select the last element in the sequence after the sector load is in descending order as the low storage area, allocate space for the migration key-value pair in this area and create a migration entry, copy the original data to the new entry, update the address of the leaf node of the relevant index string to point to the new entry, and mark the original entry as deleted.

[0047] For example, continuing the above case, for the 95 valid key-value pair entries, sort them in descending order according to the load, select the storage area with the last element in the descending sequence with a lower load, and create new migration entries for them. Copy the data of the original key-value pair entry to the new migration entry, and update the address field of the leaf node of the corresponding index string in the dictionary tree. Then, mark the state of the original key-value pair entry as deleted.

[0048] S106b, if the key-value pair entry of the starting sector has not been traversed, continue to traverse the next key-value pair entry; if the traversal is completed, the load is low and there are many invalid entries in the sector, enter step S107; if the load is high or there are few invalid entries, the sector recovery can be delayed, return to the normal operation process or wait for the next evaluation.

[0049] For example, continuing with the above case, if the key-value pair entries of the starting sector have been traversed and it is found that there are many invalid entries (more than 50%), enter S107 to recycle the sector; if there are fewer invalid entries or the load is high, delay the recycling and return to the normal operation process.

[0050] S107, only the space occupied by the key-value pair entries marked as deleted in the starting sector is erased to release the storage space, and the next sector of the starting sector is used as a new starting sector.

[0051] For example, continuing the above case, erase the space occupied by the 5 key-value pair entries marked as deleted in the starting sector and release the storage space. Then, use the next sector as the new starting sector for subsequent operations. If 85% of the current sector is recycled, you can choose to skip several sectors to find the next underused sector as the starting sector.

[0052] It should be noted that, when using the storage method of the database adopted by the present invention, it is not necessary to carefully plan the sector allocation according to the project attribution or parameter category of the key-value pair database. Instead, it adopts a more concise way, that is, uniformly allocates multiple sectors to the key-value pair database, and these sectors together constitute a complete and large-capacity storage area of ​​the database. In this area, the key-value pair entry is composed of a key name and a key value, wherein the key value has a high degree of flexibility and can store any project and any type of data.

[0053] When the device is first started or reset, the database will format all sectors to completely remove any invalid data that may exist. Subsequently, the key-value pair data will be written into the database in order, and the dictionary tree structure will be constructed. By traversing each key-value pair entry in the database, the database will generate index strings based on the key name and cleverly insert these strings into the dictionary tree. At the leaf node of the dictionary tree, the database will record the storage address of the corresponding key-value pair entry to form an accurate index.

[0054] When the key-value database using this storage method needs to update the data of a key-value pair entry, it does not need to perform tedious erase and write operations on the entire sector. Instead, it simply creates a new key-value pair entry and updates the address field in the corresponding index string leaf node to point to the new data storage location. This efficient update mechanism greatly improves the efficiency of the key-value database in processing parameter data changes, making the database operation faster and more flexible.

[0055] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: 1. By finely classifying key-value pairs by type and length, and implementing a regional storage strategy within the sectors of the Flash storage device, we have achieved highly optimized and refined management of storage space. This classified storage mechanism not only speeds up the data retrieval and access process, but also effectively avoids the problem of idle storage space caused by differences in data type or length. In addition, we dynamically adjust the recycling frequency according to the real-time load status of each storage area, abandoning the traditional inefficient practice of one-size-fits-all recycling of the entire database storage device, significantly improving the operating efficiency and resource utilization of storage devices, while also reducing operation and maintenance costs, providing strong support for the efficient operation of the database.

[0056] 2. When faced with the need to recycle storage space, this database storage method innovatively adopts a strategy of erasing only key-value pairs marked as deleted, avoiding the full erasure of the entire storage sector, thereby effectively reducing the generation of storage fragmentation and maintaining the continuity and high availability of storage space. This optimized storage and recycling mechanism not only greatly improves the read and write performance of the database, but also further extends the service life of the storage device, providing a more efficient, stable and reliable solution for database data storage and management, and meeting the dual needs of modern database systems for high performance and low cost.

[0057] Embodiment 2: The above-mentioned first embodiment improves the utilization of storage space, reduces fragmentation, extends the life of Flash storage devices, and improves overall storage performance through strategies such as key-value pair classification storage, regional load adjustment of recycling frequency, and small-range erasure. In order to further solve the problem of storage fragmentation and improve the utilization and performance of storage space, the first embodiment is now further improved by introducing a merged sector pool and new key-value pair storage. When sectors that exceed the threshold range are recycled, adjacent sectors with a status of "empty" or "invalid data" are merged into a larger sector. When a new key-value pair needs to be stored, sectors are preferentially allocated from the merged sector pool.

[0058] Now, further improvements are made on the basis of step S107 in the second embodiment, specifically: S201, initializing a storage device, establishing a sector mapping table, and setting a preset threshold for storage space usage.

[0059] The sector mapping table is used to store the data structure of the location, status (empty, valid data, invalid data) and other attributes (storage classification rules) of each sector in the device; The set lower limit of storage space usage. When the remaining space in a sector is lower than this value, the sector recycling process is triggered. The preset threshold is set to 10% of the total sector capacity. In actual applications, the threshold can be set according to the type and performance of the storage device.

[0060] Specifically, create a data structure that stores detailed information about each sector in the device; The mapping table includes the sector's physical location (such as start address, end address), current status (empty, valid data, invalid data), storage classification rules (such as data type, access rights, etc.), and other attributes that may be required; The preset threshold is used to determine whether a sector needs to be reclaimed. It is set to 10% of the total capacity of the sector, that is, when the remaining space of the sector is less than 10% of its total capacity, the sector reclaim process is triggered.

[0061] In actual applications, the threshold can be dynamically adjusted according to the storage device type (such as SSD, HDD), performance (such as read and write speed, IOPS) and application scenario (such as database, file storage).

[0062] S202: When the remaining space of a sector is lower than a preset threshold, a sector recycling process is triggered.

[0063] Specifically, the remaining space of each sector is checked in real time. When the remaining space of a sector is found to be lower than a preset threshold, the sector is marked as to be recycled and the sector recycling process is triggered.

[0064] S203, traverse these sectors and their adjacent sectors, and mark sectors that are empty and have invalid data.

[0065] Among them, the status of Empty represents unused or completely erased sectors, and Invalid Data represents sectors with deleted or damaged data.

[0066] Specifically, starting from the sector to be reclaimed, traverse its adjacent sectors; check the status of each sector, if the status is empty (unused or completely erased) or invalid data (deleted or damaged data), mark the sector as mergeable; during the traversal process, it is necessary to ensure the physical continuity of adjacent sectors for subsequent merging operations.

[0067] S204, merging adjacent sectors marked as empty and invalid data to form a continuous storage area, and updating the sector mapping table.

[0068] The merged sectors shall inherit the storage classification rules of the original sectors.

[0069] Specifically, a merge operation is performed according to the marked mergeable sectors; The merge operation may involve the reallocation of the physical addresses of multiple sectors to ensure that the merged sectors are continuous. During the merge process, the storage classification rules of the original sectors need to be retained so that the new sectors can inherit these rules; the sector mapping table is updated to record the merged sector information, including the new physical location, status, storage classification rules, etc. The size of the sector after merging is equal to the sum of the sizes of all sectors before merging.

[0070] S205, erasing invalid data in the merged sector, releasing storage space, and recording the merged sector information in the management information of the storage device.

[0071] Specifically, an erasing operation is performed on invalid data in the merged sector to release storage space; The erase operation may involve physical data clearing, such as the TRIM command in an SSD or head erase in an HDD; the merged sector information is recorded in the management information of the storage device (metadata, log) for subsequent management and use; the released storage space can be reallocated to new key-value pairs or used for other storage needs.

[0072] S206: When a new key-value pair needs to be stored, first check whether there is an available sector in the merged sector pool.

[0073] Specifically, when a new key-value pair needs to be stored, the database first checks the merged sector pool. The merged sector pool is a collection of merged sectors that is used to optimize the allocation of storage space; The database traverses the merged sector pool and searches for sectors with an "empty" status and a size that meets the requirements of the new key-value pair.

[0074] S207: If there are available sectors, sectors matching the key-value pair requirements are allocated from the merged sector pool for storage.

[0075] S208, associating the newly allocated sector with the new key-value pair, and recording its storage information.

[0076] For example, assume that there are three adjacent sectors A, B, and C in the existing sector S, of which A and C are empty and B contains invalid data. After executing S204, A, B, and C are merged into a larger sector D, and D inherits the storage classification rules of A and C (assuming it is "normal data"); Erase and merge invalid data in sector D (i.e., data in sector B) to release storage space. Record detailed information of sector D in the management information of the storage device; When a new key-value pair needs to be stored, first check whether there is an available sector in the merged sector pool; If there is an available sector (such as D), D sectors are allocated from the merged sector pool for storage based on the size and demand of the key-value pair; Associate the D sector with the new key-value pair and record its storage location, size and other detailed information.

[0077] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: 1. When reclaiming sectors that exceed the threshold range, adjacent sectors with "empty" or "invalid data" status are merged into a larger sector. This technical strategy significantly improves the utilization and access efficiency of storage space. By merging scattered idle and invalid sectors, storage fragmentation is reduced, allowing storage devices to maintain more continuous storage space. This not only optimizes the storage layout, but also reduces the addressing and allocation overhead when writing subsequent data, thereby speeding up data reading and writing and improving overall storage performance.

[0078] 2. When new key-value pairs need to be stored, sectors are allocated from the merged sector pool first. This strategy brings high efficiency and flexibility to storage allocation. As a collection of optimized storage space, the merged sector pool can quickly locate continuous storage space that meets the needs, reducing the search and decision-making time during storage allocation. At the same time, since the merged sectors have been pre-processed, their storage classification rules are clear and can be directly matched with new key-value pairs, further simplifying the storage process. This priority allocation mechanism not only improves the response speed of storage operations, but also makes the management of storage space more flexible and intelligent, and can better adapt to the storage needs of different application scenarios.

[0079] Embodiment three: The above-mentioned second embodiment forms a larger continuous storage space by merging adjacent empty or invalid data sectors, and preferentially allocates sectors from the merged sector pool to new key-value pairs to reduce storage fragmentation and improve the utilization of storage space. In order to solve the problems existing in the existing Flash storage method, by grouping the merged sectors, when the grouped merged sectors need to update a key-value pair, the key-value pair is locally modified directly in the grouped merged sectors to improve storage efficiency, reduce maintenance difficulty, and extend the service life of Flash. Now, further improvements are made on the basis of step S204 in the second embodiment, specifically: S301, formulate a grouping standard for data access frequency and sector erase life.

[0080] The grouping criteria can be found in Figure 2 Grouping criteria for data access frequency and sector erase life.

[0081] S302, grouping the merged sectors according to a set grouping standard during the sector merging process.

[0082] Wherein, a group identification field is added to the sector mapping table to record the group information to which each sector belongs; When performing a sector merging operation, the physical and logical attribute information of each sector before the merging is recorded; These attributes include sector size, physical layout, numbering and addressing methods, specific partitions, and verification mechanisms; Determine the rules for merging sectors to inherit physical and logical attributes based on the characteristics of the storage device and application requirements; applying corresponding physical and logical attributes according to the determined inheritance rules when forming new grouped merged sectors; The attribute information of the new sector is recorded in the sector mapping table.

[0083] S303: when receiving a key-value pair update request, locate the sector group in the sector mapping table according to the identification information of the key-value pair, and further locate the specific sector and the position in the sector in the sector group.

[0084] Specifically, the identification information of the key-value pair is used to search for the corresponding sector group in the sector mapping table. If the mapping table adopts a hash table structure, it can be directly located according to the hash value; if it adopts a tree structure, it is necessary to traverse the nodes to search; after locating the sector group, the specific sector and the position within the sector are further located according to the data structure within the sector.

[0085] S304: Perform a local modification operation on the target key-value pair in the located sector.

[0086] Specifically, after locating the sector where the target key-value pair is located, the sector is read to extract the target key-value pair. Then, only the key-value pair is updated or modified; after the modification is completed, the updated sector data is written back to the storage device. This involves reading and writing sector data, and the consistency and integrity of the data must be ensured.

[0087] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: 1. By combining data access frequency and sector erase life, we innovatively achieve fine grouping of merged sectors. This method not only optimizes the allocation of storage resources, but also significantly improves data access efficiency and the service life of storage devices. Secondly, for the update of key-value pairs within the grouped merged sectors, we use direct local modification technology to reduce data movement and rewriting, thereby greatly improving the efficiency and speed of update operations.

[0088] 2. While inheriting the physical and logical properties of the original sectors, the grouped merged sectors effectively integrate sectors with different data access modes and sector lifespans, achieving efficient use of storage resources. This inheritance mechanism ensures the stability and compatibility of the storage database, avoiding performance degradation or data loss caused by inconsistent sector properties. In addition, local modification of key-value pairs directly within the grouped merged sectors further improves the response speed and processing power of the storage database, providing users with a smoother and more efficient data storage and access experience.

[0089] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A database storage method, characterized in that: include: S101, constructing a dictionary tree structure according to the type and length of the database key-value pairs and storing the key-value pairs in categories; S102, when writing a new key-value pair, first search the index string in the dictionary tree according to the key name, if it exists, go to step S103a to process the duplicate key, if it does not exist, set a new index string and go to step S103b; S103a, creating a new key-value pair entry and storing it, and updating the index string leaf node address to point to the new entry; S103b, construct an index string, create a key-value pair entry, and update the leaf node address to point to the new entry; S104, judging whether the remaining space is less than a preset threshold according to the trigger condition; dynamically adjusting the recycling frequency according to the load of each storage area; if recycling is required, proceeding to step S105; otherwise, not recycling the starting sector; S105, traverse all key-value pair entries in the starting sector and determine the validity of the key-value pair entries; if valid, proceed to step S106a; if invalid, proceed to step S106b; S106a, selecting the last element in the sequence after the sector load is in descending order as the low storage area, allocating space for the migration key-value pair in the area and creating a migration entry, copying the original data to the new entry, updating the address of the leaf node of the relevant index string to point to the new entry, and marking the original entry as deleted; S106b, if the key-value pair entry of the starting sector has not been traversed, continue to traverse the next key-value pair entry; if the traversal is completed, the load is low and the invalid entries exceed 30% of the sector capacity, then enter step S107; if the load is high and the invalid entries do not exceed 30% of the sector capacity, delay sector recovery and return to the normal operation process to wait for the next evaluation; S107, only the space occupied by the key-value pair entries marked as deleted in the starting sector is erased to release the storage space, and the next sector of the starting sector is used as a new starting sector.

2. A database storage method as claimed in claim 1, characterized in that: The S101, classifying and storing key-value pairs includes: Classification criteria for key-value pairs Key-value pairs are classified into different types according to their purpose, nature and data type; For each type, the type format is further broken down; Based on the type classification, it is further subdivided according to the length of the key name and key value.

3. A database storage method as claimed in claim 1, characterized in that: The step S104, dynamically adjusting the recycling frequency, includes: The recycling frequency is inversely proportional to the load of the storage area. Dynamic adjustment of the recycling frequency means that when the load is lower than 30%, the recycling frequency is increased by 30% of the base frequency, and when the load is higher than 70%, the recycling frequency is reduced by 20% of the base frequency.

4. A database storage method as claimed in claim 1, characterized in that: The S107 includes: S201, initializing the storage device, establishing a sector mapping table, and setting a preset threshold for storage space usage; S202, when the remaining space of a sector is lower than a preset threshold, a sector recycling process is triggered; S203, traverse these sectors and their adjacent sectors, and mark sectors with empty status and invalid data; S204, merging adjacent sectors marked as empty and invalid data to form a continuous storage area, and updating the sector mapping table; S205, erasing invalid data in the merged sector, releasing storage space, and recording the merged sector information in the management information of the storage device; S206, when a new key-value pair needs to be stored, first check whether there is an available sector in the merged sector pool; S207, if there are available sectors, allocate sectors matching the key-value pair requirements from the merged sector pool for storage; S208, associating the newly allocated sector with the new key-value pair, and recording its storage information.

5. A database storage method as claimed in claim 4, characterized in that: The sector mapping table in S201 includes: The sector mapping table stores the location, status and data structure of each sector in the storage device and the storage classification rules; The set storage space usage lower limit. When the remaining space in the sector is lower than this value, the sector recovery process is triggered. The preset threshold is set to 10% of the total sector capacity.

6. A database storage method as claimed in claim 4, characterized in that: The S203 includes: Starting from the sector to be reclaimed, traverse its adjacent sectors; Check the status of each sector. If the status is empty or contains invalid data, mark the sector as mergeable. During the traversal process, the physical continuity of adjacent sectors is consistent.

7. A database storage method as claimed in claim 6, characterized in that: The state is empty, invalid data, including: The status of Empty represents sectors that are not used or have been completely erased, and Invalid Data represents sectors with deleted or corrupted data.

8. A database storage method as claimed in claim 4, characterized in that: The step S204, merging adjacent sectors, includes: Perform a merge operation according to the marked mergeable sectors; The merge operation involves the reallocation of physical addresses of multiple sectors; During the merging process, the storage classification rules of the original sectors are retained, and the merged sectors inherit the storage classification rules of the original sectors. Update the sector mapping table to record the merged sector information; The size of the sector after merging is equal to the sum of the sizes of all sectors before merging.

9. A database storage method as claimed in claim 4, characterized in that: The S204 further includes: S301, formulating a grouping standard for data access frequency and sector erase life; S302, grouping the merged sectors according to a set grouping standard during the sector merging process; S303, when receiving a key-value pair update request, locate the sector group to which the key-value pair belongs in the sector mapping table according to the identification information of the key-value pair, and further locate the specific sector and the position within the sector in the sector group; S304: Perform a local modification operation on the target key-value pair in the located sector.

10. A database storage method as claimed in claim 9, characterized in that: The S304 is partially modified, including: After locating the sector where the target key-value pair is located, a read operation is performed on the sector to extract the target key-value pair; Only update and modify key-value pairs; After the modification is completed, the updated sector data is written back to the storage device.

Images