A data storage method, device, equipment and readable storage medium
By dynamically managing the number of memory tables and optimizing the index structure in the buffer pool, the problem of low data writing efficiency in the LSM structure is solved, achieving efficient data writing and query response.
Patent Information
- Application Number
- CN202610299772.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-12
- Publication Date
- 2026-06-12
AI Technical Summary
The existing Log Merge Tree (LSM) structure suffers from low query efficiency, unreasonable data merging, high index maintenance costs, and rigid memory configuration, resulting in low data writing efficiency, especially since data in active memory tables cannot be solidified before the memory tables are flushed to disk.
By dynamically managing the number of second memory tables in the buffer pool, periodically obtaining system status, adjusting the number of memory tables based on the backlog deviation, generating second memory tables and flushing them to the storage device, and optimizing the query process by combining hash buckets and B+ tree index structures.
It improves the efficiency of data writing, ensures uninterrupted data writing, reduces index maintenance costs, and improves query response speed.
Smart Images
Figure CN122195353A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data storage technology, and in particular to a data storage method, apparatus, device, and readable storage medium. Background Technology
[0002] The Log Merge Tree (LSM) structure effectively alleviates write performance bottlenecks by writing data to memory first and then merging it in batches to disk, and is widely used in data storage. However, this structure currently suffers from several drawbacks. First, it suffers from low query efficiency: during metadata queries, the lack of effective indexing mechanisms and query optimization strategies necessitates traversing layer by layer from memory to disk, resulting in long query response times and failing to meet the demands for fast queries. Second, data merging is often inefficient: existing data merging strategies are simplistic, frequently triggering merging based solely on data size, which can easily lead to merge storms, consuming significant system resources and making it difficult to guarantee data consistency and integrity. Third, index maintenance costs are high: traditional index structures become complex to maintain during frequent data updates and merging, leading to a significant drop in index performance and impacting overall query efficiency. Furthermore, memory configuration is relatively rigid; data in active memory tables cannot be solidified until the data in the solidified memory tables is flushed to disk, affecting data write efficiency. Summary of the Invention
[0003] This application provides a data storage method, apparatus, device, and readable storage medium, which at least solves the problem that data in an active memory table cannot be persisted before the data in the persisted memory table is flushed to disk.
[0004] Firstly, this application provides a data storage method, including: Obtain the data write request and determine the metadata to be written corresponding to the data write request based on the data write request; The system status is acquired at each control cycle, the backlog deviation of data in the buffer pool is determined based on the system status, the number of second memory tables is determined based on the backlog deviation, and the number of second memory tables is updated. The metadata to be written is written to the first memory table. A second memory table is generated based on the storage status of the first memory table and stored in the buffer pool. The first memory table is the active memory table that stores the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. Flush the second memory table stored in the buffer pool to the storage device.
[0005] Secondly, this application also provides a data storage device, comprising: The data extraction module is used to obtain data write requests and determine the metadata to be written corresponding to the data write requests. The quantity update module is used to obtain the system status at each interval control cycle, determine the backlog deviation of data in the buffer pool based on the system status, determine the quantity of the second memory table based on the backlog deviation, and update the quantity of the second memory table. The data writing module is used to write the metadata to be written into the first memory table, generate a second memory table according to the storage status of the first memory table, and save the second memory table in the buffer pool. The first memory table is the active memory table that saves the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. The data flushing module is used to flush the second memory table stored in the buffer pool to the storage device.
[0006] Thirdly, this application also provides a computer device, including a memory, a processor, and a data storage program stored in the memory and executable on the processor. When the processor executes the data storage program, it implements the data storage method described in the first aspect, including: Obtain the data write request and determine the metadata to be written corresponding to the data write request based on the data write request; The system status is acquired at each control cycle, the backlog deviation of data in the buffer pool is determined based on the system status, the number of second memory tables is determined based on the backlog deviation, and the number of second memory tables is updated. The metadata to be written is written to the first memory table. A second memory table is generated based on the storage status of the first memory table and stored in the buffer pool. The first memory table is the active memory table that stores the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. Flush the second memory table stored in the buffer pool to the storage device.
[0007] Fourthly, this application also provides a computer-readable storage medium having a data storage program stored thereon. When the data storage program is executed by a processor, it implements the data storage method described in the first aspect, including: Obtain the data write request and determine the metadata to be written corresponding to the data write request based on the data write request; The system status is acquired at each control cycle, the backlog deviation of data in the buffer pool is determined based on the system status, the number of second memory tables is determined based on the backlog deviation, and the number of second memory tables is updated. The metadata to be written is written to the first memory table. A second memory table is generated based on the storage status of the first memory table and stored in the buffer pool. The first memory table is the active memory table that stores the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. Flush the second memory table stored in the buffer pool to the storage device.
[0008] Fifthly, this application also provides a computer program product, including a computer program, which, when executed by a processor, implements the data storage method described in the first aspect, including: Obtain the data write request and determine the metadata to be written corresponding to the data write request based on the data write request; The system status is acquired at each control cycle, the backlog deviation of data in the buffer pool is determined based on the system status, the number of second memory tables is determined based on the backlog deviation, and the number of second memory tables is updated. The metadata to be written is written to the first memory table. A second memory table is generated based on the storage status of the first memory table and stored in the buffer pool. The first memory table is the active memory table that stores the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. Flush the second memory table stored in the buffer pool to the storage device.
[0009] The beneficial effects of the technical solution provided in this application embodiment are: by implementing the data storage method, apparatus, device and readable storage medium provided in this application embodiment, the number of the second memory table in the buffer pool is dynamically managed to ensure that data can be written continuously, thereby improving the efficiency of data writing. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is a schematic diagram of a data storage method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the hash bucket in the first memory table provided in an embodiment of this application; Figure 3 This is a schematic diagram of the data merging process provided in an embodiment of this application; Figure 4This is a schematic diagram of the data storage execution logic provided in the embodiments of this application; Figure 5 This is a schematic diagram of a B+ tree index provided in an embodiment of this application; Figure 6 This is a schematic diagram of a data storage device provided in an embodiment of this application; Figure 7 This is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0013] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The numbers in the accompanying drawings are only used to distinguish individual functional parts or modules and do not indicate logical relationships between parts or modules. The terms “comprising,” “including,” or “including,” and similar terms mean that the element or object preceding the term encompasses the element or object listed following the term and its equivalents, without excluding other elements or objects. The terms “connected,” “linked,” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. “Above,” “below,” “left,” “right,” etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0014] The various embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that in the drawings, the same reference numerals are assigned to components having substantially the same or similar structure and function, and repeated descriptions of them will be omitted.
[0015] In existing technologies, writing metadata to memory first requires saving it to an active memory table. Once the active memory table reaches 128MB of storage space, it is frozen and then flushed to disk. However, this process suffers from a discrepancy between the write rate to the active memory table and the flush rate. Furthermore, the failure to release a single frozen memory table can prevent the active memory table from freezing, thus hindering the reception of new data and impacting the data write rate. This application provides the following implementation method: In some embodiments, such as Figure 1 As shown, a data storage method includes: A100: Obtain the data write request and determine the metadata to be written corresponding to the data write request based on the data write request; A200: Acquire the system status at each interval control cycle, determine the backlog deviation of data in the buffer pool based on the system status, determine the number of second memory tables based on the backlog deviation, and update the number of second memory tables. A300: Write the metadata to be written to the first memory table, generate the second memory table according to the storage status of the first memory table, and save the second memory table in the buffer pool. The first memory table is the active memory table that stores the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. A400: Flush the second memory table stored in the buffer pool to the storage device.
[0016] Specifically, the process involves obtaining a data write request and determining the metadata to be written corresponding to the data write request, including: A110: Data write request.
[0017] A data write request includes: the raw data to be written, and additional information related to that raw data. The raw data can be files, objects, or records; the additional information can be namespaces, operation types, timestamps, etc.
[0018] A120: Parse the data write request and extract the metadata attribute fields from the data write request.
[0019] Metadata attribute fields include, but are not limited to: unique data identifier, data size, timestamp, namespace or path structure, data version number, data verification information, permission information, and storage tier recommendations. The generation of metadata attribute fields is based on the write request, but not solely on the information carried by the write request.
[0020] A unique identifier for data includes the data's "key," such as a file path or object ID. Typically, this is extracted directly from fields like "target path" and "object name" in the data write request. For example, a write request like `PUT / user / data / file1.txt` has the key ` / user / data / file1.txt`. The data's "key" represents its logical location and uniqueness.
[0021] The data size is obtained by extracting the actual data length carried in the write request. Illustratively, in the HTTP protocol, this can be obtained via Content-Length.
[0022] A timestamp can be represented by the creation time of the write request or the time the data was written to the storage device. It is automatically generated when the system receives the write request or when data is written to the storage device. To ensure consistency, the server-side time is usually used.
[0023] The storage device described in this application refers to a non-volatile storage medium, such as a disk drive (DDR) or a solid-state drive (SSD).
[0024] Namespaces are used for multi-tenant isolation or logical grouping; path structures are used to support hierarchical data queries. The namespace or path structure is parsed from the "key" of the write request. This information is implicit in the "key" and needs to be extracted through string parsing. For example, if the "key" is / projectA / dataset1 / file.parquet, its namespace could be "projectA" and its path could be " / dataset1".
[0025] The data version number is automatically generated by the system. For data with the same key, the version number can be set to increment automatically. Alternatively, a timestamp can be used as the version number; this application does not limit this choice.
[0026] Data verification information is used to verify data integrity and prevent data loss during transmission or storage. This information can be attached to the write request header or obtained by the system after receiving the data by calculating a checksum. Common verification methods include MD5 and CRC checksums.
[0027] Permission information determines the access control policy in the metadata. It is included in the request header of a write request and can also be inherited from the default policy of a namedspace or path.
[0028] Storage tiers are inferred by the system based on access patterns, data types, and other strategies.
[0029] A130: Generate metadata to be written based on metadata attribute fields.
[0030] Based on the aforementioned metadata attribute fields, a structured metadata object is created, and the metadata to be written is determined based on this object. This metadata object is typically represented in key-value pair format. Specifically, it includes: A131: Obtain the metadata object structure template; A132: Map the extracted metadata attribute fields to the metadata object structure template and fill the missing fields in the metadata object structure template with default values.
[0031] A133: Serializes the assigned metadata object structure template into formatted text.
[0032] A134: Generate key-value pairs based on the metadata values after serialization into formatted text, as the metadata to be written.
[0033] This key-value pair is the metadata that will be written to the first memory table.
[0034] A300: Writes the metadata to be written to the first memory table, including: A310: Obtain the storage status of the first memory table, which includes at least the stored capacity of the first memory table.
[0035] A320a: In response to the fact that the stored capacity has not reached the capacity threshold, the metadata to be written is saved in the first memory table.
[0036] A320b: In response to the storage capacity reaching the capacity threshold, a second memory table is generated based on the first memory table, and the second memory table is placed in the buffer pool.
[0037] The capacity threshold is typically set to 128MB. The first memory table (MemTable) stores the key-value pairs of metadata to be written in the form of a hash table. These key-value pairs are stored in several hash buckets, such as... Figure 2 As shown. When the data stored in the first memory table reaches 128MB, the first memory table is frozen to obtain the second memory table (Immutable MemTable). After the first memory table is frozen, new metadata to be written is no longer written to the second memory table. Instead, a new active first memory table is created to store the new metadata to be written, ensuring that new metadata to be written can be continuously written.
[0038] A330b: Create a new active memory table as the first memory table to store metadata to be written.
[0039] A400: Flush the second memory table stored in the buffer pool to the storage device, including: A410: In response to the generation of a new second memory table, the second memory table stored in the buffer pool is flushed to the storage device.
[0040] The second memory table can only be flushed to disk after the first memory table is frozen. To improve system throughput, asynchronous execution is typically used to flush the data in the second memory table to disk.
[0041] A420: Snapshot lock the contents of the newly generated second memory table.
[0042] After the system performs a snapshot lock on the second memory table, it ensures that the data within it is not changed during the disk flushing process, thus guaranteeing data consistency. This prevents data contention or alteration during the writing of data to the storage device.
[0043] A430: Sort the metadata stored in the second memory table after the snapshot by key value.
[0044] Sorting can be used to filter out and delete duplicate data.
[0045] A440: Add a timestamp to any metadata in the second memory table after sorting by key value.
[0046] Adding timestamps to metadata facilitates version management and the merging of identical data.
[0047] A451: Create a temporary string table at a specified location on the storage device as the first-level metadata unit in the storage device.
[0048] This is the process of persistently saving data from memory to storage devices. The first-level metadata unit is typically the first storage layer (represented as Level-0) in an LSM structure within the storage device. As the amount of data stored in the first-level storage layer increases, the data in the first-level storage layer is merged and saved to the second-level storage layer (Level-1). As the amount of data increases further, the second-level storage layer can be merged into the third-level storage layer, and the third-level storage layer can be merged into the fourth-level storage layer, performing a layer-by-layer merging process.
[0049] To illustrate, the merging module employs multiple merging strategies to merge disk data. One strategy, based on data size, triggers a merge operation when the data size at a certain level (e.g., Level-0 has a capacity threshold of 1GB, Level-1 has a capacity threshold of 2GB, and so on) exceeds the capacity threshold for that level. Another strategy, based on data timestamps, includes data in the merge scope if its storage time at a certain level exceeds a preset time threshold. A third strategy, based on data version numbers, retains the latest version of metadata with the same key value but different version numbers, deleting older versions during the merge process.
[0050] The merging process is as follows Figure 3 As shown, the data to be merged is first read from different levels and sorted according to timestamp or key-value order. Then, deduplication is performed to avoid storing identical metadata repeatedly. For conflicting data, the merging strategy is applied, updating relevant metadata information such as version number and timestamp. Finally, the merged data is written to the next level, and the merged data in the original level is deleted.
[0051] A452: Write the metadata with added timestamps into a temporary string table (temporary SSTable).
[0052] A453: Call a file synchronization system, such as fsync(), to completely save a temporary string table containing metadata to the storage device.
[0053] Executes the operating system-level fsync() system call, forcing file data to be flushed from the operating system page cache to the physical storage device.
[0054] A455: Converts a temporary string table stored on a storage device into a sorted string table (the official SSTable file). This conversion is typically achieved by changing the .tmp file name to a .sst file name.
[0055] A461: Receive notification that the sorted string table has been saved; A462: Mark the second memory table as the default mark; A463: Releases the storage space occupied by the second memory table marked with the preset tag. After flushing to disk, the space in the buffer pool occupied by the second memory table will be released to save total memory space and enable timely updates to the system status, facilitating the configuration of the number of second memory tables in the buffer pool.
[0056] A200: Acquires system status at interval control cycles, determines the backlog deviation of data in the buffer pool based on the system status, determines the number of items in the second memory table based on the backlog deviation, and updates the number of items in the second memory table, including: A210: Determine the set of control parameters.
[0057] The set of control parameters includes: N min N max P target K p , δ, T. Where, N min This is the lower limit for the number of second memory tables; typically, the lower limit for the number of second memory tables is 1. max This represents the upper limit for the number of second memory tables, which is determined by memory capacity constraints, but typically the number of second memory tables will not reach this limit; P targetThe target backlog of data represents the total amount of data to be flushed to disk that is expected to be maintained; K p δ is the proportional gain coefficient, used to map the backlog deviation to the adjustment amount of the second memory table quantity; δ is the hysteresis interval width, used to prevent frequent adjustments to the second memory table quantity; T is the control period, indicating that the second memory table quantity is adjusted once every control period, which has the function of preventing frequent adjustments to the second memory table quantity.
[0058] A220: Obtain the system status at each adjustment cycle.
[0059] System status includes: P t N represents the amount of backlog data at the current moment. t N represents the maximum number of second memory tables allowed by the current system. t N is a positive integer, and N is a positive integer. t The value in [N min N max Within the range.
[0060] A231: Determine the backlog deviation e based on the current backlog data volume and the target backlog data volume. t Its expression is: e t =P t -P target .
[0061] A232a: In response to the absolute value of the backlog deviation being less than the hysteresis interval width, the number of the second memory table remains unchanged.
[0062] A232b: In response to the absolute value of the backlog deviation being greater than or equal to the hysteresis interval width, the adjustment amount for the number of second memory tables is calculated.
[0063] Specifically, when according to |e t When |<δ, the number of items in the second memory table remains unchanged.
[0064] when|e t When |≥δ, adjust the number of the second memory table.
[0065] The adjustment amount for the number of the second memory table is determined by the formula: N t =K p ·e t Calculated.
[0066] A233b: Determine the target number of the second memory table based on the adjustment amount of the second memory table number.
[0067] Specifically, according to the formula: N new，t =round(N t +ΔN t Determine the target number of the second memory table.
[0068] Wherein, round(N) t +ΔN t ) represents N t +ΔN t The calculation result is rounded to the nearest integer.
[0069] A234b: Set the target number N of the second memory table. new，t Set the number of the second memory table for the next adjustment cycle.
[0070] Data storage methods also include: B100: Create a data index; B110: Establish the index relationship between the newly written metadata and the first memory table; B120: Establish the index relationship between the data waiting to be flushed to the storage device and the second memory table; B130: Establish the index relationship between the data flushed to the storage device and the sorted string table.
[0071] By creating an index, it becomes easier to quickly locate data using a binary search method based on pointers and key-value ranges when searching metadata. The index can be in the form of a B+ tree, starting from Level-0 and searching layer by layer to guide the search to find the target metadata or traverse all levels.
[0072] Data storage methods also include: C100: In response to receiving a metadata query request, the target metadata is retrieved from the cache, where the metadata query request contains the target metadata; C200a: If the target metadata is found in the cache, the target metadata is returned; C200b: In response to a missing target metadata in the cache, the target metadata is queried sequentially from the first memory table, the second memory table, and the storage device, and the hit target metadata is returned. C300b: Writes target metadata to the cache and updates the popularity of target metadata; C400: In response to a full cache, delete the least frequently used metadata.
[0073] A cache of a certain size is configured to cache frequently accessed metadata. The cache employs an LRU (Least Recently Used) replacement strategy; when the cache space is full, the least recently used metadata is removed. When a query request is received, the cache first searches for the target metadata in its own cache. If a cache hit occurs, the result is returned directly, reducing disk I / O operations. When new metadata is written or data is updated, the cache is updated promptly to ensure consistency between cached and disk data.
[0074] The data storage method execution logic described in this application can be as follows: Figure 4 The functional units shown are expressed in the form of the diagram.
[0075] Figure 4 In this system, the write unit is responsible for receiving metadata write requests. When new metadata is received, it is written to the first memory table in memory. The first memory table uses a hash table structure for fast metadata insertion. Figure 2 As shown, metadata is stored in key-value pairs in the hash bucket of the first in-memory table.
[0076] When the space occupied by the first memory table reaches the capacity threshold (e.g., 128MB), a conversion operation is triggered, converting the first memory table into a second memory table. At this time, the first memory table no longer accepts new write requests. Simultaneously, the system immediately creates a new first memory table to continue receiving subsequent metadata writes. Data in the second memory table is written asynchronously to the Level-0 layer of the storage device, and a timestamp is added to each piece of data for subsequent data merging and version management.
[0077] The query unit, upon receiving a metadata query request, performs a search in order from memory to disk. First, it searches for the target metadata in the current first memory table using a hash index. If found, the result is returned directly; otherwise, it searches in the second memory table. If the target metadata is still not found in the second memory table, it begins searching at the storage device level. Data on the disk is indexed using a B+ tree in the order of Level-0, Level-1, and Level-2. The query unit uses the B+ tree index to search layer by layer, starting from Level-0, until the target metadata is found or all levels have been traversed. The B+ tree index structure is as follows: Figure 5 As shown, its leaf nodes store the actual metadata, while non-leaf nodes store pointers to subtrees and key-value ranges, allowing for fast data location using a binary search algorithm.
[0078] The merging unit employs multiple merging strategies to merge disk data. One strategy is based on data size: when the data size at a certain level (e.g., Level-0 has a capacity threshold of 1GB, Level-1 has a capacity threshold of 2GB, and so on) exceeds the capacity threshold for that level, a merge operation is triggered. Another strategy is based on data timestamps: if data has been stored at a certain level for more than a preset time threshold (e.g., 7 days), that data is included in the merge. A third strategy is based on data version numbers: for metadata with the same key value but different version numbers, the latest version number is retained during the merge, while the older version data is deleted. The merging process is as follows: Figure 3 As shown, the data to be merged is first read from different levels and sorted according to timestamp or key-value order. Then, deduplication is performed to avoid storing identical metadata repeatedly. For conflicting data, the merging strategy is applied, updating relevant metadata information such as version number and timestamp. Finally, the merged data is written to the next level, and the merged data in the original level is deleted.
[0079] The indexing unit creates multi-level indexes for metadata. In memory, hash indexes are created for the first and second memory tables to facilitate fast location of metadata in memory. On disk, B+ tree indexes are created for different levels of data. The indexing unit regularly maintains and optimizes the indexes, including deleting index entries that have become invalid due to data deletion or merging, updating index information that has changed due to data updates, and adjusting the structure of the B+ tree (such as splitting or merging nodes) to maintain the balance of the B+ tree, reduce the tree height, and improve query efficiency.
[0080] The caching unit allocates a certain amount of cache space to cache frequently accessed metadata. The cache employs an LRU (Least Recently Used) replacement strategy; when the cache space is full, the least recently used metadata is removed from the cache. When a query request is received, the caching unit first searches for the target metadata in the cache. If the cache is hit, the result is returned directly, reducing disk I / O operations. When new metadata is written or data is updated, the caching unit promptly updates the data in the cache to ensure consistency between cached data and disk data.
[0081] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0082] By implementing the data storage method provided in this application embodiment, the number of second memory tables in the buffer pool is dynamically managed to ensure that data can be written continuously, thereby improving the efficiency of data writing.
[0083] In other embodiments, such as Figure 6 As shown, a data storage device includes: The data extraction module is used to obtain data write requests and determine the metadata to be written corresponding to the data write requests. The quantity update module is used to obtain the system status at each interval control cycle, determine the backlog deviation of data in the buffer pool based on the system status, determine the quantity of the second memory table based on the backlog deviation, and update the quantity of the second memory table. The data writing module is used to write the metadata to be written into the first memory table, generate a second memory table according to the storage status of the first memory table, and save the second memory table in the buffer pool. The first memory table is the active memory table that saves the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. The data flushing module is used to flush the second memory table stored in the buffer pool to the storage device.
[0084] For specific limitations regarding the data access device described above, please refer to the limitations regarding the data access method above, which will not be repeated here. Each module in the aforementioned data access device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.
[0085] By implementing the data storage device provided in the embodiments of this application, the number of second memory tables in the buffer pool is dynamically managed to ensure that data can be written continuously, thereby improving the efficiency of data writing.
[0086] In other embodiments, such as Figure 7 As shown, a computer device includes a memory, a processor, and a data storage program stored in the memory and executable on the processor. When the processor executes the data storage program, it implements the data storage method described in the first aspect, including: A100: Obtain data write requests and determine the metadata to be written corresponding to the data write requests; A200: Acquire the system status at each interval control cycle, determine the backlog deviation of data in the buffer pool based on the system status, determine the number of second memory tables based on the backlog deviation, and update the number of second memory tables. A300: Write the metadata to be written to the first memory table, generate the second memory table according to the storage status of the first memory table, and save the second memory table in the buffer pool. The first memory table is the active memory table that stores the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. A400: Flush the second memory table stored in the buffer pool to the storage device.
[0087] By implementing the computer device provided in the embodiments of this application, the number of second memory tables in the buffer pool is dynamically managed to ensure that data can be written continuously, thereby improving the efficiency of data writing.
[0088] In other embodiments, a computer-readable storage medium stores a data storage program thereon, which, when executed by a processor, implements the data storage method described in the first aspect, including: A100: Obtain data write requests and determine the metadata to be written corresponding to the data write requests; A200: Acquire the system status at each interval control cycle, determine the backlog deviation of data in the buffer pool based on the system status, determine the number of second memory tables based on the backlog deviation, and update the number of second memory tables. A300: Write the metadata to be written to the first memory table, generate the second memory table according to the storage status of the first memory table, and save the second memory table in the buffer pool. The first memory table is the active memory table that stores the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. A400: Flush the second memory table stored in the buffer pool to the storage device.
[0089] By implementing the computer-readable storage medium provided in the embodiments of this application, the number of second memory tables in the buffer pool is dynamically managed to ensure that data can be written continuously, thereby improving the efficiency of data writing.
[0090] In other embodiments, a computer program product includes a computer program that, when executed by a processor, implements the data storage method described in the first aspect, including: A100: Obtain data write requests and determine the metadata to be written corresponding to the data write requests; A200: Acquire the system status at each interval control cycle, determine the backlog deviation of data in the buffer pool based on the system status, determine the number of second memory tables based on the backlog deviation, and update the number of second memory tables. A300: Write the metadata to be written to the first memory table, generate the second memory table according to the storage status of the first memory table, and save the second memory table in the buffer pool. The first memory table is the active memory table that stores the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. A400: Flush the second memory table stored in the buffer pool to the storage device.
[0091] By implementing the computer program product provided in the embodiments of this application, the number of second memory tables in the buffer pool is dynamically managed to ensure that data can be written continuously, thereby improving the efficiency of data writing.
[0092] By implementing the data storage method, apparatus, device, and readable storage medium provided in the embodiments of this application, the number of second memory tables in the buffer pool is dynamically managed to ensure that data can be written continuously, thereby improving the efficiency of data writing.
[0093] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0094] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as steps controlled by a computer software program. For example, embodiments of this application include a computer program product comprising a computer program loaded on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from memory, or installed from ROM. When the computer program is executed by an external processor, it performs the functions defined in the methods of embodiments of this application.
[0095] It should be noted that the computer-readable medium in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.
[0096] The aforementioned computer-readable medium may be included in the aforementioned server; or it may exist independently and not assembled into the server. The aforementioned computer-readable medium carries one or more programs that, when executed by the server, cause the server to: in response to detecting that the peripheral mode of the terminal is not activated, acquire the frame rate of the application on the terminal; when the frame rate meets the screen-off condition, determine whether the user is acquiring the terminal's screen information; and in response to the determination that the user is not acquiring the terminal's screen information, control the screen to enter an immediate dimming mode.
[0097] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0098] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0099] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
[0100] The data storage method, device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. These embodiments are merely preferred embodiments of this application, used to help understand the method and core ideas of this application, and are not intended to limit this application. It should be noted that for those skilled in the art, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application without departing from its principles are also within the protection scope of this application.
Claims
1. A data storage method, characterized in that, include: Obtain a data write request, and determine the metadata to be written corresponding to the data write request based on the data write request; The system status is acquired at each interval adjustment cycle, the backlog deviation of data in the buffer pool is determined based on the system status, the number of second memory tables is determined based on the backlog deviation, and the number of second memory tables is updated. The metadata to be written is written to a first memory table. A second memory table is generated according to the storage status of the first memory table, and the second memory table is stored in a buffer pool. The first memory table is an active memory table that stores the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. The second memory table stored in the buffer pool is flushed to the storage device.
2. The data storage method according to claim 1, characterized in that, The step of obtaining a data write request and determining the metadata to be written corresponding to the data write request includes: Obtain the data write request; Parse the data write request and extract the metadata attribute fields from the data write request; Retrieve the metadata object structure template; The extracted metadata attribute fields are mapped to the metadata object structure template, and the missing fields in the metadata object structure template are filled with default values; Serialize the assigned metadata object structure template into formatted text; Generate key-value pairs from the metadata values after serialization into formatted text, and use them as metadata to be written.
3. The data storage method according to claim 1, characterized in that, The step of writing the metadata to be written to the first memory table includes: Obtain the storage status of the first memory table, wherein the storage status includes at least the stored capacity of the first memory table; In response to the fact that the stored capacity has not reached the capacity threshold, the metadata to be written is saved in the first memory table; In response to the storage capacity reaching the capacity threshold, a second memory table is generated based on the first memory table, and the second memory table is placed in the buffer pool. Create a new active memory table as the first memory table to store the metadata to be written.
4. The data storage method according to claim 1, characterized in that, The step of flushing the second memory table stored in the buffer pool to the storage device includes: In response to the generation of a new second memory table, the second memory table stored in the buffer pool is flushed to the storage device; Snapshot lock the contents of the newly generated second memory table; Sort the metadata stored in the second memory table after the snapshot by key value; Add a timestamp to any metadata in the second memory table after sorting by key value; A temporary string table is created at a specified location in the storage device as the first-level metadata unit in the storage device; The metadata with added timestamps is written into the temporary string table in chunks; The file synchronization system is invoked to completely save the temporary string table containing metadata to the storage device; Convert the temporary string table stored on the storage device into a sorted string table; Receive a notification that the sorted string table has been saved; Mark the second memory table with a preset label; Release the storage space occupied by the second memory table that is marked with a preset tag.
5. The data storage method according to claim 1, characterized in that, The system status is acquired at each interval adjustment cycle. Based on the system status, the backlog deviation of data in the buffer pool is determined. Based on the backlog deviation, the number of second memory tables is determined and the number of second memory tables is updated, including: Determine the set of control parameters; Acquire the system status at each control interval; The backlog deviation is determined based on the current backlog volume and the target backlog volume. If the absolute value of the backlog deviation is less than the hysteresis interval width, the number of the second memory tables remains unchanged. If the absolute value of the backlog deviation is greater than or equal to the hysteresis interval width, then the adjustment amount for the number of second memory tables is calculated. The target number of the second memory table is determined based on the adjustment amount of the second memory table number; Set the target number of the second memory table to the number of the second memory table in the next adjustment cycle.
6. The data storage method according to claim 1, characterized in that, The method further includes: Create a data index; Establish an index relationship between the newly written metadata and the first memory table; Establish an index relationship between the data awaiting flushing to the storage device and the second memory table; Establish an index relationship between the data that has been flushed to the storage device and the sorted string table.
7. The data storage method according to claim 1, characterized in that, The method further includes: In response to receiving a metadata query request, the target metadata is retrieved from the cache, wherein the metadata query request contains the target metadata; If the target metadata is found in the cache, the target metadata is returned. In response to a cache failure of the target metadata, the target metadata is queried sequentially from the first memory table, the second memory table, and the storage device, and the target metadata that is found is returned. Write the target metadata into the cache and update the popularity of the target metadata; If the cache space is full, delete the least frequently used metadata.
8. A data storage device, characterized in that, include: The data extraction module is used to acquire data write requests and determine the metadata to be written corresponding to the data write requests based on the data write requests. The quantity update module is used to obtain the system status at each interval adjustment cycle, determine the backlog deviation of data in the buffer pool based on the system status, determine the quantity of the second memory table for the next adjustment cycle based on the backlog deviation, and update the quantity of the second memory table when the next adjustment cycle arrives. The data writing module is used to write the metadata to be written into a first memory table, generate a second memory table according to the storage status of the first memory table, and save the second memory table in a buffer pool. The first memory table is an active memory table that saves the metadata to be written. The number of second memory tables that can coexist in the buffer pool is the number of updated second memory tables. The second memory table is generated from the first memory table. The data flushing module is used to flush the second memory table stored in the buffer pool to the storage device.
9. A computer device, characterized in that, The device includes a memory, a processor, and a data storage program stored in the memory and executable on the processor. When the processor executes the data storage program, it implements the data storage method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores a data storage program, which, when executed by a processor, implements the data storage method according to any one of claims 1 to 7.