Data storage method and device, computer equipment and readable storage medium

Abstract

The embodiment of the invention provides a data storage method and device, computer equipment and a readable storage medium. The method comprises the steps of obtaining to-be-written data for a target sub-file, and dividing the to-be-written data into at least one logic data block; determining a block identification key of the target sub-file according to the file identifier and the file position index of the target sub-file; determining a modification attribute corresponding to each logic data block, determining an offset of each logic data block relative to the target sub-file, and generating a block data value corresponding to each logic data block in combination with the logic data blocks, the modification attributes and the offset; for each logic data block, generating a target key value pair of each logic data block based on the corresponding block identification key and the block data value, and storing the target key value pair; wherein the target key value pair is used for indicating that the logic data block is combined with the corresponding target basic data block. Therefore, the system performance can be effectively improved, and the write operation delay is reduced.

Description

Data storage methods, devices, computer equipment and readable storage media Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data storage method, apparatus, computer equipment, and readable storage medium. Background Technology

[0002] In applications such as logging and IoT sensor data acquisition, file systems often face the need for high-frequency, small-write data processing. These operations are characterized by extremely frequent write requests, but each request involves a very small amount of data, possibly only a few bytes to a few kilobytes. Therefore, a storage solution capable of achieving efficient, low-latency data storage in such scenarios is needed.

[0003] In related technologies, small files are typically stored directly in the file system, or large files are split and then stored in the file system to improve the storage and access efficiency of small files. However, the data organization and management of these technologies still essentially use files as the smallest storage unit. This means that even if only one byte in a file is modified, the entire file needs to be rewritten. This not only consumes I / O bandwidth and storage space, leading to reduced system performance, but also introduces significant write operation latency. Summary of the Invention

[0004] This application proposes a data storage method, apparatus, computer equipment, and readable storage medium, which can effectively improve system performance and reduce write operation latency.

[0005] To achieve the above objectives, a first aspect of this application proposes a data storage method, the method comprising: acquiring data to be written for a target sub-file, dividing the data to be written into at least one logical data block; acquiring a file identifier and a file position index in the target sub-file, and determining a block identifier key for the target sub-file based on the file identifier and the file position index; determining a modification attribute corresponding to each logical data block, and determining an offset of each logical data block relative to the target sub-file, and generating a block data value corresponding to each logical data block by combining the logical data block, the modification attribute, and the offset; for each logical data block, generating a target key-value pair for each logical data block based on the corresponding block identifier key and the block data value, and storing the target key-value pair; wherein the target key-value pair is used to indicate merging the logical data block with a corresponding target base data block, and the logical data block is modified data relative to the target base data block.

[0006] Accordingly, a second aspect of this application provides a data storage device, the device comprising: an acquisition module, configured to acquire data to be written for a target sub-file, and divide the data to be written into at least one logical data block; a determination module, configured to acquire a file identifier and a file position index in the target sub-file, and determine a block identifier key for the target sub-file based on the file identifier and the file position index; a generation module, configured to determine a modification attribute corresponding to each logical data block, and determine an offset of each logical data block relative to the target sub-file, and generate a block data value corresponding to each logical data block by combining the logical data block, the modification attribute, and the offset; and a storage module, configured to generate a target key-value pair for each logical data block based on the corresponding block identifier key and the block data value, and store the target key-value pair; wherein the target key-value pair is used to indicate merging the logical data block with a corresponding target base data block, and the logical data block is modification data relative to the target base data block.

[0007] In some embodiments, the data storage method apparatus further includes a merging module, configured to: in response to a merging trigger operation for the block identifier key, obtain a target basic data block and at least one logical data block associated with the block identifier key; obtain the number of operations corresponding to the at least one logical data block, and determine a corresponding merging function based on the number of operations; and merge the target basic data block and the at least one logical data block using the merging function to obtain a target data block.

[0008] In some implementations, the merging module is further configured to: read the offset of each logical data block in the target subfile from the block data value of each logical data block using the partial merging function; calculate the merging range of the at least one logical data block in the target subfile based on the offset of the at least one logical data block; allocate a first buffer according to the merging range, and copy the at least one logical data block to the first buffer for data merging based on the offset of each logical data block, thereby obtaining the target data block stored in the first buffer.

[0009] In some implementations, the merging module is further configured to: allocate a second buffer using the full merge function, and copy the target base data block to the second buffer; when the target base data block contains logical data blocks to be merged, read the offset of each logical data block in the target subfile from the block data value of each logical data block; and copy each logical data block to the corresponding position in the second buffer according to the offset of each logical data block relative to the target subfile, so as to modify the target base data block until the target subfile does not contain any logical data blocks to be merged, thereby obtaining the target data block stored in the second buffer.

[0010] In some embodiments, the data storage device further includes a reading module, configured to: when the at least one logical data block does not have a corresponding target base data block in the target subfile, read the offset of each logical data block in the target subfile from the block data value of each logical data block; copy each logical data block to the corresponding position in the target subfile according to the offset of each logical data block relative to the target subfile, until the target subfile does not have any logical data blocks to be merged, thereby obtaining an updated target subfile.

[0011] In some embodiments, the data storage device further includes a setting module, configured to: obtain an initial data bitmap corresponding to the target sub-file; determine multiple valid data elements and invalid data elements from multiple data elements contained in the target sub-file; determine a first distribution position corresponding to each valid data in the initial data bitmap, and a second distribution position corresponding to each invalid data in the initial data bitmap; set the first distribution position to a set state and the second distribution position to an unset state in the initial data bitmap to obtain a target data bitmap; then, the step of combining the logical data block, the modified attribute, and the offset to generate a block data value corresponding to each logical data block includes: combining the target data bitmap, the logical data block, the modified attribute, and the offset to generate a block data value corresponding to each logical data block.

[0012] In some embodiments, the data storage device further includes a construction module, configured to: determine a target file, a parent directory corresponding to the target file, and a parent file identifier corresponding to the parent directory; wherein the target file includes multiple target sub-files; obtain the file name of the target file, and construct a directory key for the target file based on the parent file identifier and the file name; obtain the file identifier of the target file, and store the directory key and the file identifier in a key-value storage column family to establish a mapping relationship between the parent directory and the target file.

[0013] Accordingly, a third aspect of the present application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the data storage method of any one of the embodiments of the first aspect of the present application.

[0014] Accordingly, a fourth aspect of the embodiments of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data storage method of any one of the embodiments of the first aspect of this application.

[0015] This application embodiment obtains the data to be written for a target sub-file, divides the data into at least one logical data block; obtains the file identifier and file position index in the target sub-file, and determines the block identifier key of the target sub-file based on the file identifier and file position index; determines the modification attribute corresponding to each logical data block, and determines the offset of each logical data block relative to the target sub-file, and generates the block data value corresponding to each logical data block by combining the logical data block, modification attribute, and offset; for each logical data block, a target key-value pair is generated based on the corresponding block identifier key and block data value, and the target key-value pair is stored; wherein, the target key-value pair is used to indicate merging the logical data block with the corresponding target base data block, and the logical data block is the modified data relative to the target base data block. In this way, a file write operation can be transformed into an incremental merging of only the changed blocks, completely skipping the necessary path of rewriting the entire file in related technologies. Specifically, in high-frequency, small-write scenarios, this application only needs to process the logical data blocks involved in the changes, rather than the entire file. This allows the system to complete persistence with only byte-level differences, eliminating the need to read back the original file and then write back the entire file. This compresses the I / O data volume from the file level to a partially incremental level, minimizing the write path and significantly reducing the amount and frequency of I / O operations. It also reduces write amplification and improves storage space utilization and write processing efficiency. In summary, this application effectively improves system performance and reduces write operation latency. Attached Figure Description

[0016] Figure 1 is a flowchart of the data storage method provided in the embodiments of this application; Figure 2 is a diagram of the generation process of the target data bitmap provided in the embodiments of this application; Figure 3 is an example diagram of the target key-value pair provided in the embodiments of this application; Figure 4 is an example diagram of the directory key structure provided in the embodiments of this application; Figure 5 is an overall processing flow for merging logical data blocks provided in the embodiments of this application; Figure 6 is a functional module diagram of the data storage device provided in the embodiments of this application; Figure 7 is a hardware structure diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] In applications such as logging and IoT sensor data acquisition, file systems often face the need for high-frequency, small-write data processing. These operations are characterized by extremely frequent write requests, but each request involves a very small amount of data, possibly only a few bytes to a few kilobytes. Therefore, a storage solution capable of achieving efficient, low-latency data storage in such scenarios is needed.

[0021] In related technologies, small files are typically stored directly in the file system, or large files are split and then stored in the file system to improve the storage and access efficiency of small files. However, the data organization and management of these technologies still essentially use files as the smallest storage unit. This means that even if only one byte in a file is modified, the entire file needs to be rewritten. This not only consumes I / O bandwidth and storage space, leading to reduced system performance, but also introduces significant write operation latency.

[0022] Based on this, embodiments of this application provide a data storage method, apparatus, computer device, and readable storage medium, which can effectively improve system performance and reduce write operation latency.

[0023] The data storage method, apparatus, computer equipment, and readable storage medium provided in the embodiments of this application are specifically described through the following embodiments.

[0024] In some implementations, the data storage method can be applied to a computer device that has the LogMergeFS generic file system installed and running. The computer device can be any device with computing and storage capabilities, such as a personal computer, server, mobile terminal, or embedded device. On the computer device, LogMergeFS is mounted as a user-space file system (implemented through the FUSE framework), with its underlying data organization and management based on the key-value store system RocksDB, thereby providing file storage services for applications on the device.

[0025] Specifically, when the computer device faces high-frequency, small-granularity data write scenarios, LogMergeFS operates as follows: First, the target file is divided into target subfiles of fixed size (e.g., 64KB). Each target subfile is uniquely located using a block identifier key containing a file identifier (inode_no) and a block index (block_index). The data to be written is encapsulated into block data values ​​containing modification attributes, offsets, and the actual data content, and submitted directly to RocksDB's Merge interface as an incremental write operation, forming target key-value pairs. This effectively avoids the read-modify-write overhead of related technologies, achieving low-latency block-level incremental updates.

[0026] Furthermore, LogMergeFS can automatically merge multiple incremental data blocks belonging to the same block identifier key with the base data block in the background or upon triggering, through built-in merging logic on the computer device, generating a new valid data block. Simultaneously, it can utilize bitmaps in the metadata to compress and store sparse data blocks, retaining only valid sub-blocks, significantly improving storage space utilization. Through these mechanisms, the computer device can significantly reduce write amplification and improve write throughput and storage efficiency when processing frequent small-scale writes, partial updates, and sparse data.

[0027] The data storage method in this application can be illustrated through the following examples.

[0028] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user will be obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent will the necessary user-related data for the normal operation of the embodiments of this application be obtained.

[0029] In this embodiment, the description will focus on the data storage device, which can be integrated into a computer device. Referring to Figure 1, which is a flowchart of the data storage method provided in this embodiment, this embodiment takes the data storage device being integrated into a terminal or server as an example. When the processor on the terminal or server executes the program instructions corresponding to the data storage method, the specific process is as follows: Step 101, obtain the data to be written for the target sub-file, and divide the data to be written into at least one logical data block.

[0030] In some implementations, in order to achieve fine-grained, block-level incremental writing of file data, the data to be written to a local area of ​​the file can be divided into at least one logical data block according to the block size preset by the underlying storage system (such as 64KB). This decouples a single write operation into subsequent processing of one or more independent storage units, thereby laying the foundation for supporting efficient local updates and merging.

[0031] The target subfile can be a logical block of the file to be written. For example, it can be a data block with a continuous logical address range obtained by dividing the original file according to a preset size (such as 64KB). This data block is uniquely addressed in the key-value storage system by a block identifier key containing a file identifier and a block index.

[0032] The data to be written can be new data content requested to be written to the target sub-file. For example, it can be the data payload carried by a write request from a user program or system service, and the data to be written can only cover a portion of the logical address range of the target sub-file.

[0033] In this context, a logical data block can be a data unit formed after the data to be written is divided, and aligned with the granularity of the storage system management. For example, it can be a data segment obtained by dividing the data to be written according to a fixed size (such as 4KB) or a write request boundary. The logical data block can serve as the basic data unit for subsequent key-value pair generation and storage and merging operations.

[0034] In some implementations, the target subfile can be a logical block of the file to be written to. Specifically, in a general-purpose file system (LogMergeFS) based on a key-value store system (such as RocksDB), the file content can be pre-divided into multiple fixed-size logical blocks, for example, each block is 64KB in size. Each such logical block, i.e., the target subfile, is addressed and managed in the system using a unique block identifier key, which is typically generated by combining the file identifier (inode_no) and the index number of the target subfile within the file (block_index). When a write request for a file is received, one or more target subfiles affected (or needing to be modified) by the write operation can be identified.

[0035] For example, the data to be written can be the data payload carried by the write request. Specifically, after acquiring the data to be written, it can be divided into one or more finer-grained logical data blocks according to a predetermined data organization granularity. This partitioning process ensures that the partitioning unit is the smallest unit of underlying storage management (e.g., to support subsequent sparse compression, it may be further managed in 4KB sub-blocks) and the boundaries of the data to be written itself, so as to ensure that each logical data block carries a continuous piece of data content.

[0036] Furthermore, a preset partitioning granularity (e.g., 4KB) can be maintained. Starting from the beginning of the data to be written, the data is divided according to this granularity until all data to be written is covered. The size of the last logical data block may be smaller than the standard granularity. For example, if the data to be written is 10KB and the partitioning granularity is 4KB, it can be divided into three logical data blocks of 4KB, 4KB, and 2KB respectively. Each logical data block records its internal offset (merge_offset) relative to the beginning of its target subfile.

[0037] In some implementations, the data to be written may not need to be divided. That is, the data to be written may be directly treated as a single logical data block, or it may be divided into 64KB blocks. Whether or not to divide the data, and how to divide it, can be determined based on the actual situation.

[0038] This decouples a write request, which may cover a local area of ​​a file and is of variable size, into operations on several independent data units of standard size, each with its own location information. In this way, a write request that might cover a local area of ​​a file can be logically organized into operations on one or more logical data blocks, thereby reducing the update granularity from the file level to the block level or sub-block level. This provides the necessary prerequisite for subsequent precise positioning based on block identifier keys and incremental merging based on offsets and modification attributes without reading the original data. Furthermore, it creates conditions for achieving efficient, low-latency random writes and sparse data compressed storage.

[0039] Step 102: Obtain the file identifier and file position index of the target sub-file, and determine the block identifier key of the target sub-file based on the file identifier and file position index.

[0040] In some implementations, in order to achieve efficient and accurate location and block-level management of file data in a key-value storage system, the unique identifier (file identifier) ​​of the file to which the target sub-file belongs and its logical position number (file position index) within the file can be extracted and combined to construct a globally unique key (block identifier key), so that each logical data block can be independently and directly addressed and manipulated in the underlying storage.

[0041] The file identifier can be a number used to uniquely identify a file within the file system. For example, it can be the number of the file inode, which is assigned by the system when the file is created and remains unchanged throughout the file's lifecycle, used to associate all target subfiles of the file.

[0042] The file position index can be a sequence number indicating the logical position of a target subfile within its parent file. For example, it could be the block index number corresponding to the target subfile after it has been divided into segments of a fixed size (e.g., 64KB) starting from the beginning of the file.

[0043] The block identifier key can be a key generated by combining a file identifier and a file position index according to a preset rule, used to uniquely identify and retrieve a specific target sub-file in the key-value storage system.

[0044] In some embodiments, the file system may assign a globally unique file identifier to each target file, which is determined when the target file is created and remains unchanged during its life cycle. In a specific implementation, this file identifier is the inode number of the target file (which can be represented by inode_no) and is stored in the metadata structure of the target file. When processing a write request for a certain target sub-file, the system can directly read this file identifier from the metadata of the target file to which the target sub-file belongs.

[0045] Specifically, the file position index is used to identify the logical sequential position of the target sub-file within its target file. Each target file is logically divided into blocks of a fixed size (e.g., 64KB) (i.e., target sub-files), and these blocks are sequentially numbered starting from 0. The file position index is the block number (block_index) corresponding to the target sub-file. For example, for a 1MB file (divided into 16 64KB target sub-files), the block_index of its 5th target sub-file is 5.

[0046] It can be understood that the chunk identification key can be a combined encoding of the file identifier and the file position index into a whole, such that the key itself contains both the file identity information of the target file and the position information of the target sub-file, thus enabling direct addressing without external mapping.

[0047] In some embodiments, the chunk identification key can be an integer key value obtained by combining the file identifier and the file position index using binary bit operations. Specifically, the file identifier can be left-shifted by a sufficient number of bits to make room for the file position index in the lower bits, and then added to the file position index.

[0048] Exemplarily, the chunk identification key Key can be calculated by the following formula: Key = (inode_no << N) + block_index; where inode_no represents the file identifier, block_index represents the file position index, and N is a preset offset bit number, whose value ensures that block_index can be fully accommodated in the lower bits of the chunk identification key and there is no bit overlap with inode_no.

[0049] For example, if N=64 and 64-bit unsigned integers are used to store inode_no and block_index, the final generated key is a 128-bit integer (or processed in two 64-bit chunks in actual storage). Its high 64 bits are filled with inode_no, and its low 64 bits are filled with block_index. Assuming inode_no=12345 and block_index=7, the calculation process is as follows: first, shift 12345 left by 64 bits to obtain a value where the high 64 bits are 12345 and the low 64 bits are all 0s. Then, add 7. The final key has its high 64 bits representing the target file 12345 and its low 64 bits representing the 7th target subfile of that target file.

[0050] By using the above method, a globally unique and precise address (block identifier key) containing location information can be generated for each logical data block. This allows for direct operations on this key (such as reading and writing incremental data) without needing to consider other parts of the file, achieving true block-level random access and management. This lays the technical foundation for subsequent efficient incremental data merging based on this key and for supporting fine-grained writing under high concurrency.

[0051] Step 103: Determine the modification attributes corresponding to each logical data block and the offset of each logical data block relative to the target subfile. Combine the logical data block, modification attributes, and offset to generate the block data value corresponding to each logical data block.

[0052] In some implementations, to achieve self-description and efficient merging of incremental data, each logical data block can be labeled with its data type attributes (i.e., modification attributes) and its specific starting position (i.e. offset) within the target subfile. This information, along with the original data content, is then encapsulated into structured block data values. This allows the storage system to complete correct and accurate data merging and updating based solely on the information contained within the data value itself, without relying on external context.

[0053] The modification attribute can be information used to identify the type of logical data blocks during the data merging process. For example, it can be a flag indicating whether the logical data block is used as a base data block for subsequent merging or as an incremental data block that modifies the base data block, thus determining the logic used in the merging operation.

[0054] The offset can be a quantity used to indicate the starting write position of the data content in the logical data block within the target subfile. For example, it can be a numerical value (merge_offset) indicating from which byte position in the target subfile (or the corresponding base data block) the valid data carried by the logical data block should be applied or overwritten.

[0055] The chunked data value can be the specific content corresponding to the chunk identifier key and used for storage in the key-value store system. It can include data header information and valid data payload. For example, it can be a binary structure composed of a header structure (containing fields such as modification attributes and offsets) and the actual data area (i.e., logical data block), which is persistently stored as the Value in the key-value pair.

[0056] In some implementations, modification attributes can be used to distinguish whether a data block is a base block used as the basis for subsequent merging, or an incremental block that only represents a partial modification of existing data. Modification attributes can be encoded using one or more flags. For example, flags=0x01 indicates a base block, and flags=0x02 indicates an incremental block. Specifically, modification attributes can be determined based on the write scenario: when writing data to a completely new target subfile for the first time, the modification attributes of all logical data blocks written in this instance can be determined as the base block; when making partial modifications to an existing target subfile, the modification attributes of the corresponding logical data blocks are determined as incremental blocks.

[0057] Furthermore, the offset can be an internal offset relative to the start position (0 bytes) of the target subfile. For example, if the starting global offset for a write request is global_offset, the starting global offset of the target subfile is block_start_offset (equal to block_index). If the target file contains a block of data of size BLOCK_SIZE, then the offset `merge_offset` within the target subfile can be determined as follows: `merge_offset = global_offset - block_start_offset`; where `global_offset` can be the starting position of the write operation in the target file, `block_start_offset` can be the starting position of the target subfile within the target file, and `BLOCK_SIZE` is the standard size of the target subfile (e.g., 64KB). This calculation converts the global offset at the target file level into the internal offset at the level of its target subfile, thus achieving precise positioning.

[0058] In some implementations, the block data values ​​can be obtained by concatenating the header structure and the data area in sequence. The header structure (block_head) can be filled with modified attributes and offsets. Specifically, for incremental data blocks, the header structure can include the modified attribute flags (identifying incrementality) and the offset merge_offset field. For the data area, logical data blocks (i.e., the effective data payload after partitioning) can be directly used as the content of the data area. Finally, the filled header structure and data area are concatenated sequentially in the memory buffer to form a complete binary data block, i.e., the final block data value.

[0059] By using the above methods, each logical data block can be transformed into a self-contained data unit (block data value) containing complete semantics (what data it is and where it should be placed). In this way, when a storage engine (such as RocksDB) triggers a merge operation, it does not need to pre-read the original basic data on disk; it can complete the sorting, deduplication, and integration of multiple data units in memory based solely on the offsets and modified attributes recorded within the current block data value, thus improving write efficiency and system performance.

[0060] In some implementations, in order to achieve efficient compressed storage and precise management of sparse data blocks, a bitmap representing the validity of its internal data can be maintained for each target subfile, and the bitmap can be updated according to the actual data distribution to generate a target data bitmap that accurately reflects the location of valid data. This allows the storage system to persist only valid data content and omit invalid areas by default, achieving bit-level space saving. For example, before step 103, "combining logical data blocks, modifying attributes, and offsets to generate the segmented data value corresponding to each logical data block," the process may further include: (A.1) obtaining the initial data bitmap corresponding to the target sub-file; (A.2) determining multiple valid data elements and invalid data elements from the multiple data elements contained in the target sub-file; (A.3) determining the first distribution position corresponding to each valid data in the initial data bitmap, and the second distribution position corresponding to each invalid data in the initial data bitmap; (A.4) in the initial data bitmap, setting the first distribution position to a set state and setting the second distribution position to an unset state to obtain the target data bitmap; then, combining logical data blocks, modifying attributes, and offsets to generate the segmented data value corresponding to each logical data block includes: combining the target data bitmap, logical data blocks, modifying attributes, and offsets to generate the segmented data value corresponding to each logical data block.

[0061] The initial data bitmap can be a bit sequence used to initially record the validity status of each preset-size data unit within a target subfile. For example, it can be a 16-bit bitmap (data_bitmap), where each bit may initially be 0, and each bit corresponds to a fixed-size (e.g., 4KB) sub-block in the target subfile.

[0062] Here, a data element can be the smallest logical storage unit that constitutes the target subfile. For example, it can be multiple fixed-size (e.g., 4KB) sub-blocks into which the target subfile is further subdivided.

[0063] Among them, a valid data element can be a data element in the target subfile that stores actual data that is not a default value (not all zeros).

[0064] Invalid data elements can be data elements in the target subfile that have not been written with valid data and whose content is treated as a default value (such as all zeros). For example, it could be an unused 4KB logical subblock at the end of the file or that has been logically deleted.

[0065] The first distribution position can be the position of a specific bit in the initial data bitmap corresponding to a valid data element. For example, if the third 4KB sub-block is valid data, then its third bit in the 16-bit bitmap is the first distribution position.

[0066] The second distribution position can be the position of a specific bit in the initial data bitmap corresponding to an invalid data element. For example, if the fifth 4KB sub-block is invalid data, then the fifth bit in the 16-bit bitmap is the second distribution position.

[0067] In this context, a set state can be a bit value in a bitmap that indicates that the corresponding data element is valid. For example, a bit can be set to "1" to indicate a set state, signifying that the logical sub-block corresponding to that bit stores valid data.

[0068] In this context, the unset state can be a bit value in the bitmap that indicates that the corresponding data element is invalid. For example, the value of a bit can be kept as "0" to indicate the unset state, indicating that the logical sub-block content corresponding to that bit is the default value (all zeros) and does not need to be stored physically.

[0069] The target data bitmap can be an updated bitmap obtained by updating the initial data bitmap based on the actual validity of data elements within the target subfile, and is ultimately used to control data storage. For example, it could be an updated 16-bit bitmap where bits of "1" indicate valid 4KB sub-blocks that need to be physically stored.

[0070] As shown in Figure 2, for example, the initial data bitmap corresponding to the target sub-file can be obtained. The initial data bitmap can be a preset sequence of all-zero bitmaps, such as a 16-bit binary number where all bits are initially 0 (0000000000000000), indicating that all data elements are invalid (the content is considered all zeros). This initial data bitmap can be updated as data elements of the target sub-file are created.

[0071] Furthermore, after three logical data blocks (file 1, file 2, and file 3) are written to the target subfile, a merge operation can be performed to determine the positions corresponding to data elements 1 and 2 in the target subfile. Data elements 1 and 2 are valid data elements, corresponding to the first distribution position in the data bitmap (data_bitmap), which is the first and second bits of the initial data bitmap. The first and second bits are set to the set state and marked as 1. Elements that have not been overwritten in this write operation and have never been written with valid data in the past are considered invalid data elements. Data elements 3-16 are all invalid data elements, corresponding to the second distribution position in the initial data bitmap (data_bitmap). In the initial data bitmap, bits 3 to 16 are set to the unset state, which is marked as 0.

[0072] The resulting bitmap after the update is the target data bitmap, which accurately reflects the distribution of all valid data in the target sub-file. Subsequent updates to the target sub-file will update the target data bitmap accordingly.

[0073] Using the above method, the data filling situation within a data block (target sub-file) can be precisely recorded at the bit-level granularity, generating a target data bitmap. Therefore, when persistently storing the target sub-file, this bitmap allows for selective writing of only valid data elements, while skipping all data elements marked as invalid (not set), achieving on-demand storage on the physical medium. This provides accurate metadata guidance for subsequent data reading and merging operations, ensuring that significant storage space is saved (especially for sparse data) without compromising the integrity of the logical data view.

[0074] Step 104: For each logical data block, generate a target key-value pair for each logical data block based on the corresponding block identifier key and block data value, and store the target key-value pair; wherein, the target key-value pair is used to indicate merging the logical data block with the corresponding target base data block, and the logical data block is the modified data relative to the target base data block.

[0075] In some implementations, in order to achieve direct, atomic persistence of incremental data modifications and lay the foundation for subsequent efficient merging, a self-contained incremental operation unit (target key-value pair) can be constructed by combining a block identifier key that precisely points to the storage location with a block data value that contains complete operation semantics. This is then directly submitted to the underlying key-value storage system, which abstracts a data modification into an independent storage transaction, thereby improving storage efficiency and optimizing system performance.

[0076] In this context, the target key-value pair can be a complete data storage unit composed of a block identifier key and a corresponding block data value. For example, it can be an entry in the underlying key-value storage system (such as RocksDB), where the key is a block identifier key encoded by a file identifier and a block index, and the value is a block data value encapsulating the modification attributes, offset, and the actual logical data block. The target key-value pair, as a whole, represents a specific write or modification intent targeting a target subfile (which may contain target underlying data blocks).

[0077] In some implementations, the target key-value pair is the basic unit of operation for the interaction between the LogMergeFS file system and the underlying key-value storage engine (such as RocksDB). For each logical data block, the system associates and encapsulates its corresponding block identifier key (Key) and block data value (Value) to form a complete key-value pair, i.e., the target key-value pair. The data content contained in the Value part (block data value) of the target key-value pair should be treated as an incremental modification and merged with the target underlying data block associated with the same Key (block identifier key) that already exists in the target subfile at a future point in time; if the target underlying data block does not exist in the target subfile, then the logical data block will be merged and stored in the target subfile later, or directly written to the target subfile.

[0078] As shown in Figure 3, the directory structure and file data of a file system can be organized and managed using a unified key-value pair format. The left side of Figure 3 shows the tree-like relationship from the root directory (inode_no:0) to subdirectories (such as A, B, C) and files (such as aa.jpg, bb.jpg), with this metadata also stored in key-value pairs. The right side specifically illustrates the mapping method of file data blocks in key-value storage. For example, the file "aa.jpg" (inode_no:3, size 100KB) is logically divided into multiple data blocks. Its block identifier key consists of "inode_no<<64+block_index" (e.g., "3<<64+0" corresponds to the first 64KB block, and "3<<64+1" corresponds to the remaining part), and the value is a composite structure containing the block header and the actual data. This intuitively demonstrates the mapping relationship between the file system's logical view and the underlying key-value storage, as well as the explicit semantics carried by each target key-value pair: the data content contained in its Value part (block data value) should be treated as an incremental modification and, at some point in the future, merged with the target base data block already existing in the system and associated with the same Key (block identifier key). The target base data block is a baseline version representing the current complete state of the data block, previously generated through the merge operation.

[0079] Specifically, this can be done by calling the merge operation interface provided by the underlying key-value storage engine (such as RocksDB's Merge interface). The target key-value pair can be passed as a parameter to the Merge interface of the storage engine corresponding to LogMergeFS. Upon receiving the Merge operation request, the storage engine does not immediately perform the data merge. Instead, it appends the target key-value pair as an operation record to be merged to the currently active in-memory table (MemTable). MemTable is an ordered key-value pair buffer maintained in memory, used to temporarily store all recent write operations (including Put, Delete, Merge, etc.).

[0080] Furthermore, when the size of MemTable reaches a preset threshold, the storage engine converts it to an immutable state and sequentially flushes it to disk, forming a persistent sorted string table file (SSTable, or SST file for short). During this process and in subsequent background compaction operations, the storage engine collects all related operation records for the same key (i.e., the same block identifier key). At this point, the storage engine invokes the merge operator that has been pre-registered with the system.

[0081] Furthermore, when the merge operator is invoked, it retrieves the target base data block associated with the current key (if it exists) and all logical data blocks to be merged (i.e., incremental data written via the Merge operation). Based on information such as flags, merge_offset, and data_bitmap in the block data values, it performs the corresponding merge calculations in memory to generate a new target data block. This newly generated target data block will serve as the new base value for the key.

[0082] Furthermore, after the merge is complete, this new key-value pair (Key is the original block identifier key, and Value is the new target data block obtained after the merge) will be written to a new, lower-level SST file. The original SST file containing incremental operation records will be cleaned up in subsequent garbage collection.

[0083] In this way, the system transforms a fine-grained data modification into an atomic, deferred-mergeable storage operation. By leveraging the inherent Merge mechanism of the LSM-Tree storage engine, all multiple incremental modifications to the same data block are efficiently buffered, sorted, and deferred-merged, thereby transforming intensive random small writes into sequential batch writes and background merging, significantly improving write throughput while avoiding synchronous I / O overhead and write amplification issues. The storage engine is responsible for triggering the merge at appropriate times (such as compaction), allowing the file system to be unconcerned about the specific scheduling and execution of the merge, thus decoupling storage logic from computation logic.

[0084] This application embodiment obtains the data to be written for a target sub-file, divides the data into at least one logical data block; obtains the file identifier and file position index in the target sub-file, and determines the block identifier key of the target sub-file based on the file identifier and file position index; determines the modification attribute corresponding to each logical data block, and determines the offset of each logical data block relative to the target sub-file, and generates the block data value corresponding to each logical data block by combining the logical data block, modification attribute, and offset; for each logical data block, a target key-value pair is generated based on the corresponding block identifier key and block data value, and the target key-value pair is stored; wherein, the target key-value pair is used to indicate merging the logical data block with the corresponding target base data block, and the logical data block is the modified data relative to the target base data block. In this way, a file write operation can be transformed into an incremental merging of only the changed blocks, completely skipping the necessary path of rewriting the entire file in related technologies. Specifically, in high-frequency, small-write scenarios, this application only needs to process the logical data blocks involved in the changes, rather than the entire file. This allows the system to complete persistence with only byte-level differences, eliminating the need to read back the original file and then write back the entire file. This compresses the I / O data volume from the file level to a partially incremental level, minimizing the write path and significantly reducing the amount and frequency of I / O operations. It also reduces write amplification and improves storage space utilization and write processing efficiency. In summary, this application effectively improves system performance and reduces write operation latency.

[0085] In some implementations, to optimize the incremental data merging process and achieve efficient resource utilization, the most suitable merging algorithm (merging function) can be dynamically selected based on the number of incremental data blocks to be merged (number of operations) when the merge is triggered. This allows for differentiated merging strategies to be adopted for modification scenarios with different levels of accumulation, thereby minimizing computational overhead and memory usage while ensuring the correctness of the merge and improving the overall efficiency of the merge operation. For example, after step 104, i.e., after "storing the target key-value pair", the process may further include: (B.1) in response to the merge trigger operation for the block identifier key, obtaining the target basic data block and at least one logical data block associated with the block identifier key; (B.2) obtaining the number of operations corresponding to at least one logical data block and determining the corresponding merging function based on the number of operations; (B.3) merging the target basic data block and at least one logical data block using the merging function to obtain the target data block.

[0086] The merge trigger operation can be an event or instruction that initiates the integration of all associated data under a specific block identifier key. For example, it can be an operation initiated by the underlying key-value storage system (such as RocksDB) during the compaction process, or when the system explicitly calls the merge interface, targeting that block identifier key.

[0087] The number of operations can be the number of logical data blocks (incremental data blocks) associated with the same block identifier key and waiting to be merged. For example, if there are 3 unmerged incremental writes for a certain target subfile location, the number of operations is 3.

[0088] The merge function can be an algorithm or function determined by the number of operations, used to specifically execute the data integration logic. For example, it can be a partial merge function, used to merge only the incremental data blocks of the target subfile (i.e., logical data blocks that are not the target base data block); or it can be a full merge function, called when the target base data block and the incremental data blocks need to be finally integrated.

[0089] The target data block can be a new data block generated by merging a target base data block and / or at least one logical data block using a merge function. For example, it can be an updated, complete data block containing all committed modifications, which will be stored as the new base data block, invalidating previously accumulated incremental data blocks; or it can be a larger data block obtained by merging multiple incremental data blocks.

[0090] In some implementations, a merge trigger is an event that initiates the consolidation of all accumulated data under a specific chunk identifier key. This operation is typically initiated by the background compaction process of the underlying key-value storage engine (such as RocksDB). Compaction is a critical maintenance operation for LSM-Tree structures, used to merge multiple SST files at different levels, eliminate duplicate or outdated data, and reclaim space. When the compaction process needs to process an SST file containing a specific chunk identifier key, it can trigger a merge operation for that key. Additionally, when a MemTable is flushed to disk to form an SST file, an initial merge of its chunk identifier keys can also be triggered.

[0091] Specifically, when performing a merge, the storage engine (RocksDB) searches for all values ​​associated with the corresponding chunk identifier key from the SST file (and possibly MemTable) currently participating in the compaction. These values ​​can include the target underlying data block and at least one logical data block.

[0092] Specifically, the target base data block can be the latest baseline state of the block identifier key in the currently persisted data, typically located in an older or lower-level SST file, and its flags are marked as base_block. Logical data blocks can be incremental modifications written via a Merge operation after the target base data block (if it exists), stored in an updated or higher-level SST file or MemTable, and their flags are marked as incremental_block.

[0093] Furthermore, the number of operations corresponding to all logical data blocks that need to be processed can be obtained, i.e., the number of logical data blocks that need to be processed in this merge. Then, the corresponding merge function is determined based on the number of operations. Specifically, if the number of operations is 2, a partial merge function can be selected, merging multiple logical data blocks into a larger logical data block without first reading the base data block; the target data block in this case is the merged logical data block. If the number of operations is greater than 2, a full merge function can be selected, writing multiple logical data blocks based on offsets on top of the base data block to obtain the target data block.

[0094] For example, when a partial merge function is called, it processes only multiple incremental data blocks. It reads the merge_offset and length of each incremental block, calculates a total merge range in memory, allocates a buffer, and merges the multiple incremental data blocks into a new, unified incremental data block by offset. This process does not read the target base data block.

[0095] Alternatively, when the full merge function is called, it performs the final integration. It first reads the target base data block and loads the valid data into the buffer according to the data_bitmap described earlier. Then, it sequentially writes each logical data block to the corresponding position in the buffer based on the merge_offset of each logical data block, and synchronously updates the valid data bitmap in the buffer. For areas covered by incremental blocks, the corresponding bits in the bitmap are set. The target data block generated after the merge has its flags set to base_block and carries the updated complete data_bitmap. This new key-value pair (Key is the original block identifier key, Value is the new target data block) is written by the storage engine to the new SST file generated by Compaction, thus completing the update and persistence of the data block's state.

[0096] By employing the above methods, the most suitable merging path can be intelligently selected based on the actual accumulation of incremental data. This avoids a full merge involving the underlying data when only a small amount of incremental data exists; and performs a complete integration when there are many incremental data or when persistence is required. This tiered merging strategy effectively reduces unnecessary data movement and memory allocation, lowering the latency of the merging operation itself and its consumption of system resources. Consequently, the system can smoothly and efficiently integrate scattered write operations in the background, maintaining the storage engine's high performance and low write amplification, ultimately improving the overall throughput and responsiveness of the file system when facing continuous high-frequency small writes.

[0097] In some implementations, to achieve efficient and lightweight merging of multiple incremental data blocks and avoid redundant I / O operations involving the basic data blocks, the actual continuous data interval (merging range) affected by each incremental block can be determined by analyzing the offset of each incremental block, and a memory buffer (first buffer) can be allocated only for this interval for data integration. This strictly limits the memory and computational overhead of the merging operation to the necessary range, thereby improving merging performance. For example, when the number of operations corresponding to the logical data block is 2, the merging function is a partial merging function, which (B.3) may include: (B.3.a1) reading the offset of each logical data block in the target subfile from the block data value of each logical data block using the partial merging function; (B.3.a2) calculating the merging range corresponding to at least one logical data block in the target subfile based on the offset of at least one logical data block; (B.3.a3) allocating the first buffer according to the merging range, and copying at least one logical data block to the first buffer for data merging based on the offset corresponding to each logical data block, to obtain the target data block stored in the first buffer.

[0098] Some merge functions can be specifically designed to merge logical data blocks that belong to the same block identifier key, without needing to read the merge function of the underlying data block. For example, when the number of operations indicates the existence of two logical data blocks, the system calls this function to sort and integrate these logical data blocks only by offset, generating a new logical data block that aggregates all modifications.

[0099] The merging range can be a continuous logical address interval determined by the offset and data length of at least one logical data block. For example, by calculating the minimum start offset and maximum end offset of all incremental data blocks to be merged, a continuous byte range covering all these modifications can be obtained, and the merge operation only needs to process the data within the merging range.

[0100] The first buffer can be a temporary memory storage area whose size matches the merging range, allocated for executing a partial merge function. It can be a contiguous buffer area allocated in memory based on the calculated merging range size (e.g., 8KB), used to sequentially write and integrate the contents of each logical data block, ultimately forming the merged target data block.

[0101] In some implementations, some merge functions are specifically designed for efficiently merging multiple logical data blocks. They can integrate multiple scattered incremental modifications into a larger, unified incremental modification block without reading the original target base data block.

[0102] Specifically, the header (block_head) of each logical data block contains flags and merge_offset fields. Some merge functions can extract the merge_offset value for each block by parsing the structure of these logical data blocks. For example, assuming there are two incremental data blocks, their block data values ​​(Value) can be parsed to yield offset1=4096 (indicating starting from the 4KB position of the target subfile) and offset2=12288 (indicating starting from the 12KB position), respectively.

[0103] Furthermore, all logical data blocks can be traversed to find the smallest starting offset, `min_offset`. Then, for each logical data block, its ending position, `end_pos`, is calculated as `offset` + `length`, where `length` is the length of the data area of ​​that logical data block. Finally, the maximum value among all `end_pos`, `max_end`, is found. The merging range can then be a continuous interval from `min_offset` to `max_end`, with a size `merge_range_size` = `max_end` - `min_offset`.

[0104] For example, if the first block of data is 4KB long and the second block of data is 8KB long, then min_offset = 4096, max_end = max(4096+4096, 12288+8192) = 20480. The merging range is from offset 4096 bytes in the target subfile to offset 20480 bytes, with a total size of 20480 - 4096 = 16384 bytes (16KB).

[0105] Furthermore, the partial merge function can allocate a contiguous first buffer in memory based on the calculated merge range size (merge_range_size). This buffer must be at least merge_range_size in size. Subsequently, the partial merge function can iterate through all logical data blocks.

[0106] For example, for each logical data block, the starting position where its data should be written to the first buffer can be calculated: buffer_start = offset - min_offset. This calculation converts the absolute offset of this logical data block in the target subfile into an offset within the first buffer relative to the start of the merge range. Then, the data portion of the logical data block's chunked data values, i.e., the logical data block itself, can be copied (or written) to the buffer_start position of the first buffer.

[0107] It should be noted that if the data ranges of multiple logical data blocks overlap, the data copied later will overwrite the data copied earlier, in order to achieve update semantics based on the write order.

[0108] After all logical data blocks have been copied to the first buffer, the first buffer stores a contiguous data block that integrates all incremental modifications; this is the target data block stored in the first buffer. At this point, if the target subfile contains a target base data block, this target data block can be considered a new, aggregated "incremental data block." To allow for potential future merging with the target base data block, the system can generate a new block data value for it. The block identifier key remains unchanged, and its merge_offset field can be set to min_offset; the data area is the content of the first buffer.

[0109] In this way, some merge functions can efficiently integrate multiple discrete incremental modifications into a contiguous block of memory through sophisticated range calculations and buffer management. This process is completed entirely in memory, avoiding any read I / O to the underlying data blocks on disk, greatly reducing the overhead of the merge operation itself. The resulting single data block, whether used as an intermediate state to participate in subsequent merges or ultimately merged with the target underlying data block, significantly simplifies the complexity of data processing and is a key means of optimizing merge performance in high-frequency write scenarios.

[0110] In some implementations, when the target subfile does not contain a target base data block, it means that the current write to that block identifier key is the first occurrence, and there is no historical data that needs to be merged. In this case, some merge functions (PartialMerge) can directly enter the process of "directly copying the data on the right to new_value". The data on the right is the data payload contained in the incremental data block (incremental_block) carried by this write. The system will completely copy this data to the new value (new_value) and generate or update the corresponding metadata according to its header information (such as merge_offset), thereby directly creating a new base data block for the target subfile. In this way, the processing efficiency in the first write scenario can be improved.

[0111] By employing the above method, multiple discrete incremental modifications targeting the same data location can be efficiently integrated within a minimal, contiguous space in memory. This avoids the disk I / O overhead required to read the original base data block for merging, and also avoids the waste caused by allocating memory space equal to the entire data block. Consequently, it significantly reduces the system resource consumption and latency of background merging operations, enabling the system to integrate write operations more quickly and frequently, effectively controlling the accumulation of incremental data, and providing a crucial guarantee for maintaining low-latency write performance under high concurrency.

[0112] In some implementations, to achieve eventual consistency and persistence of incremental and base data, a working memory area (second buffer) can be allocated to hold the complete data block content. The base data is then completely copied to this area, and modifications are applied sequentially according to the precise positions (offsets) specified for each incremental data block. This systematically integrates all cumulative updates within a controlled temporary space, ensuring the generation of a final, correct, and complete data block version. For example, the merge function can be a full merge function, (B.3) which may include: (B.3.b1) allocating the second buffer using the full merge function and copying the target base data block to the second buffer; (B.3.b2) when the target base data block contains logical data blocks to be merged, reading the offset of each logical data block in the target subfile from the block data value of each logical data block; (B.3.b3) copying each logical data block to the corresponding position in the second buffer according to its offset relative to the target subfile to modify the target base data block until no logical data blocks to be merged remain in the target subfile, resulting in the target data block stored in the second buffer.

[0113] The second buffer can be a temporarily allocated memory storage area large enough to hold the entire target base data block for executing the full merge function. For example, it could be a contiguous memory space of 64KB. First, the complete contents of the target base data block can be loaded, and then all logical data blocks to be merged (incremental data) are sequentially applied to the designated locations in the second buffer, ultimately forming the merged target data block.

[0114] In some implementations, the full merge function can be a merge function used to generate a final, complete version of the data block. When the number of operations is greater than two, it can integrate the target base data block contained in the target subfile with a series of logical data blocks to produce a completely new base data block representing the latest state, i.e., the target data block.

[0115] In some implementations, the full merge function can allocate a second buffer in memory. The size of the second buffer matches the logical size of the target subfile (e.g., fixed at 64KB). After allocating the second buffer, the contents of the target base data block can be completely copied into this second buffer as the initial state for the merge operation. The block data values ​​of the target base data block can contain a data_bitmap and a corresponding valid data area. During the copying process, valid data can be placed into the corresponding positions in the second buffer according to its logical position in the target subfile, based on the data_bitmap; for data elements marked as invalid (unset) in the target data bitmap, the corresponding area in the buffer can be initialized to a default value (e.g., all zeros).

[0116] In some implementations, the full merge function can continuously determine whether there are logical data blocks to be merged. If so, each logical data block is processed sequentially. For each logical data block, the full merge function can parse the offset (merge_offset) from the header of its block data values, which indicates the specific starting position where the current logical data block should be applied to the second buffer.

[0117] Furthermore, for each logical data block, its target starting address in the second buffer can be calculated based on its offset (merge_offset). Then, the data area content of the block data value of the logical data block is copied to the target starting address calculated in the second buffer, thereby overwriting or updating the original data in the corresponding area of ​​the second buffer.

[0118] Furthermore, the data_bitmap associated with the second buffer of the target subfile can be updated synchronously. Specifically, based on the range of data elements covered by this write (determined by the offset and data length), the corresponding bit in the data_bitmap can be set to the set state (1) to reflect that there is currently valid data in the corresponding area.

[0119] In some implementations, the processing described above for each logical data block can be performed cyclically until all logical data blocks to be merged have been applied sequentially to the second buffer. Ultimately, the target data block stored in the second buffer contains complete data that integrates all historical and current incremental modifications, along with an updated data_bitmap that accurately reflects the distribution of all valid data.

[0120] Furthermore, after the merge is complete, a new block data value, flagged as `base_block`, can be generated based on this target data block. This key-value pair (Key is the original block identifier key, Value is the new block data value) will be written to the currently active MemTable as a regular Put operation. When this MemTable is subsequently flushed, this new key-value pair will be sequentially written to the newly generated SST file, thus completing the persistent update of the data state. The old SST file, containing the original base data block and a series of incremental data blocks, will be gradually cleaned up during the storage engine's background compaction garbage collection process.

[0121] By using the above methods, a separate merge environment of the same size as the final storage target can be created in memory. This allows all subsequent incremental modifications to be efficiently and sequentially applied in memory after the base data is read out all at once, avoiding the complexity and potential errors of repeated partial updates in the storage engine. Furthermore, this ensures the absolute correctness and integrity of the merged data version, providing a reliable basis for writing this data back to persistent storage as the new base version and cleaning up historical incremental data.

[0122] In some implementations, to achieve efficient initialization of data blocks that are newly written or being overwritten for the first time, when it is determined that there is no corresponding historical base data block, the logical data blocks can be directly combined and filled into a new data block structure according to the offset information recorded by each logical data block itself, thereby directly generating a complete target sub-file. For example, the data storage method further includes: (C.1) when at least one logical data block does not have a corresponding target base data block in the target sub-file, reading the offset of each logical data block in the target sub-file from the block data value of each logical data block; (C.2) copying each logical data block to the corresponding position in the target sub-file according to the offset of each logical data block relative to the target sub-file, until there are no logical data blocks to be merged in the target sub-file, thus obtaining an updated target sub-file.

[0123] In some implementations, by querying the underlying key-value storage engine, it can be confirmed that there are no data blocks marked as `base_block` in the Value list associated with the current block identifier key, meaning there is no historical base data. In this case, it can be determined that the logical data block is the first write to the target subfile in the file system, or a full overwrite of an existing target subfile whose content is all default values ​​(such as all zeros). Therefore, the step of reading the old base data can be skipped, and the complete final data block can be directly constructed based on the logical data block written this time, thereby obtaining the updated target subfile.

[0124] Specifically, the internal offset (merge_offset) can be extracted from the block data value of each logical data block to be written. This offset indicates the specific starting position within the target subfile (a logically empty container, such as a 64KB address space) where the logical data block should be placed.

[0125] Subsequently, for each logical data block, its corresponding position in the target subfile's buffer can be determined based on its merge_offset. The data area content of the logical data block's segmented data value is then copied to the corresponding position in the buffer. This process continues until all logical data blocks to be merged under the same segment identifier key are placed in the correct positions in the buffer according to their respective offsets. Finally, the data stored in the memory buffer constitutes the complete content of the updated target subfile. At this point, a new flag marked as base_block can be generated for this data, and its data_bitmap will be set according to the actual range of data written (for example, the corresponding bit in the written area is set to 1, and the rest are 0). The data area is the content of this buffer.

[0126] Therefore, for initial write or full overwrite scenarios, the read path overhead in the merge operation can be completely eliminated, and the storage data of the target sub-file can be generated directly using the offset information of the incremental data block, thus achieving the most efficient write path.

[0127] In some implementations, to achieve high efficiency and scalability of file system directory operations, especially to optimize the performance of operations such as directory renaming, the full path mapping relationship of files can be decoupled into a hierarchical key structure (directory key) based on the parent directory identifier and the file name. This mapping relationship is then stored independently in a dedicated logical storage area (key-value storage column family). This enables independent management of directory metadata and file data, so that only a minimal amount of mapping relationship needs to be modified when the directory structure changes, avoiding large-scale data migration. For example, the data storage method may further include: (D.1) determining the target file, the parent directory corresponding to the target file, and the parent file identifier corresponding to the parent directory; wherein the target file includes multiple target subfiles; (D.2) obtaining the file name of the target file, and constructing the directory key of the target file based on the parent file identifier and the file name; (D.3) obtaining the file identifier of the target file, and storing the directory key and the file identifier in a key-value storage column family to establish a mapping relationship between the parent directory and the target file.

[0128] The parent directory can be the parent directory to which the target file directly belongs in the file system's directory tree structure. For example, if the file path is / home / user / data.txt, then the directory named user is the parent directory of the file data.txt.

[0129] The parent file identifier can be a unique identifier for the parent directory within the file system. For example, it could be the inode number of the parent directory itself, which is unique throughout the entire file system.

[0130] The directory key can be a key used to uniquely identify the mapping relationship between a target file and its parent directory in a key-value storage system. For example, it can be in the form of the string "<parent directory identifier>_<filename>", such as "12345_data.txt", through which the file identifier of the corresponding target file can be retrieved.

[0131] In this context, a key-value store column family can be an independent namespace or data set used in the underlying key-value store system to logically isolate and store specific types of data. For example, in RocksDB, a dedicated column family can be created to centrally store the mapping relationship between all directory keys and file identifiers, thus physically or logically separating it from the column family that stores file data blocks, facilitating independent management and optimization.

[0132] In some implementations, the content of a target file consists of multiple target sub-files (i.e., fixed-size data blocks), each of which can be independently addressed by a block identifier key.

[0133] In some implementations, when creating or locating a target file in the file system, it is first necessary to determine its directory context. Specifically, the filename of the target file (e.g., aa.jpg) and the path of its parent directory (e.g., / A / B) can be resolved based on the operation request (e.g., creating file / A / B / aa.jpg). Subsequently, the parent file identifier of the parent directory can be retrieved. This parent file identifier can be the unique inode number of the parent directory itself, system-wide.

[0134] For example, for the path / A / B / aa.jpg, its parent directory is B. The system query finds that the inode number of directory B is 2, so 2 is the determined parent file identifier (parent_dir_inode_no).

[0135] Furthermore, to efficiently and uniquely represent the mapping relationship of a file with a specific name in a parent directory, a dedicated directory key can be constructed. The directory key design abandons the traditional full path string and adopts a compact and easily computed hierarchical encoding method. Specifically, the directory key `file_name_key` can be constructed using the following formula: `file_name_key = concat(parent_dir_inode_no, '_', file_name)`, where `concat` is a string concatenation operation, `parent_dir_inode_no` is the string form of the obtained parent directory identifier, '_' is a selected separator (or other non-conflicting characters), and `file_name` is the name string of the target file. This ensures that within the same parent directory, different filenames will necessarily have different keys; and within different parent directories, even if the filenames are the same, their keys will also be different.

[0136] For example, if the inode number of the parent directory B is 2 and the target file name is aa.jpg, then the constructed directory key is "2_aa.jpg". In this way, the file path can be encoded into a short key value that is independent of the path depth, which greatly simplifies the storage and query structure of metadata.

[0137] In some implementations, to persist and query the above mapping relationship, the system needs to store the key-value pairs. First, the file identifier of the target file can be obtained, which is the unique inode number (inode_no) assigned to the target file by the system. For newly created target files, this number is generated by the inode allocator; for existing target files, it can be obtained by querying existing metadata.

[0138] Subsequently, the constructed `file_name_key` can be used as the key, and the `inode_no` of the target file as the value, written to a dedicated key-value store column family in the RocksDB storage engine. This key-value store column family is a logically independent data set specifically used to store all directory entry mappings, physically or logically isolated from the column family storing the actual target subfile content. This isolation allows for independent storage optimizations (such as different compression strategies and caching strategies) for metadata and data.

[0139] For example, if the target file aa.jpg is assigned inode number 3, then the key-value pair {Key:“2_aa.jpg”,Value:3} is atomically written to the key-value storage column family dedicated to directory mapping. When it is necessary to parse the target file, or the file path of any target subfile contained in the target file (e.g., / A / B / aa.jpg), it is only necessary to construct the same key “2_aa.jpg” again based on the parent directory inode (2) and the file name (aa.jpg), and its inode number 3 can be found in this column family. Then, all target subfiles contained in the target file can be accessed through a key of the form (inode_no,block_index).

[0140] Referring to Figure 4, for example, the process of renaming target file B to E, when renaming directory B (assuming the target file's inode is 2 and the parent directory's inode is 0) is required, first, the parent inode value is found to be 0. Then, the old directory key "0_B" is constructed based on parent_dir_inode_no and the old name, and the corresponding key-value pair (with a value of 2) is found in the directory mapping column family. Next, the mapping is updated, marking the old key-value pair {"0_B":2} as logically deleted (or directly replaced), and inserting the new key-value pair {"0_E":2}.

[0141] Therefore, all target subfiles (such as aa.jpg) and subdirectories under directory B (now E) follow the same rule for their directory keys: "parent directory inode_filename". Since the "parent directory inode" part of these sub-items is B's own inode (2), and renaming B to E does not change its own inode number (which remains 2). Therefore, the directory key of the sub-item aa.jpg is still "2_aa.jpg", and its mapping relationship remains completely unchanged.

[0142] Therefore, this application achieves complete decoupling between the logical location of a directory and its physical data by designing the mapping key in the form of (parent_inode, name). A single directory renaming operation has a constant cost, regardless of the number of subdirectories or millions of files within that directory, as it only involves modifying a single entry in the directory's parent directory. This fundamentally solves the performance bottleneck in related technologies that require recursive traversal and updating of massive amounts of metadata for lower-level entries during renaming due to the use of full paths as indexes, providing highly efficient metadata manipulation capabilities for ultra-large-scale file systems.

[0143] Please refer to Figure 5. In conjunction with Figure 5, in some embodiments, this application provides an overall processing flow for merging logical data blocks in a data storage method.

[0144] Specifically, when the storage engine triggers a merge operation for a specific block identifier key, it first determines the number of logical data blocks to be merged (i.e., operands). If there are two operands, the PartialMerge function is called for processing; if there are multiple operands (more than two), the FullMerge function is called for processing.

[0145] Furthermore, in some merge functions, the system first checks whether an existing baseline value (existing_value) already exists under the block identifier key. If it does, the header information of the two operands (left and right) is parsed, and the data range involved in this merge (start position begin and end position end) is calculated based on the offsets recorded therein. A buffer of the appropriate size is then allocated accordingly. Subsequently, the data from the left operand and the right operand are copied sequentially into this buffer to obtain the target data block (i.e., the data block obtained by merging multiple logical data blocks), and the header information of the merged data (such as merge_offset) is updated. If existing_value does not exist, it indicates that there is no historical baseline data, and the data from the right operand is directly used as the merge result (new_value).

[0146] Specifically, in the full merge function, a fixed-size buffer (e.g., the same size as the target subfile) is first allocated. Then, it checks if an existing baseline value exists. If it does, the baseline value is copied completely to the buffer as the starting point for the merge; otherwise, the copying step is skipped, and the buffer is initially empty or at its default value. Next, the system iterates through the list of all operands to be merged. For each operand: if its valid data length is not 0, its header is parsed to obtain the offset, and the data is copied to the corresponding position in the buffer according to that offset; if the operand length is 0, the operand is skipped, and the next one is processed. After the iteration is complete, the buffer stores the target data block after successful merging.

[0147] Through the above process, this application can intelligently select a merging strategy based on the amount of incremental data, ensuring data consistency while minimizing unnecessary read and write overhead, and improving storage efficiency and system performance in high-frequency, low-write scenarios.

[0148] Referring to Figure 6, this application embodiment also provides a data storage device that can implement the above-described data storage method. The data storage device includes: an acquisition module 61, used to acquire data to be written for a target sub-file and divide the data to be written into at least one logical data block; a determination module 62, used to acquire a file identifier and a file position index in the target sub-file, and determine a block identifier key for the target sub-file based on the file identifier and the file position index; a generation module 63, used to determine the modification attribute corresponding to each logical data block and the offset of each logical data block relative to the target sub-file, and generate a block data value corresponding to each logical data block by combining the logical data block, the modification attribute, and the offset; and a storage module 64, used to generate a target key-value pair for each logical data block based on the corresponding block identifier key and the block data value, and store the target key-value pair; wherein, the target key-value pair is used to indicate merging the logical data block with the corresponding target base data block, and the logical data block is the modification data relative to the target base data block.

[0149] The specific implementation of this data storage device is basically the same as the specific embodiments of the data storage method described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this application, the data storage device may also be equipped with other functional modules to implement the data storage method in the above embodiments.

[0150] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described data storage method. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0151] Please refer to Figure 7, which illustrates the hardware structure of a computer device according to another embodiment. The computer device includes: a processor 71, which can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing related programs to implement the technical solutions provided in the embodiments of this application; and a memory 72, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 72 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 72 and is called and executed by the processor 71 to implement the data storage method of the embodiments of this application. The input / output interface 73 is used to implement information input and output. The communication interface 74 is used to realize communication interaction between this device and other devices. Communication can be realized through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus 75 transmits information between the various components of the device (such as the processor 71, memory 72, input / output interface 73 and communication interface 74). The processor 71, memory 72, input / output interface 73 and communication interface 74 realize communication connection between each other within the device through the bus 75.

[0152] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data storage method.

[0153] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0154] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0155] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0158] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0159] It should be understood that in this application, "at least one" and "several" refer to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0160] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0161] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0162] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0163] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A data storage method, characterized in that, The method includes: acquiring data to be written for a target sub-file, dividing the data to be written into at least one logical data block; acquiring a file identifier and a file position index in the target sub-file, and determining a block identifier key for the target sub-file based on the file identifier and the file position index; determining a modification attribute corresponding to each logical data block, and determining an offset of each logical data block relative to the target sub-file, and generating a block data value corresponding to each logical data block by combining the logical data block, the modification attribute, and the offset; for each logical data block, generating a target key-value pair based on the corresponding block identifier key and the block data value, and storing the target key-value pair; wherein the target key-value pair is used to indicate merging the logical data block with the corresponding target base data block, and the logical data block is the modification data relative to the target base data block.

2. The data storage method according to claim 1, characterized in that, After storing the target key-value pair, the method further includes: in response to a merge trigger operation for the block identifier key, obtaining a target basic data block and at least one logical data block associated with the block identifier key; obtaining the number of operations corresponding to the at least one logical data block, and determining a corresponding merge function based on the number of operations; and merging the target basic data block and the at least one logical data block using the merge function to obtain a target data block.

3. The data storage method according to claim 2, characterized in that, The merge function is a partial merge function. The process of merging the target base data block and the at least one logical data block using the merge function to obtain the target data block includes: reading the offset of each logical data block in the target subfile from the block data value of each logical data block using the partial merge function; calculating the merge range corresponding to the at least one logical data block in the target subfile based on the offset of the at least one logical data block; allocating a first buffer according to the merge range; and copying the at least one logical data block to the first buffer for data merging based on the offset corresponding to each logical data block, thereby obtaining the target data block stored in the first buffer.

4. The data storage method according to claim 2, characterized in that, The merge function is a full merge function. The process of merging the target base data block and the at least one logical data block using the merge function to obtain the target data block includes: allocating a second buffer using the full merge function and copying the target base data block to the second buffer; when the target base data block contains logical data blocks to be merged, reading the offset of each logical data block in the target subfile from the block data value of each logical data block; and copying each logical data block to the corresponding position in the second buffer according to the offset of each logical data block relative to the target subfile to modify the target base data block until the target subfile no longer contains logical data blocks to be merged, thus obtaining the target data block stored in the second buffer.

5. The data storage method according to claim 1, characterized in that, The method further includes: when the at least one logical data block does not have a corresponding target base data block in the target sub-file, reading the offset of each logical data block in the target sub-file from the block data value of each logical data block; copying each logical data block to the corresponding position in the target sub-file according to the offset of each logical data block relative to the target sub-file, until the target sub-file does not have any logical data blocks to be merged, thereby obtaining an updated target sub-file.

6. The data storage method according to claim 1, characterized in that, Before generating the segmented data value corresponding to each logical data block by combining the logical data block, the modified attribute, and the offset, the method further includes: obtaining an initial data bitmap corresponding to the target sub-file; determining multiple valid data elements and invalid data elements from multiple data elements contained in the target sub-file; determining a first distribution position corresponding to each valid data in the initial data bitmap, and a second distribution position corresponding to each invalid data in the initial data bitmap; setting the first distribution position to a set state and setting the second distribution position to an unset state in the initial data bitmap to obtain a target data bitmap; then, generating the segmented data value corresponding to each logical data block by combining the logical data block, the modified attribute, and the offset includes: generating the segmented data value corresponding to each logical data block by combining the target data bitmap, the logical data block, the modified attribute, and the offset.

7. The data storage method according to claim 1, characterized in that, The method further includes: determining a target file, a parent directory corresponding to the target file, and a parent file identifier corresponding to the parent directory; wherein the target file includes multiple target sub-files; obtaining the file name of the target file, and constructing a directory key for the target file based on the parent file identifier and the file name; obtaining the file identifier of the target file, and storing the directory key and the file identifier in a key-value storage column family to establish a mapping relationship between the parent directory and the target file.

8. A data storage device, characterized in that, The apparatus includes: an acquisition module, configured to acquire data to be written for a target sub-file, and divide the data to be written into at least one logical data block; a determination module, configured to acquire a file identifier and a file position index in the target sub-file, and determine a block identifier key for the target sub-file based on the file identifier and the file position index; a generation module, configured to determine a modification attribute corresponding to each logical data block, and determine an offset of each logical data block relative to the target sub-file, and generate a block data value corresponding to each logical data block by combining the logical data block, the modification attribute, and the offset; and a storage module, configured to generate a target key-value pair for each logical data block based on the corresponding block identifier key and the block data value, and store the target key-value pair; wherein the target key-value pair is used to indicate merging the logical data block with the corresponding target base data block, and the logical data block is the modification data relative to the target base data block.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data storage method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data storage method according to any one of claims 1 to 7.

Images