Write-time redirection data mapping methods, apparatuses, and devices

Abstract

The application provides a write-time redirection data mapping method, device and equipment, and is used for solving the technical problem of low I / O storage performance in a distributed system. The application adds a mapping record of an extension block and a row object in a write-time redirection database rowdb, introduces an extension block bitmap and a reflection mapping relationship state, and realizes the purposes of mixed mapping of a size data block of a user object and garbage release, so as to improve the write performance of large I / O and the garbage release efficiency.

Description

Technical Field

[0001] This invention relates to the fields of communication and cloud computing technology, and in particular to a write-time redirection data mapping method, apparatus, and device. Background Technology

[0002] In enterprise-level software-defined storage (SDS) distributed scenarios, a three-tier architecture has gradually become the mainstream solution for various vendors. From top to bottom, the three-tier architecture consists of: a protocol client, an acceleration layer, and a persistence layer. The protocol client is primarily responsible for representing different protocol semantics for blocks, files, and objects. It transmits user-issued input / output (I / O) requests over the network to the acceleration layer. The acceleration layer uses Redirect on Write (ROW) technology to redirect user objects to ROW objects, thereby transforming small, random overwrite I / Os into large, sequential append I / Os written to the persistence layer. This improves space utilization and write performance in erasure coding (EC) scenarios.

[0003] User objects refer to high-level data objects used by applications, such as files, documents, and images, while ROW objects refer to the underlying data objects stored in a distributed object storage system. User objects are mapped to underlying ROW objects according to certain logical rules; the purpose of this mapping is to achieve distributed storage and management of data.

[0004] Because the ROW write-time redirection technique transforms overwrite writes into sequential append writes, a garbage collection (GC) mechanism is needed to reclaim the overwritten data. Currently, commonly used GC techniques require garbage identification for data of a fixed size, so ROW needs to redirect the data according to a fixed-size base block (usually 8K).

[0005] In real-world user scenarios, there are often mixed I / O writes of varying sizes. To ensure proper garbage collection in these mixed-write scenarios, the Row on Write (ROW) needs to be redirected to a uniform size. Therefore, large I / Os (e.g., 1MB) also need to be managed at the basic block granularity. Current technology involves dividing large I / Os of user objects into smaller I / Os at the basic block granularity, and then redirecting these smaller I / Os to ROW objects via ROW. This ensures sequential append writing to the persistent layer and guarantees proper garbage collection by the GC. However, dividing large I / Os into smaller I / Os at the basic block granularity increases the processing latency and CPU overhead of large I / Os, impacting storage performance. Summary of the Invention

[0006] In view of this, the present invention provides a write-time redirection data mapping method, apparatus, and device to solve the technical problem of low large I / O storage performance in distributed systems.

[0007] Based on one aspect of the embodiments of the present invention, the present invention provides a write-time redirection data mapping method, the method comprising:

[0008] When a redirection operation request for writing data to a user object is received, the mapping state is determined based on the state machine to determine the mapping relationship between the data blocks within the extended block range of the user object and the write-time redirection object (row object).

[0009] When writing data blocks as extended blocks (pass-through write), the mapping state is switched to extended block independent mapping state. During the write, the metadata recorded in the database (rowdb) is redirected, including the mapping relationship between extended blocks and row objects, the mapping state, and the corresponding garbage marking processing is performed.

[0010] When writing a data block as a base block and currently in an independent mapping state for extended blocks, the position of the written base block in the extended block is identified by the extended block bitmap. The mapping state is then switched to a mixed mapping state. The metadata recorded in rowdb includes: the mapping relationship between the written base block and the row object, the extended block bitmap, the mapping state, and the corresponding garbage marking process.

[0011] Furthermore, the method further includes: when the written data block is a base block and the current state is a hybrid mapping state, using an extended block bitmap to identify the position of the written base block in the extended block;

[0012] When the base block covers all extended blocks, switch the mapping state to base block independent mapping state, delete the mapping relationship between the extended block where the base block is located and the row object in rowdb, record the mapping relationship between the base block and the row object, update the extended block bitmap and perform the corresponding garbage marking process;

[0013] If the base block does not cover all the extended blocks, the mapping state remains unchanged. The mapping relationship between the base block and the row object is recorded in the rowdb, the extended block bitmap is updated, and the corresponding garbage marking process is performed.

[0014] Furthermore, the mapping relationship recorded in rowdb for an extended block of a user object is related to the mapping status;

[0015] In the independent mapping state of the base block, the mapping relationship recorded in rowdb only includes the mapping relationship between the base block and the row object;

[0016] In the state of independent mapping of extended blocks, the mapping relationship recorded in rowdb only includes the mapping relationship between extended blocks and row objects;

[0017] In a hybrid mapping state, the mapping relationships recorded in rowdb include both the mapping relationship between the base block and the row object and the mapping relationship between the extended block and the row object.

[0018] Furthermore, the execution of the corresponding garbage marking process includes:

[0019] When the extension block is overridden, the garbage amount and garbage bitmap are directly marked for the row object in the redirection operation request processing flow;

[0020] When the base block is overwritten, the garbage disposal mechanism identifies and marks garbage data through the rowdb compaction process.

[0021] Furthermore, in the redirection operation request processing flow, metadata update operations, including writing mapping relationship records (rowkv), deleting mapping relationship records, updating extended block bitmaps, and updating one or more of the mapping status, are placed into a transaction and committed to rowdb for persistence.

[0022] Based on another aspect of the embodiments of the present invention, the present invention also provides a write-time redirection data mapping apparatus, the apparatus comprising:

[0023] The judgment module is used to determine the mapping state of the mapping relationship between the data blocks within the extended block range of the user object and the write-time redirection object (row object) based on the state machine when a redirection operation request for writing data to the user object is received.

[0024] The state machine module is used to maintain the state machine and perform the switching of mapped states;

[0025] The bitmap maintenance module is used to maintain the extended block bitmap;

[0026] The garbage marking module is used to perform corresponding garbage marking processing in the redirection operation request according to the state machine switching method;

[0027] The metadata maintenance module is used to maintain metadata related to write-time redirection mapping relationships in the write-time redirection database (rowdb).

[0028] When writing data blocks as extended blocks (pass-through write), the state machine module is used to switch the mapping state to an extended block independent mapping state; the metadata recorded in rowdb by the metadata maintenance module includes: the mapping relationship between extended blocks and row objects, and the mapping state;

[0029] When writing a data block as a base block and currently in an independent mapping state for extended blocks, the bitmap maintenance module uses an extended block bitmap to identify the position of the written base block in the extended block; the state machine module is used to switch the mapping state to a mixed mapping state; the metadata recorded in the rowdb by the metadata maintenance module includes: the mapping relationship between the written base block and the row object, the extended block bitmap, and the mapping state.

[0030] Furthermore, when the data block being written is a base block and the current state is a hybrid mapping state, the bitmap maintenance module is used to identify the position of the base block being written in the extended block using an extended block bitmap;

[0031] When the base block covers all extended blocks, the state machine module is used to switch the mapping state to the base block independent mapping state; the metadata maintenance module is used to delete the mapping relationship between the extended block where the base block is located and the row object in the rowdb, record the mapping relationship between the base block and the row object, and update the extended block bitmap.

[0032] The mapping state remains unchanged if the base block does not cover all the extended blocks; the metadata maintenance module is used to record the mapping relationship between the base block and the row object in the rowdb and update the extended block bitmap.

[0033] Furthermore, the mapping relationship recorded in rowdb for an extended block of a user object is related to the mapping status;

[0034] In the independent mapping state of the base block, the mapping relationship recorded in rowdb by the metadata maintenance module only includes the mapping relationship between the base block and the row object;

[0035] In the state of independent mapping of extended blocks, the mapping relationship recorded in rowdb by the metadata maintenance module only includes the mapping relationship between extended blocks and row objects;

[0036] In the hybrid mapping state, the mapping relationships recorded in rowdb by the metadata maintenance module include both the mapping relationship between the base block and the row object and the mapping relationship between the extended block and the row object.

[0037] Furthermore, when the extended block is overridden, the garbage marking module directly marks the garbage amount and garbage bitmap for the row object in the redirection operation request processing flow; when the base block is overridden, the garbage handling mechanism identifies and marks garbage data through the rowdb compaction process.

[0038] Based on another aspect of the embodiments of the present invention, the present invention also provides an electronic device, including a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium communicate with each other through the communication bus;

[0039] Storage medium used to store computer programs;

[0040] When a processor executes a computer program stored on a storage medium, it implements the write-time redirection data mapping method provided by the present invention.

[0041] The device provided by this invention can be implemented in whole or in part as a software module, a hardware unit, or a combination of both. When implemented as a software module, the program code of the software module is loaded into the storage medium of the electronic device, and the processor reads and executes the program code from the storage medium. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained from these drawings of the embodiments of the present invention.

[0043] Figure 1 This is a schematic diagram of a state machine reflecting the state of data block mapping relationships of user objects, provided in an embodiment of the present invention.

[0044] Figure 2 This is a flowchart illustrating the steps of a write-time redirection data mapping method according to an embodiment of the present invention;

[0045] Figure 3 This is a flowchart of the data mapping method steps in the case of new extension block writing according to an embodiment of the present invention;

[0046] Figure 4 This is a flowchart of the data mapping method steps in the case of a newly written basic block according to an embodiment of the present invention;

[0047] Figure 5 This is a flowchart of the data mapping method steps in the case of an extended block overwriting a base block in one embodiment of the present invention;

[0048] Figure 6 This is a flowchart of the data mapping method steps in one embodiment of the present invention for incremental writing of the base block or writing of the base block overwriting the base block;

[0049] Figure 7 This is a flowchart of the data mapping method steps in the case of extension block overwriting extension block writing according to an embodiment of the present invention;

[0050] Figure 8 This is a flowchart of the data mapping method steps in an embodiment of the present invention when the base block is overwritten in the extended block.

[0051] Figure 9 This is a flowchart of the data mapping method steps for switching from a mixed write state to an independent write state of the extended block in one embodiment of the present invention;

[0052] Figure 10 This is a flowchart of the data mapping method steps in a mixed write state when writing to a base block according to an embodiment of the present invention;

[0053] Figure 11 This is a schematic diagram of an electronic device structure for implementing the write-time redirection data mapping method provided by the present invention, according to an embodiment of the present invention. Detailed Implementation

[0054] The terminology used in this embodiment of the invention is for the purpose of describing specific embodiments only and is not intended to limit the embodiments of the invention. The singular forms “a,” “the,” and “the” used in this embodiment are also intended to include the plural forms unless the context clearly indicates otherwise. It should be understood that although the terms first, second, third, etc., may be used to describe various information in this embodiment, this information should not be limited to these terms. These terms are only used to distinguish similar information, entities, or steps, and not to describe a specific order or sequence. For example, without departing from the scope of this embodiment, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Furthermore, the word “if” used can be interpreted as “when…”, “when…”, or “in response to a determination.” The term “and / or” in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. And, in the description of this invention, unless otherwise stated, “multiple” means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0055] In distributed object storage systems based on write-on-redirection (ROW) technology, large application-layer objects are typically divided into user objects of a fixed size, and the metadata service records the metadata of these user objects. Since garbage collection in distributed object storage systems is usually based on a fixed base block granularity (e.g., 8KB), when the storage engine receives a large I / O (e.g., 1MB) write from an upper-layer application, it typically divides the large I / O into smaller I / Os according to the base block granularity, and then redirects these smaller I / Os to ROW objects using ROW redirection technology, thus ensuring sequential append writing to the persistent layer. While dividing large I / Os according to base block granularity before ROW redirection ensures that the garbage collection mechanism can reclaim garbage data normally, it increases the processing latency and CPU overhead of large I / Os, impacting the storage performance of large I / O data.

[0056] To address the aforementioned technical problems, this invention proposes a write-time redirection data mapping scheme to solve the low performance of large I / O storage in distributed storage systems. The basic idea of ​​this invention is to add a mapping record between extended blocks and row objects in the write-time redirection database rowdb, and introduce an extended block bitmap and a method to reflect the mapping relationship status to achieve the purpose of mixed mapping of large and small data blocks of user objects and garbage release, thereby improving the write performance and garbage release efficiency for large I / O.

[0057] First, some technical terms involved in the embodiments of this invention will be explained:

[0058] Redirect on Write (ROW): Redirect on write is a data management and storage technique commonly used in distributed file systems and distributed storage systems. When performing a write operation on a user object using redirect on write, the data is not directly overwritten at the original location. Instead, the write operation is redirected to a new location, and the overwritten data at the original location is marked as garbage data and awaits garbage collection.

[0059] The write-time redirection key-value database rowdb is a metadata database used to persistently record the aggregation mapping relationship between user objects and write-time redirection objects (row objects). rowdb typically uses a high-performance key-value database (such as RocksDB, HyperLevelDB, etc.). In rowdb, the record recording the mapping relationship between user object data blocks and row object data blocks includes two fields: key and value. In this embodiment, this key-value pair record is simply referred to as rowkv, the key field is simply referred to as rowkey, and the value field is simply referred to as rowvalue. For example, the rowkv recording the mapping relationship between a 1M data block of a user object and a 1M data block of a row object is expressed as 1M_rowkv, the key field in 1M_rowkv is expressed as 1M_rowkey, and the value field in 1M_rowkv is expressed as 1M_rowvalue. In a distributed storage system, rowdb can be used to aggregate small I / O operations of user objects into large row objects and write them to the storage system in a large I / O write format, thereby improving storage performance. ROWDB itself also uses ROW technology, which requires a compaction mechanism to identify which records with the same rowkey have been overwritten, thereby identifying garbage in the row object.

[0060] In the rowdb of this embodiment of the invention, there are both key-value records rowkv based on basic blocks and key-value records rowkv based on extended blocks. The key field of rowkv includes information such as the user object name, data block type (basic block type or extended block type), and the offset of the data block within the user object. The value field includes information such as the row object name and the offset of the data block within the row object.

[0061] Application layer objects: These refer to logical data objects used by users at the application layer, such as text files, video files, image files, and database records.

[0062] User object: refers to the basic unit for the storage engine to process application layer object data. A large application layer object can be divided into multiple user objects, and the amount of user object data that the storage engine can process has an upper limit. In this embodiment of the invention, the upper limit of user object data size is 4M, and the basic block size in the user object is 8k, as an example.

[0063] Row objects, also known as write-on-redirect objects, are a data storage model that uses write-on-redirect technology to append data to a new location during writing, ensuring data consistency and improving write performance. Distributed storage systems improve storage performance by aggregating small I / O operations of user objects to row objects and then using row objects to perform large I / O writes.

[0064] Extended block: refers to a large data block division unit in a user object. A user object is divided into N extended blocks, and each extended block is further divided into M basic blocks, where N and M are system configuration values. In this embodiment of the invention, an example of an extended block of 1M and a basic block of 8k is used for illustration.

[0065] Basic block: refers to the smallest data block segment in a user object. It is the basic unit of garbage collection. One basic block corresponds to a binary flag bit in the extended block bitmap.

[0066] Extended Block Bitmap: Used to mark whether each base block contained in an extended block stores data in a bitmap format. The binary bits in the bitmap correspond one-to-one with the base blocks in sequence. A value of 1 indicates that data has been written to the corresponding base block position, and a value of 0 indicates that no data has been written. Each extended block of a user object corresponds to one extended block bitmap. In this embodiment of the invention, the extended bitmap is persisted in rowdb as metadata.

[0067] Garbage Bitmap: The garbage bitmap is used to identify garbage data in the garbage collection mechanism. One row object corresponds to one garbage bitmap. Each bit in the garbage bitmap corresponds to a storage space of one basic block size. A binary bit of 1 indicates that the corresponding storage space contains garbage data that needs to be reclaimed by the garbage collection mechanism. For example, a 4MB user object's corresponding row object would have a 512-bit garbage bitmap.

[0068] Based on the basic idea of ​​the present invention, the present invention provides the following embodiments. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0069] Figure 1 This is a schematic diagram of a state machine reflecting the data block mapping relationship of a user object, provided in an embodiment of the present invention. In this invention, a state machine reflecting the data block mapping relationship within the scope of each extended block of each user object is provided. By using this state machine in conjunction with the extended block bitmap corresponding to each extended block, it is possible to know which user object data blocks and row objects are stored in rowdb.

[0070] In this invention, there are three main mapping states: independent mapping state of basic blocks (state number 01), independent mapping state of extended blocks (state number 10), and mixed mapping state (state number 11). The first digit from the left in the state number corresponds to the 1M_written flag in the later specific embodiments, and the second digit from the left corresponds to the 8k_written flag in the later specific embodiments. When it is 1, it indicates that the corresponding flag is logically true, and when it is 0, it indicates that the corresponding flag is logically false.

[0071] The storage engine switches the state machine to the next state based on the type of data block written in the currently received redirection operation (ROW op) request and the state of the state machine corresponding to the extended block of the user object recorded in rowdb.

[0072] Figure 2 This is a flowchart illustrating the steps of a write-time redirection data mapping method according to an embodiment of the present invention. The method is invoked by the storage engine when processing a write request for a user object. The storage engine invokes this method to aggregate small I / O redirections for the user object into large I / O writes to the persistence layer for a row object. The method includes the following steps:

[0073] Step 201. Receive a redirection operation request for writing data to a user object;

[0074] When the storage engine receives a redirection request for writing data to a user object, if it's writing a new user object or a new extension block of a user object, it creates a state machine corresponding to the extension block. If it's not writing a new user object, it needs to read the user object's metadata from the metadata service and retrieve the current mapping state of the extension block's state machine from the metadata.

[0075] Step 202. Determine the mapping state of the mapping relationship between data blocks within the extended block range of the user object and the write-time redirection object (row object) based on the state machine;

[0076] Step 203. When writing data blocks as extended blocks (pass-through write), switch the mapping state to extended block independent mapping state, redirect the metadata recorded in the database (rowdb) during writing, including: the mapping relationship between extended blocks and row objects, the mapping state, and perform corresponding garbage marking processing;

[0077] This invention refers to writing the size of the extended block as a pass-through write. A pass-through write requires recording the mapping relationship between the extended block and the row object in the rowdb. This invention no longer splits the extended block into basic blocks and records the mapping relationship between each basic block and the row object, but directly records the mapping relationship between the entire extended block and the rowdb object.

[0078] For example, in common practices in this field, a 1M-sized extended block needs to be split into 128 8K-granular basic blocks when writing, and 128 mapping relationship records from basic blocks to row objects need to be recorded in rowdb. Although 128 rowkv are written in one transaction, the time complexity is O(128). However, by adopting the technical solution of this invention, the mapping relationship between the 1M-granular extended block and the row object is established directly, and only one mapping record needs to be recorded, with a time complexity of O(1). For scenarios with a large number of pass-through writes, storage efficiency can be significantly improved.

[0079] Step 204. When the written data block is a base block and the current state is an independent mapping state of the extended block, the position of the written base block in the extended block is identified by the extended block bitmap. The mapping state is switched to a mixed mapping state. The metadata recorded in rowdb includes: the mapping relationship between the written base block and the row object, the extended block bitmap, the mapping state, and the corresponding garbage marking process.

[0080] When the write granularity is based on the block size, this invention needs to combine the extended block bitmap established for the extended block where the write base block is located to identify the position of the write base block in the extended block, and needs to record the mapping relationship between the base block and the row object and the extended block bitmap in the rowdb.

[0081] based on Figure 1 As can be seen from the state machine, when the base block overwrite is received for the base block in the independent mapping state of the extended block, the state machine will switch to the mixed mapping state. In the mixed mapping state, the mapping relationship recorded in rowdb includes both the mapping relationship between the base block and the row object and the mapping relationship between the extended block and the row object.

[0082] Based on the steps described above, the write-time redirection data mapping method also includes the following steps:

[0083] Step 205. When the data block being written is a base block and the current state is a hybrid mapping state, the position of the base block being written in the extended block is identified by the extended block bitmap, and different processing is performed depending on whether the extended block is full.

[0084] When the base block covers all extended blocks, switch the mapping state to base block independent mapping state, delete the mapping relationship between the extended block containing the base block and the row object in the rowdb, record the mapping relationship between the base block and the row object, update the extended block bitmap and perform the corresponding garbage marking process.

[0085] If the base block does not cover all the extended blocks, the mapping state remains unchanged. The mapping relationship between the base block and the row object is recorded in the rowdb, the extended block bitmap is updated, and the corresponding garbage marking process is performed.

[0086] based on Figure 1 As can be seen from the state machine, in the mixed mapping state, the state machine will only switch to the basic block independent mapping state after the basic block has filled the entire extended block. Correspondingly, the invalidated mapping relationship between the extended block and the row object will be deleted from the rowdb. In the basic block independent mapping state, the mapping relationship recorded in the rowdb only includes the mapping relationship between the basic block and the row object.

[0087] The following embodiments are described using the example of a user object consisting of four 1M extension blocks and each base block consisting of 8k blocks. The state machine uses two extension block write flags, namely 1M_written and base block write flags, namely 8k_written, to mark the state.

[0088] Figure 3 This is a flowchart of the data mapping method steps in the case of new extension block writing according to an embodiment of the present invention.

[0089] Step 301. Upon receiving the redirection operation request (ROW op), perform redirection processing to generate the mapping relationship record rowkv and persist the row object;

[0090] After receiving a redirection operation request, the storage engine performs multiple small I / O operations to row object redirection and aggregation, aggregating the written data of the user object (user_obj) into the row object (p_obj). The aggregated row object is then appended and persisted in one go, reducing the number of I / O operations.

[0091] After the row object is written to the persistence layer, the persistence of metadata related to the write-time redirection mapping relationship still needs to be performed. In a distributed storage system, the database rowdb that persists the metadata related to the write-time redirection mapping relationship can be a key-value database like rocksdb, or other types of databases, as long as the purpose of the invention can be achieved. This invention does not limit the specific database type.

[0092] The storage engine performs a persistence operation on the mapping metadata for each aggregated user object data block, and performs subsequent metadata persistence steps based on the size of the written data block.

[0093] Step 302. Determine if the current state is a newly written extension block (1M new write)? If the determination is yes, proceed to step 303; if the determination is no, determine other state transition scenarios.

[0094] In cases where a new user object is created, the state machine (1M_written and 8k_written) is not read from rowdb, or the state machine is in its initial state, it can be determined that the current write is new. Then, based on whether the size of the written data block is 1M or 8k, it can be determined whether it is a new extension block (1M new write) or a new base block (8k new write).

[0095] In the initial state, the default values ​​for 1M_written and 8k_written are both 0, which is equivalent to the boolean value false.

[0096] Step 303. Since this is a pass-through new write, we only need to mark 1M_written as true for the currently written extended block, without marking the extended block bitmap.

[0097] This step is equivalent to switching the state machine of the currently written extension block from the initial state 00 to the extension block independent mapping state 10. Since it does not involve recording the 8k mapping relationship of the base block, there is no need to mark and record the extension block bitmap of the currently written extension block.

[0098] Step 304. Transactionally persist the mapping relationship record 1M_rowkv and mapping status 10 to the distributed rowdb.

[0099] After the transactional writing of mapping relationship records and mapping status to rowdb, the memory cache of metadata also needs to be updated synchronously. For the sake of simplicity, the description is omitted in this figure and subsequent figures.

[0100] The following is a comparative description of the changes in metadata in rowdb caused by new writes of extension blocks:

[0101] State conditions: The state machine is in the initial state 00, and there are no state records for 1M_written and 8K_written in rowdb.

[0102] State change: A 1M new write causes the state machine to switch from the initial state 00 to the extended block independent write state 10. The extended block independent write state 10 is marked as 1M_written = true and 8K_written = false.

[0103] Bitmap changes: The 1M new write is a pass-through write of the entire extended block, so it does not involve changes to the extended block bitmap (or simply bitmap) and does not require the creation of a bitmap.

[0104] Mapping relationship changes: Generate and record 1M_rowkv metadata of extended block mapping relationship. Since the basic block mapping relationship record 8K_rowkv is not involved, there is no need to record 8K_rowkv.

[0105] Persistence: Transactional write of 1M_rowkv metadata + mapping state 1M_written = true (equivalent to writing state 10).

[0106] Figure 4 This is a flowchart of the data mapping method steps in the case of rewriting the basic block in one embodiment of the present invention.

[0107] Step 401. Upon receiving the redirection operation request (ROW op), perform the redirection process to generate the mapping relationship record rowkv and persist the row object;

[0108] Step 402. Determine if the current operation is a newly written base block (8k new write)? If the determination is yes, proceed to step 403.

[0109] Step 403. Create an extended block bitmap and mark the binary bits corresponding to 8k to be written, and set 8k_written in the state machine to 1;

[0110] Since it is a basic block write, a bitmap of the extended block needs to be created if no bitmap exists. This bitmap is used to mark the write position of the 8k basic block being written. The binary bit corresponding to the 8k basic block being written in this bitmap is set to 1, and the other bits are set to 0 by default.

[0111] This step is equivalent to switching the state machine of the currently written extension block from the initial state 00 to the basic block independent mapping state 01.

[0112] Step 404. Transactionally write the mapping relationship record 8k_rowkv, mapping state 01, and extended block bitmap to the distributed rowdb.

[0113] The following is a comparative description of the changes in metadata in rowdb caused by writing base blocks:

[0114] State conditions: The state machine is in the initial state 00, and there are no state records for 1M_written and 8K_written in rowdb.

[0115] State change: The 8k new write causes the state machine to switch from the initial state 00 to the basic independent write state 01;

[0116] Bitmap transformation: A bitmap needs to be generated, and the corresponding 8K position in the bitmap needs to be marked as 1;

[0117] Mapping relationship changes: Generate and record the basic block mapping relationship record 8K_rowkv metadata;

[0118] Persistence: Transactional writing of 8K_rowkv metadata + mapping state 8K_written = true (equivalent to writing state 01) + bitmap.

[0119] Figure 5 This is a flowchart of the data mapping method steps in the case of an extended block overwriting a base block in one embodiment of the present invention. Figure 5 This corresponds to the data mapping process of the state machine from the independent write state 01 of the basic block to the independent write state 10 of the extended block. In this process, the write of the 1M extended block granularity will overwrite the previous mapping relationship of the write of the 8k basic block granularity.

[0120] Step 501. Upon receiving the redirection operation request (ROW op), obtain the user object-level granular metadata, perform redirection processing to generate the mapping relationship record rowkv and persist the row object;

[0121] This step requires first obtaining the user object's metadata to understand the mapping relationship before writing, such as reading the mapping status and bitmap from rowdb. Other operations in this step are detailed in the preceding steps and will not be repeated here.

[0122] Step 502. Determine whether the current situation is an extended block overwriting the base block write, i.e., the state machine switching from 01 to 10, based on the current mapping state of the state machine and the write block granularity. If the determination is yes, then proceed to step 503.

[0123] Step 503. Mark 1M_written = true, delete 8K_written, and perform a state switch;

[0124] Step 504. Traverse the bitmap, generate the rowkey of all covered 8k base blocks, and clear the bitmap;

[0125] The purpose of generating the key field rowkey in the mapping record of the overwritten 8k base block is to delete the mapping record of the overwritten 8k base block based on rowkey.

[0126] Step 505. The transaction will delete 8k_rowkv, write 1M_rowkv, map state 10, and commit the bitmap to the distributed rowdb for persistence.

[0127] The following is a comparative description of the changes in metadata caused by writing a 1M extended block:

[0128] State conditions: Before writing, the state machine is in state 01 when writing to the basic block independently. There is a bitmap in rowdb and only 8K_rowkv with bitmap position 1. There is no 1M_rowkv record.

[0129] State change: Update 8K_written in rowdb to false; set 1M_written to true, which is equivalent to the state machine switching from basic independent write state 01 to extended block independent write state 10;

[0130] Bitmap changes: Since the 1M extension block covers all 8k base blocks, all positions in the bitmap that are 1 need to be changed to 0;

[0131] Mapping relationship changes: It is necessary to generate 8k_rowkey based on the bitmap, delete all mapping relationship records of the overwritten base blocks, and add 1M_rowkv for writing;

[0132] Persistent transaction: Transactional write 1M_rowkv + 1M_written = true and 8K_written = false (equivalent to write state 10) + delete the overwritten 8K_rowkv + update the bitmap (all 0s).

[0133] Since the 1M extended block covers the data of the 8k base block, this process involves a garbage data marking step. When the base block is covered, the garbage handling mechanism will identify and mark garbage data according to the rowdb compaction process. Through the compaction operation, it can be known which records have been deleted, and then garbage data in the row object is marked according to the mapping relationship records of this deletion.

[0134] Figure 6 This is a flowchart of the data mapping method steps in one embodiment of the present invention for incremental writing of the base block or writing of the base block overwriting the base block. Figure 6 The steps and procedures will not cause a state machine switch (equivalent to the state machine switching from state 01 to state 01).

[0135] Step 601. Upon receiving the redirection operation request (ROW op), obtain the metadata at the user object level, perform redirection processing to generate the mapping relationship record rowkv and persist the row object;

[0136] Step 602. Determine whether the current situation is an incremental write of a basic block or an overwrite of an existing basic block based on the current mapping state of the state machine and the granularity of the write block. If the determination is yes, proceed to step 603.

[0137] Incremental write to the base block refers to a situation where new data is written to an 8k base block position that has not been written to before, without overwriting the previously written 8k base block.

[0138] Step 603. For cases where a base block covers another base block, generate the rowkey of the covered 8k base block based on the bitmap;

[0139] Step 604. The transaction will delete 8k_rowkv, write a new 8k_rowkv, and commit the write to the distributed rowdb for persistence.

[0140] Since writing to an 8k base block does not cause state machine switching or bitmap changes, transactional commits do not need to include these two operations.

[0141] Step 605. For incremental writes of the base block, the bitmap needs to be updated by setting the corresponding binary bit of the 8k base block to 1.

[0142] Step 606. The transaction will write to the new 8k_rowkv and update the bitmap, then commit the write to the distributed rowdb for persistence.

[0143] The following is a comparative description of the changes in metadata caused by 8k base block overwrite or incremental writes:

[0144] State conditions: Before writing, the state machine is in state 01 for independent writing of the basic block. There is a bitmap in rowdb, there is an 8K_rowkv with bitmap position 1, and there is no 1M_rowkv record.

[0145] State change: No state switching is involved;

[0146] Bitmap changes: Basic block overwrite does not involve bitmap changes, while basic block incremental write requires updating the bitmap;

[0147] Mapping relationship changes: It is necessary to generate 8k_rowkey based on the bitmap, delete the mapping relationship records of the overwritten base block, and add the newly written 8k_rowkv;

[0148] When 8k base blocks are overwritten, the process involves a garbage data marking step. The garbage handling mechanism identifies and marks garbage data based on the rowdb compaction process.

[0149] Figure 7 This is a flowchart of the data mapping method steps in the case of extension block overwriting extension block writing according to an embodiment of the present invention. Figure 7 The steps will not cause a state machine switch (equivalent to the state machine switching from state 10 to state 10).

[0150] Step 701. Upon receiving the redirection operation request (ROW op), obtain the metadata at the user object level, perform redirection processing to generate the mapping relationship record rowkv and persist the row object;

[0151] This step generates a corresponding 1M_rowkv based on the written 1M extended block.

[0152] Step 702. Determine whether the current state is an extension block overwrite extension block write (equivalent to the state machine switching from 10 to 10) based on the current mapping state of the state machine and the write block granularity. If the determination is yes, then proceed to step 703.

[0153] Step 703. Generate the mapping key field 1M_rowkey for the overwritten 1M extended block, and read the 1M_rowvalue of the overwritten extended block from the rowdb based on the generated 1M_rowkey;

[0154] Step 704. Based on the 1M_rowvalue of the overwritten extended block, mark the corresponding row object with 1M garbage amount and the garbage bitmap corresponding to the 1M range.

[0155] The read 1M_rowvalue includes the row object name and the position offset of the overridden extension block stored in the row object. Based on the read position offset, the corresponding 1M garbage amount and the corresponding garbage bitmap within the 1M range can be marked by the garbage collection module, so that the garbage collection mechanism can release the marked garbage.

[0156] During this process, when the GC merges compaction in the rowdb background, it does not mark the amount of garbage and the garbage bitmap of the 1M_rowkv that is being covered.

[0157] Step 705. The transactional operation of deleting the 1M_rowkv of the overwritten extension block and writing the newly written 1M_rowkv of the extension block is committed and persisted to the distributed rowdb.

[0158] The following is a comparative description of the changes in metadata caused by writing a 1M extended block over another 1M extended block:

[0159] State conditions: Before writing, the state machine is in the extended block independent write 10 state, which can be expressed in rowdb as 1M_written=true and 8K_written=false.

[0160] State changes: Since both the write operation and the write operation involve 1MB, the state machine does not need to change. The state machine remains in 10 states before and after the write operation.

[0161] Bitmap changes: Since both before and after are 1M pass-through writes, bitmap operations are not involved;

[0162] Mapping relationship change: The 1M_rowkv metadata of the overwritten extended block needs to be deleted from rowdb, and the 1M_rowkv metadata of the newly written extended block needs to be written.

[0163] Persistent transactions: In a transaction, delete the 1M_rowkv of the overwritten extension block and add the 1M_rowkv of the newly written extension block without updating the bitmap and state machine.

[0164] Figure 8 This is a flowchart of the data mapping method steps in the case of writing a base block in an extension block overlaying a base block, according to an embodiment of the present invention.

[0165] Step 801. Upon receiving the redirection operation request (ROW op), obtain the user object-level granular metadata, perform redirection processing to generate the mapping relationship record rowkv and persist the row object;

[0166] This step will obtain the 1M_rowkv of the extended block where the 8k position is written and the extended block bitmap (if it already exists), and generate the corresponding 8k_rowkv based on the base block to be written.

[0167] Step 802. Determine whether the current state is a basic block write to the extended block based on the current mapping state of the state machine and the granularity of the write block (equivalent to the state machine switching from 10 to 11). If the determination is yes, proceed to step 803.

[0168] Step 803. Set the 8k_written flag to true, which is equivalent to switching the state machine to mixed write state 11.

[0169] In the mixed write state, both 1M_rowkv and 8k_rowkv will exist in rowdb at the same time, so both 1M_written and 8k_written are true, which is equivalent to switching the state machine to 11.

[0170] Step 804. In the bitmap created or read, write 1 to the corresponding position of the 8k base block;

[0171] In this step, if a bitmap has not been created before, a bitmap for the extended block needs to be created; if a bitmap has already been created, it has been read in the previous steps. Then, the binary bit corresponding to the currently written 8k base block is set to 1 in the bitmap.

[0172] Step 805. Generate a 1M_rowkey to be written to the extended block where the 8k base block is located, and read rowkv from rowdb based on the generated 1M_rowkey to obtain the 1M_rowvalue of the extended block;

[0173] Step 806. Based on the obtained 1M_rowvalue, mark the amount of garbage at the covered 8k position and the binary position 1 at the covered 8k position in the garbage bitmap for the corresponding row object.

[0174] This step identifies and marks garbage data by directly marking 8k garbage data for row objects and setting garbage bitmaps during the write process, without relying on the garbage collection module to identify garbage data during the rowdb background compaction.

[0175] Step 807. The transactional operations of writing to the newly written base block 8k_rowkv, switching the mapping state to 11, and updating the bitmap are committed and written to the distributed rowdb for persistence.

[0176] The following is a comparative description of the changes in metadata caused by an 8k base block overwriting a base block in a 1M extension block:

[0177] State conditions: Before writing, in rowdb, 1M_written = true and 8k_written = false, which is equivalent to the state machine's mapping state being 10;

[0178] State change: Since the 8k base block currently being written is the base block in the 1M extended block written before the first time, the state machine will switch to mixed write state 11. 8k_written in rowdb needs to be set to true, while 1M_written remains unchanged.

[0179] Bitmap change: Since the current 8k base block write will overwrite the 8k base block in the previously passed-through 1M extended block, it will cause a bitmap change, and the corresponding binary bit of the overwritten 8k base block needs to be set to 1.

[0180] Mapping changes: Since the current 8k base block only covers one base block in the 1M extended block, and there are still many valid base blocks in the previously written 1M extended block that have not been covered, the rowdb will still retain the previously written 1M_rowkv, and at the same time, the 8k_rowkv of the currently written 8k base block needs to be added to the rowdb.

[0181] Persistent transactions: Since state changes, bitmap changes, and the addition of 8k_rowkv are involved, the above operations need to be committed to rowdb and persisted in a transaction.

[0182] Figure 9 This is a flowchart of the data mapping method steps for switching from a mixed write state to an independent write state of the extended block in one embodiment of the present invention.

[0183] Step 901. Upon receiving the redirection operation request (ROW op), obtain the user object-level granular metadata, perform redirection processing to generate the mapping relationship record rowkv and persist the row object;

[0184] By using the metadata of the user object, the bitmap of the extension block to be overwritten can be obtained, and a new 1M_rowkv to be written to the extension block can also be generated.

[0185] Step 902. Determine whether the current state is an extended block overwrite write in a mixed write state (equivalent to the state machine switching from 11 to 10) based on the current mapping state of the state machine and the write block granularity. If the determination is yes, proceed to step 903.

[0186] Step 903. Traverse the bitmap of the overwritten extension block, generate the 8k_rowkey corresponding to the 1-bit value, generate the 1M_rowkey of the overwritten extension block, and read the 1M_rowvalue of the overwritten extension block from the rowdb.

[0187] In mixed write state, rowdb will simultaneously store 1M_rowkv of the extended block to be overwritten and 8k_rowkv of the base block of the base block in the extended block to be overwritten. Therefore, it is necessary to traverse the extended block bitmap to generate 8k_rowkey corresponding to the binary bit with a value of 1, and obtain 1M_rowkey of the extended block to be overwritten.

[0188] Step 904. Based on the 1M_rowvalue of the overwritten extended block, set all corresponding bits in the 1M range of the garbage bitmap of the row object to 1, then set all bits in the extended block bitmap to 0, and mark 8k_written as false.

[0189] Since the newly written extension block is equivalent to overwriting both the original extension block in the user object and the base block corresponding to bit 1 in the original extension block bitmap, it is necessary to obtain the row object name from the 1M_rowvalue of the overwritten extension block and set all the corresponding bits in the 1M range of the garbage bitmap of the corresponding row object to 1 to mark this part of the data in the row object as garbage data. At the same time, it is also necessary to delete the 8k_rowkv of the base block corresponding to bit 1 in the extension block bitmap from the rowdb, so it is also necessary to clear the extension block bitmap to zero.

[0190] Keep 1M_written as true and mark 8k_written as false, which is equivalent to switching the state machine to the extended block independent write state 10.

[0191] Step 905. Commit the operations of deleting the overwritten extension block 1M_rowkv, writing the new extension block 1M_rowkv, deleting the overwritten base block 8k_rowkv, mapping state 10, and updating the bitmap to the distributed rowdb in a single transaction.

[0192] In this step, the overwritten extended blocks are identified and marked with garbage amounts and garbage bitmaps directly on the row objects during the write process, without relying on the garbage collection module to identify garbage data during rowdb background compaction. However, the overwritten base blocks require the garbage collection mechanism to identify garbage data during rowdb background compaction.

[0193] The following is a comparative description of the changes in metadata caused by writing to extended blocks under mixed write conditions:

[0194] Status conditions: Before writing, if 1M_written = true and 8K_written = true in rowdb, then it is in a mixed write state.

[0195] State change: Since it is a pass-through overwrite, all overwritten 8k_rowkv needs to be cleared. Therefore, 8K_written needs to be set to false, while 1M_written = true remains unchanged. This is equivalent to switching the state machine to the extended block independent write state 10.

[0196] Bitmap changes: Since the current write is a 1M pass-through full overwrite, all data in the row object corresponding to the overwritten extended block needs to be marked as garbage. In addition, the base block corresponding to the value 1 in the extended block bitmap also needs to be identified as garbage. Therefore, in addition to clearing the extended block bitmap to zero, all garbage bitmaps corresponding to the extended block in the row object corresponding to the overwritten extended block need to be set to 1. The garbage bitmap is used by the garbage collection mechanism, and setting it to 1 represents garbage data.

[0197] Mapping changes: Since the newly written extension block overwrites the original extension block, it is necessary to delete 1M_rowkv of the original extension block, write 1M_rowkv of the newly mapped extension block, and also delete 8k_rowkv corresponding to the value 1 bit in the 1M bitmap.

[0198] Figure 10 This is a flowchart of the data mapping method steps for writing to a base block in a mixed write state according to an embodiment of the present invention.

[0199] Step 1001. Upon receiving the redirection operation request (ROW op), obtain the metadata at the user object level, perform redirection processing to generate the mapping relationship record rowkv and persist the row object;

[0200] By using the metadata of the user object, the bitmap of the extended block to be overwritten can be obtained, and the 8k_rowkv of the newly written base block can also be generated.

[0201] Step 1002. Determine whether the current state is a basic block 8k write under a mixed write state based on the current mapping state of the state machine and the write block granularity. If the determination is yes, proceed to step 1003.

[0202] Step 1003. Determine if the bit corresponding to the currently written base block in the extended block bitmap is 0. If yes, proceed to step 1004; if no, proceed to step 1009.

[0203] This step determines whether it's an 8k in-situ overwrite or an 8k incremental overwrite by checking if the corresponding bit in the extended block bitmap is 0. An 8k in-situ overwrite means that an 8k overwrite has occurred before the previously written position, while an 8k incremental overwrite means that an 8k overwrite has never occurred before the corresponding 8k base block position in the extended block.

[0204] Step 1004. Write the current 8k base block to the corresponding position 1 in the extended block bitmap;

[0205] Step 1005. Determine if the extended block bitmap is all 1s. If yes, proceed to step 1006; otherwise, proceed to step 1007.

[0206] An extended block bitmap with all bits set to 1 indicates that when an 8k base block is written, all 8k base blocks of the original extended block are overwritten (referred to as full overwrite).

[0207] Step 1006. For the full coverage case, set 1M_written to false, which is equivalent to switching the mapping state to the basic block independent write state 01;

[0208] Step 1007. Generate the 1M_rowkey of the overwritten extended block, use the 1M_rowkey to read the rowkv from the rowdb to obtain the corresponding 1M_rowvalue, obtain the row object identifier based on the 1M_rowvalue, and mark the row object with the corresponding 8k garbage amount and the garbage bitmap corresponding to the 8k position of the basic block.

[0209] Step 1008. In the case of full coverage, the operations of deleting the 1M_rowkv of the fully covered extended block, writing the new base block 8k_rowkv, updating the extended block bitmap, and updating the mapping status of 01 are put into a transaction commit rowdb persistence; in the case of non-full coverage, the operations of writing the new base block 8k_rowkv and updating the extended block bitmap are put into a transaction commit rowdb persistence.

[0210] Step 1008. In the case of 8k in-situ overwrite, the operation of writing to the new base block 8k_rowkv and deleting the overwritten base block 8k_rowkv is placed into a transaction commit rowdb persistence.

[0211] The following is a comparative description of the changes in metadata caused by writing to the base block under mixed write conditions:

[0212] State conditions: Before writing, 1M_written = true and 8K_written = true in rowdb, which is equivalent to the state machine being in mixed write state 11.

[0213] Changes in the extended block bitmap: There are two cases:

[0214] Case 1: 8k in-situ overwrite, the newly written 8k base block overwrites the 8k base block corresponding to the value 1 bit in the 1M bitmap, and the bitmap remains unchanged;

[0215] Scenario 2: 8k incremental overwrite, which is an 8k overwrite of position 0 in the 1M bitmap, requiring the bitmap to be updated.

[0216] Case 2 is further divided into two cases: Case 2-1 is a full coverage of 1M, and Case 2-2 is a partial coverage of 1M.

[0217] State changes: Case 1 and Case 2-2 will not cause a state switch. Case 2-1 full coverage requires switching the mixed write state 11 to the basic block independent write state 01.

[0218] Mapping relationship changes: Case 1 involves overwriting the same 8k position again, so it is necessary to delete the original 8k_rowkv of the overwritten base block and add a new 8k_rowkv.

[0219] In case 2, because there is a new 8k overwrite, it is necessary to add a new 8k_rowkv and mark the data corresponding to the overwritten base block in the row object as garbage; for case 2-1, it is also necessary to delete the 1M_rowkv that is completely overwritten.

[0220] In summary, when large I / O writes to the 1M extended block overwrite data, garbage is directly marked during the write process, without relying on the rowdb background compaction operation to identify and mark garbage. Garbage is only identified and marked by the rowdb background compaction operation when data written to the 8K base block is overwritten. This is because the metadata representation ranges corresponding to 1M and 8K are different, and the original garbage collection mechanism cannot identify the mapping metadata of different granularities. Of course, the garbage collection mechanism can be improved to identify metadata of different granularities.

[0221] Based on the write-time redirection data mapping method provided by this invention, in pure large I / O write scenarios, it can greatly simplify metadata processing, change the practice of dividing 1M large I / O into 128 small I / Os with 8K granularity for aggregation mapping, and effectively reduce latency and CPU overhead in large I / O scenarios.

[0222] Figure 11 This is a schematic diagram of an electronic device structure for implementing the write-time redirection data mapping method provided in this invention, according to an embodiment of the present invention. The device 1100 includes a processor 1110 (such as a central processing unit (CPU), a communication bus 1120, a communication interface 1140, and a memory 1130. The processor 1110 and the memory 1130 can communicate with each other via the communication bus 1120. The memory 1130 stores a computer program, which, when executed by the processor 1110, implements the write-time redirection data mapping method provided in this embodiment of the invention.

[0223] Memory refers to a device based on a storage medium used to store computer programs and / or data. It can be volatile memory (VM, often called main memory) or non-volatile memory (NVM). Main memory refers to internal storage that directly exchanges data with the processor. It can read and write data at any time and at high speed, serving as a storage medium for temporary data of the operating system and other running programs. Main memory can be Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), etc. Non-volatile memory refers to memory that uses persistent storage media, characterized by large capacity and persistent data retention. It can be Storage Class Memory (SCM), Solid State Drive (SSD), NAND flash memory, hard disks, etc. SCM is an industry term for a new type of storage medium that falls between main memory and flash memory. It is a composite storage technology that combines the characteristics of persistent storage and main memory, with access speeds slower than DRAM but faster than SSDs.

[0224] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0225] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in non-transitory (or non-persistent) memory. The method can be implemented using standard programming techniques, including a non-transitory storage medium configured with a computer program within the computer program, wherein such a storage medium causes the computer to operate in a specific and predefined manner. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit. Moreover, the operations of the processes described in this invention can be performed in any suitable order unless otherwise indicated by the invention or otherwise clearly contradicted by the context. The processes (or variations and / or combinations thereof) described in this invention can be executed under the control of one or more computer systems configured with executable instructions and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.

[0226] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention also includes the computer itself.

[0227] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A write-time redirection data mapping method, characterized in that, The method includes: When a redirection operation request for writing data to a user object is received, the mapping state of the mapping relationship between the data block within the extended block range of the user object and the write-time redirection object, i.e., the row object, is determined based on the state machine. When writing data blocks as extended blocks, the mapping state is switched to extended block independent mapping state. During writing, the metadata recorded in the database rowdb is redirected, including the mapping relationship between extended blocks and row objects, the mapping state, and the corresponding garbage marking processing is performed. When writing a data block as a base block and currently in an independent mapping state for extended blocks, the position of the written base block in the extended block is identified by the extended block bitmap. The mapping state is then switched to a mixed mapping state. The metadata recorded in rowdb includes: the mapping relationship between the written base block and the row object, the extended block bitmap, the mapping state, and the corresponding garbage marking process.

2. The method according to claim 1, characterized in that, The method further includes: When writing a data block as a base block and currently in a hybrid mapping state, the position of the base block being written within the extended block is identified by the extended block bitmap; When the base block covers all extended blocks, switch the mapping state to base block independent mapping state, delete the mapping relationship between the extended block where the base block is located and the row object in rowdb, record the mapping relationship between the base block and the row object, update the extended block bitmap and perform the corresponding garbage marking process; If the base block does not cover all the extended blocks, the mapping state remains unchanged. The mapping relationship between the base block and the row object is recorded in the rowdb, the extended block bitmap is updated, and the corresponding garbage marking process is performed.

3. The method according to claim 2, characterized in that, The mapping relationship and mapping status of an extended block for a user object recorded in rowdb are related. In the independent mapping state of the base block, the mapping relationship recorded in rowdb only includes the mapping relationship between the base block and the row object; In the state of independent mapping of extended blocks, the mapping relationship recorded in rowdb only includes the mapping relationship between extended blocks and row objects; In a hybrid mapping state, the mapping relationships recorded in rowdb include both the mapping relationship between the base block and the row object and the mapping relationship between the extended block and the row object.

4. The method according to claim 2, characterized in that, The execution of the corresponding garbage marking process includes: When the extension block is overridden, the garbage amount and garbage bitmap are directly marked for the row object in the redirection operation request processing flow; When the base block is overwritten, the garbage disposal mechanism identifies and marks garbage data through the rowdb merging process.

5. The method according to claim 1, characterized in that, In the redirection operation request processing flow, metadata update operations, including writing mapping relationship records (rowkv), deleting mapping relationship records, updating extended block bitmaps, and updating one or more of the mapping status, are put into a transaction and committed to rowdb for persistence.

6. A write-time redirection data mapping device, characterized in that, The device includes: The judgment module is used to determine the mapping status of the mapping relationship between the data blocks within the extended block range of the user object and the write-time redirection object, i.e., the row object, based on the state machine when a redirection operation request for writing data to the user object is received. The state machine module is used to maintain the state machine and perform the switching of mapped states; The bitmap maintenance module is used to maintain the extended block bitmap; The garbage marking module is used to perform corresponding garbage marking processing in the redirection operation request according to the state machine switching method; The metadata maintenance module is used to maintain metadata related to write-time redirection mapping relationships in the write-time redirection database rowdb. When writing data blocks as extended blocks, the state machine module is used to switch the mapping state to an extended block independent mapping state; the metadata recorded in rowdb by the metadata maintenance module includes: the mapping relationship between extended blocks and row objects, and the mapping state; When writing a data block as a base block and currently in an independent mapping state for extended blocks, the bitmap maintenance module uses an extended block bitmap to identify the position of the written base block in the extended block; the state machine module is used to switch the mapping state to a mixed mapping state; the metadata recorded in the rowdb by the metadata maintenance module includes: the mapping relationship between the written base block and the row object, the extended block bitmap, and the mapping state.

7. The apparatus according to claim 6, characterized in that, When writing a data block as a base block and currently in a hybrid mapping state, the bitmap maintenance module is used to identify the position of the written base block in the extended block using an extended block bitmap; When the base block covers all extended blocks, the state machine module is used to switch the mapping state to the base block independent mapping state; the metadata maintenance module is used to delete the mapping relationship between the extended block where the base block is located and the row object in the rowdb, record the mapping relationship between the base block and the row object, and update the extended block bitmap. The mapping state remains unchanged if the base block does not cover all the extension blocks; The metadata maintenance module is used to record the mapping relationship between base blocks and row objects in rowdb and to update the extended block bitmap.

8. The apparatus according to claim 7, characterized in that, The mapping relationship and mapping status of an extended block for a user object recorded in rowdb are related. In the independent mapping state of the base block, the mapping relationship recorded in rowdb by the metadata maintenance module only includes the mapping relationship between the base block and the row object; In the state of independent mapping of extended blocks, the mapping relationship recorded in rowdb by the metadata maintenance module only includes the mapping relationship between extended blocks and row objects; In the hybrid mapping state, the mapping relationships recorded in rowdb by the metadata maintenance module include both the mapping relationship between the base block and the row object and the mapping relationship between the extended block and the row object.

9. The apparatus according to claim 7, characterized in that, When the extended block is overridden, the garbage marking module directly marks the garbage amount and garbage bitmap for the row object in the redirection operation request processing flow; When the base block is overwritten, the garbage disposal mechanism identifies and marks garbage data through the rowdb merging process.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a storage medium, and a communication bus, wherein the processor, the communication interface, and the storage medium communicate with each other through the communication bus; Storage medium used to store computer programs; A processor, when executing a computer program stored on a storage medium, implements the method of any one of claims 1-5.

11. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1 to 5.

Images