Data processing method, device and equipment in block storage system and storage medium

Abstract

The application provides a data processing method and device in a block storage system, equipment and a storage medium, comprising: receiving a data write request, the data write request carrying a target logical address and to-be-written data; determining a target hybrid data block from a plurality of hybrid data blocks according to the target logical address, wherein the logical address space of each hybrid data block contains a first storage area adopting a multi-copy redundancy mode and a second storage area adopting an erasure code redundancy mode; determining a target storage area between the first storage area and the second storage area of the target hybrid data block according to the data amount specification of the to-be-written data; writing the to-be-written data into the target storage area and obtaining the actual physical storage position of the to-be-written data in the target hybrid data block; and updating index information according to the target logical address and the actual physical storage position to record the mapping relationship between the target logical address and the actual physical storage position.

Description

Technical Field

[0001] This application relates to the field of computer storage technology, and in particular to a data processing method, apparatus, device and storage medium in a block storage system. Background Technology

[0002] With the development of cloud computing technology, block storage services have become a core component of cloud infrastructure, providing persistent storage capabilities for applications such as virtual machines (VMs) and databases. Modern distributed block storage systems typically employ segmented address space management and ensure data reliability through multi-replication mechanisms, but this also leads to higher storage costs.

[0003] Current mainstream solutions employ a multi-replica log structure storage approach. The system divides the user's logical address space into multiple fixed-size segments. Data is persisted to the storage medium in an append-only manner, and storage space is reclaimed through a background garbage collection (GC) mechanism. To reduce storage overhead, some systems use erasure coding (EC) technology, but this is typically limited to cold data storage scenarios and requires offline conversion of multi-replica data into erasure-coded data.

[0004] However, existing technologies cannot intelligently select data redundancy strategies based on write characteristics during data writing. The system either always uses a costly multi-copy approach to store all data, or requires an inefficient offline data conversion process to optimize storage costs. This limits the ability of block storage systems to achieve an optimal balance between performance and cost, making it difficult to simultaneously achieve the best performance in both areas. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and storage medium in a block storage system to solve the problem in the prior art that it is impossible to intelligently select a data redundancy strategy based on the write characteristics during data writing.

[0006] In a first aspect, this application provides a data processing method in a block storage system, including:

[0007] Receive a data write request, the data write request carrying a target logical address and the data to be written;

[0008] Based on the target logical address, a target mixed data block is determined from multiple mixed data blocks, wherein the logical address space of each mixed data block includes a first storage area using a multi-replica redundancy method and a second storage area using an erasure coding redundancy method;

[0009] Based on the data volume specifications of the data to be written, a target storage area is determined between the first storage area and the second storage area of ​​the target mixed data block;

[0010] Write the data to be written to the target storage area and obtain the actual physical storage location of the data to be written within the target mixed data block;

[0011] Based on the target logical address and the actual physical storage location, update the index information to record the mapping relationship between the target logical address and the actual physical storage location.

[0012] In one possible implementation, determining the target storage area between the first and second storage areas of the target mixed data block according to the data volume specifications of the data to be written includes:

[0013] Compare the data volume specification with a first preset threshold;

[0014] If the data volume is less than the first preset threshold, the first storage area is determined as the target storage area.

[0015] If the data volume specification is greater than or equal to the first preset threshold, the second storage area is determined as the target storage area.

[0016] In one possible implementation, updating the index information based on the target logical address and the actual physical storage location includes:

[0017] The mapping relationship between the target logical address and the actual physical storage location in the index information is updated to the memory index structure as a target index entry.

[0018] When the number of index entries stored in the memory index structure reaches a second preset threshold, the index entries stored in the memory index structure are asynchronously persisted to the metadata persistence area in the corresponding mixed data block. The metadata persistence area stores data in a multi-replica redundancy manner.

[0019] In one possible implementation, the method further includes:

[0020] After the data to be written is written to the target storage area, verification information is generated for the written data;

[0021] Assign a corresponding sequence number to the data that has been written, wherein the sequence number assigned to the data written later is greater than the sequence number assigned to the data written earlier;

[0022] The serial number is embedded in the verification information to verify data integrity and write timing.

[0023] In one possible implementation, the method further includes:

[0024] During the data recovery process, the storage status of the data corresponding to the logical address to be recovered in the target mixed data block is determined based on the index information.

[0025] If it is determined that the data corresponding to the logical address to be recovered exists in both the first storage area and the second storage area, the first data and the corresponding first sequence number in the first storage area, and the second data and the corresponding second sequence number in the second storage area are read respectively.

[0026] Compare the first serial number with the second serial number;

[0027] Data with larger serial numbers are identified as valid data and recovered.

[0028] In one possible implementation, the method further includes:

[0029] Monitor the status of the first storage area and the second storage area in the target mixed data block;

[0030] If the state meets the preset sealing conditions, a new hybrid data block is created, wherein the preset sealing conditions include: the first storage area is full, the second storage area is full, the first storage area fails to write, or the second storage area fails to write.

[0031] Subsequent data write requests are allocated to the new mixed data block for processing.

[0032] In one possible implementation, the method further includes:

[0033] Receive a data read request, wherein the data read request carries the logical address of the data to be read;

[0034] The corresponding actual physical storage location is obtained by querying the index information based on the logical address of the data to be read.

[0035] If the actual physical storage location is located within the logical address range allocated to the first storage area, data is read from the first storage area;

[0036] If the actual physical storage location is within the logical address range allocated to the second storage area, data is read from the second storage area.

[0037] Secondly, this application provides a data processing apparatus in a block storage system, comprising:

[0038] A receiving module is used to receive a data write request, wherein the data write request carries a target logical address and data to be written;

[0039] The first determining module is used to determine a target mixed data block from multiple mixed data blocks according to the target logical address, wherein the logical address space of each mixed data block includes a first storage area using a multi-replica redundancy method and a second storage area using an erasure coding redundancy method.

[0040] The second determining module is used to determine a target storage area between the first storage area and the second storage area of ​​the target mixed data block according to the data volume specification of the data to be written;

[0041] The writing module is used to write the data to be written to the target storage area and obtain the actual physical storage location of the data to be written within the target mixed data block;

[0042] The update module is used to update the index information according to the target logical address and the actual physical storage location, so as to record the mapping relationship between the target logical address and the actual physical storage location.

[0043] In one possible implementation, the second determining module is specifically used for:

[0044] Compare the data volume specification with a first preset threshold;

[0045] If the data volume is less than the first preset threshold, the first storage area is determined as the target storage area.

[0046] If the data volume specification is greater than or equal to the first preset threshold, the second storage area is determined as the target storage area.

[0047] In one possible implementation, the update module is specifically used for:

[0048] The mapping relationship between the target logical address and the actual physical storage location in the index information is updated to the memory index structure as a target index entry.

[0049] When the number of index entries stored in the memory index structure reaches a second preset threshold, the index entries stored in the memory index structure are asynchronously persisted to the metadata persistence area in the corresponding mixed data block. The metadata persistence area stores data in a multi-replica redundancy manner.

[0050] In one possible implementation, the apparatus further includes a generation module for:

[0051] After the data to be written is written to the target storage area, verification information is generated for the written data;

[0052] Assign a corresponding sequence number to the data that has been written, wherein the sequence number assigned to the data written later is greater than the sequence number assigned to the data written earlier;

[0053] The serial number is embedded in the verification information to verify data integrity and write timing.

[0054] In one possible implementation, the device further includes a recovery module for:

[0055] During the data recovery process, the storage status of the data corresponding to the logical address to be recovered in the target mixed data block is determined based on the index information.

[0056] If it is determined that the data corresponding to the logical address to be recovered exists in both the first storage area and the second storage area, the first data and the corresponding first sequence number in the first storage area, and the second data and the corresponding second sequence number in the second storage area are read respectively.

[0057] Compare the first serial number with the second serial number;

[0058] Data with larger serial numbers are identified as valid data and recovered.

[0059] In one possible implementation, the apparatus further includes a creation module for:

[0060] Monitor the status of the first storage area and the second storage area in the target mixed data block;

[0061] If the state meets the preset sealing conditions, a new hybrid data block is created, wherein the preset sealing conditions include: the first storage area is full, the second storage area is full, the first storage area fails to write, or the second storage area fails to write.

[0062] Subsequent data write requests are allocated to the new mixed data block for processing.

[0063] In one possible implementation, the device further includes a reading module for:

[0064] Receive a data read request, wherein the data read request carries the logical address of the data to be read;

[0065] The corresponding actual physical storage location is obtained by querying the index information based on the logical address of the data to be read.

[0066] If the actual physical storage location is located within the logical address range allocated to the first storage area, data is read from the first storage area;

[0067] If the actual physical storage location is within the logical address range allocated to the second storage area, data is read from the second storage area.

[0068] Thirdly, this application provides an apparatus comprising: a processor and a memory, the processor being configured to execute a data processing program in a block storage system stored in the memory to implement the data processing method in the block storage system described in any one of the first aspects.

[0069] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the data processing method in the block storage system described in the first aspect.

[0070] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application first introduces a novel storage structure—a hybrid data block with multiple replicas in the first storage area and erasure coding in the second storage area—providing an infrastructure for differentiated data storage. Then, based on the data volume specifications of the data to be written, a target storage area is dynamically determined between the first and second storage areas. Next, the data to be written is written to the selected target storage area, and its actual physical storage location is obtained. Finally, data addressability is ensured by establishing and maintaining an index mapping relationship between the target logical address and the actual physical storage location. Thus, during the data writing stage, an adaptive storage strategy selection based on data characteristics is achieved, overcoming the limitations of the existing technology's single redundancy approach, thereby effectively optimizing storage costs while ensuring system performance. Attached Figure Description

[0071] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0072] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0074] Figure 1 This refers to the composition and pointing relationship of sequential data blocks in the metadata of the post-garbage collection stage, after a period of reading and writing during the traditional garbage collection scheduling process.

[0075] Figure 2 This is a schematic diagram of the structure of a novel hybrid data block provided in an embodiment of this application;

[0076] Figure 3 A flowchart illustrating an embodiment of a data processing method in a block storage system provided in this application;

[0077] Figure 4 This document defines the composition and pointer relationships of sequential data blocks in the metadata of the garbage collection scheduling process after a period of reading and writing and garbage collection.

[0078] Figure 5 A flowchart illustrating an embodiment of a data processing method in a block storage system provided in this application;

[0079] Figure 6 A flowchart illustrating an embodiment of a data processing method in a block storage system provided in this application;

[0080] Figure 7 A block diagram illustrating an embodiment of a data processing apparatus in a block storage system provided in this application.

[0081] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0083] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0084] To enable those skilled in the art to better understand this application, the basic architecture and data management mechanism of the underlying distributed block storage system involved in this application will first be explained.

[0085] In distributed storage systems underlying block storage, address segmentation is typically employed to improve input / output (I / O) parallelism. Specifically, the system linearly divides the logical address space of the user's cloud disk into multiple contiguous segment groups (typically hundreds of GB in size), and each segment group is further divided into multiple segments (typically representing tens of GB of address space). The system performs I / O scheduling and processing at the segment level, allowing different segments to be allocated to different servers for parallel processing, thereby improving system read and write performance.

[0086] Within the address space managed by each segment, the system divides I / O requests into fixed-size data pages (typically 4KB) for processing. Data is written to disk using a block-based redirection mechanism: when repeatedly writing to the same logical address, the system does not overwrite the original physical location, but instead appends the new data sequentially to the end of the currently active data block. The existing data is marked as invalid, and its storage space is reclaimed by the background garbage collection (GC) process. This type of data block that supports append write operations is called a SequenceBlock (SDB). The size of the SDB is configurable, for example, set to 2GB; when it is full, the system automatically switches to a new SDB. Each SDB contains metadata that records the physical offset information of each page within it. As data is continuously written, the composition and pointing relationship of the SDBs in the segment metadata (Segment Meta) are as follows... Figure 1 As shown, a segment may contain multiple SDBs.

[0087] During data retrieval, the system iterates through the target page from the most recently generated SDB. If the page is found in the current SDB, the system returns the data; otherwise, it continues searching through earlier SDBs. If none of the SDBs contain the page, the system returns all zeros.

[0088] To further reduce storage overhead and network transmission costs, the system also supports Erasure Coding (EC) Blocks. An EC Block consists of several data columns and a parity column, with each column essentially representing an SDB. Taking a "4+2" configuration as an example, this EC Block carries 2GB of user data, with each column having a capacity of 512MB, resulting in a total storage footprint of 3GB. The storage overhead is equivalent to 1.5 replicas, a 50% reduction compared to a three-replica SDB. If the system can directly write data to the EC Block during the I / O path, it can not only reduce storage footprint but also halve network write overhead.

[0089] However, EC encoding requires a complete stripe of data to calculate the checksum and write it to disk. For example, in a 4+2 stripe, each column needs to write at least one page (4KB), meaning the system needs to accumulate at least 16KB of data to form a valid stripe; otherwise, zero-padding will occur, leading to write amplification. Existing storage architectures typically apply EC technology to cold data compression scenarios, using offline conversion to convert three-replica data into EC data. This post-processing method cannot directly realize the advantages of EC writing on the I / O write path, making it difficult to achieve an optimal balance between storage cost and performance.

[0090] To address the aforementioned technical problems, this application proposes a data processing method based on a novel Hybrid Data Block (HDB) structure. For example... Figure 2 As shown, the HDB consists of three parts: Multi-replica Sequential Data Blocks (Triple-replica SDB): Sequential blocks employing a triple-replica redundancy mechanism, used to store user data; Erasure Coding Blocks (EC Blocks): Storage blocks employing an erasure coding redundancy mechanism, used to store user data; and Index Sequence Blocks: Sequential blocks employing a triple-replica redundancy mechanism, dedicated to storing data index information within the HDB. The multi-replica sequential data blocks and erasure coding blocks together constitute a hybrid storage area for user data, handling data write requests of different specifications. The index sequence blocks independently store the mapping relationship between the logical addresses and physical locations of all data pages within the HDB, ensuring data addressability and consistency.

[0091] Figure 3 This is a flowchart illustrating an embodiment of a data processing method in a block storage system provided by this application. The method is based on the aforementioned HDB structure and effectively solves the application limitations of EC technology in real-time write scenarios by intelligently selecting data storage strategies in the I / O write path. Figure 3 As shown, the method mainly includes the following steps:

[0092] Step 301: Receive a data write request, which carries a target logical address and the data to be written.

[0093] Data write request: A storage operation instruction initiated by the client, which follows the block storage protocol specification and includes fields such as operation type, target address, and data payload.

[0094] Target Logical Address: Refers to the Logical Block Address (LBA) from the user's perspective, representing the offset position of data in the virtual disk.

[0095] Data to be written: refers to the actual data content that needs to be persisted, usually in units of fixed-size data pages (e.g., 4KB).

[0096] In this embodiment, the storage system receives a data write request sent by the client through a network interface. After parsing the opcode field in the request header to confirm that it is a write operation, the system extracts the target logical address field and the data payload field to obtain the target logical address and the data to be written, respectively. The system can temporarily store this information in a memory buffer for subsequent processing.

[0097] Step 302: Based on the target logical address, determine the target mixed data block from multiple mixed data blocks, wherein the logical address space of each mixed data block includes a first storage area using a multi-replica redundancy method and a second storage area using an erasure coding redundancy method.

[0098] Hybrid Data Block (HDB): refers to a novel data storage unit provided in this application, whose logical address space is divided into two parts: a first storage area (multiple copy redundancy) and a second storage area (erasure coding redundancy).

[0099] Multiple-replica redundancy: Data reliability is ensured by creating multiple replicas (usually three replicas) of the same data.

[0100] Erasure coding redundancy: Data is encoded to generate a check block, which ensures data reliability with lower storage overhead. For example, the storage overhead of 4+2 erasure coding is 1.5 copies.

[0101] In this embodiment, the system maintains a global address mapping table to record the correspondence between each logical address range and a hybrid data block. When a data write request is received, the system queries the mapping table to determine the address range to which the target logical address belongs, and then finds the currently active HDB managing that address range. Each HDB is responsible for processing a contiguous logical address space, and its internal components include a multi-replica storage area and an erasure coding storage area, providing the infrastructure for subsequent intelligent data routing.

[0102] Step 303: Determine a target storage area between the first storage area and the second storage area of ​​the target mixed data block according to the data volume specifications of the data to be written.

[0103] Data volume specification: refers to the size of the data to be written this time, such as 4KB (small granularity IO) or 128KB (large granularity IO).

[0104] In this embodiment of the application, step 303 may include the following steps: comparing the data volume specification with a first preset threshold; if the data volume specification is less than the first preset threshold, determining the first storage area as the target storage area; if the data volume specification is greater than or equal to the first preset threshold, determining the second storage area as the target storage area.

[0105] First preset threshold: This is a configurable parameter preset by the system to distinguish the data size of I / O operations. Its typical value is 16KB or 32KB. The system administrator can dynamically adjust this threshold according to actual business needs.

[0106] In this scheme, the optimal data storage strategy is dynamically selected by judging the relationship between the data size of the data to be written and the preset threshold in real time: when the data size is less than the first preset threshold (i.e., small data), the data is routed to the first storage area, and the multi-replica mechanism is used to provide low-latency and high-reliability data writing services; when the data size is greater than or equal to the first preset threshold (i.e., large data), the data is routed to the second storage area, and the erasure coding mechanism is used to significantly reduce storage space overhead and network transmission costs.

[0107] This solution enables the system to adaptively select the optimal storage strategy based on data characteristics while ensuring data reliability. It avoids the write amplification problem caused by zero-padding operations, and fully leverages the advantages of multiple replicas and erasure coding, effectively improving the overall efficiency of the storage system and achieving the best balance between performance and cost.

[0108] Step 304: Write the data to be written to the target storage area and obtain the actual physical storage location of the data to be written within the target mixed data block.

[0109] Actual physical storage location: The specific location of the data on the persistent medium, which usually includes the block identifier (such as SDB ID) and the offset within the block.

[0110] In this embodiment, if the target storage area is the first storage area, the data is synchronously copied to the memory buffers of three different nodes via RDMA (Remote Direct Memory Access) protocol, and then persisted to the SSD (Solid State Drive) device, recording the triple (node ​​ID, device ID, block offset) as the actual physical storage location. If the target storage area is the second storage area, the data is segmented into data fragments and a checksum is calculated. These fragments are sent to multiple storage nodes via parallel network channels. Each node persists the data independently and returns a success response. The system records the stripe number and fragment index as the actual physical storage location. The physical location information is encapsulated as a 64-bit or 128-bit location descriptor.

[0111] After introducing HDB support for direct writing of EC blocks via I / O write path, and after a period of data writing, the composition structure and pointer relationships of data blocks in the Segment Meta are as follows: Figure 4 As shown. Figure 4 As shown, Segment Meta can contain metadata information from multiple HDBs, reflecting the writing results at different times and with different data characteristics.

[0112] Step 305: Update the index information according to the target logical address and the actual physical storage location to record the mapping relationship between the target logical address and the actual physical storage location.

[0113] Index information: A metadata table used to maintain the mapping relationship between logical addresses and physical locations.

[0114] In this embodiment of the application, step 305 may specifically include the following steps: updating the mapping relationship between the target logical address and the actual physical storage location in the index information as a target index entry to the memory index structure; when the number of index entries stored in the memory index structure reaches a second preset threshold, asynchronously persisting the index entries stored in the memory index structure to the metadata persistence area in the corresponding hybrid data block, wherein the metadata persistence area stores data in a multi-replica redundancy manner.

[0115] Target index entry: refers to the single mapping record that needs to be added or updated in this write operation; In-memory index structure: refers to the data structure (such as a hash table or B+ tree) residing in memory, used to efficiently manage currently active mapping relationships; Second preset threshold: a system-configurable parameter (such as 1000 index entries) used to control the trigger frequency of index persistence; Asynchronous persistence: refers to the process of writing data to persistent storage media in the background without blocking; Metadata persistence area: refers to the Sequence Block inside the HDB specifically used to store index information.

[0116] This scheme first caches and manages the latest index changes (i.e., target index entries) in high-speed memory. When the accumulated number of changes (i.e., the number of index entries) reaches a second preset threshold, they are then written in batches to the persistent storage area, effectively reducing the performance overhead caused by frequent small-scale I / O operations. The metadata persistent area employs a multi-replica mechanism to ensure high reliability of the index data, while the asynchronous persistence mechanism avoids blocking foreground data write requests, ensuring high throughput and low latency characteristics of the system. Through this scheme, the system achieves efficient and reliable management of index data, ensuring real-time updates and fast queries of mapping relationships, while significantly reducing the overhead of metadata management through the batch asynchronous persistence mechanism, providing a stable and reliable metadata service foundation for the entire storage system.

[0117] The technical solution provided in this application first introduces a novel storage structure—a hybrid data block with multiple replicas in the first storage area and erasure coding in the second storage area—to provide an infrastructure for differentiated data storage. Then, based on the data volume specifications of the data to be written, a target storage area is dynamically determined between the first and second storage areas. Next, the data to be written is written to the selected target storage area, and its actual physical storage location is obtained. Finally, data addressability is ensured by establishing and maintaining an index mapping relationship between the target logical address and the actual physical storage location. Thus, during the data writing stage, an adaptive storage strategy selection based on data characteristics is achieved, overcoming the limitations of existing technologies that use a single redundancy method, thereby effectively optimizing storage costs while ensuring system performance.

[0118] Figure 5 A flowchart illustrating an embodiment of a data processing method in a block storage system provided in this application. Figure 5 The process shown is in Figure 1 Based on the illustrated process, the following steps are included:

[0119] Step 501: During the data recovery process, based on the index information, determine the storage status of the data corresponding to the logical address to be recovered in the target mixed data block;

[0120] Step 502: If it is determined that the data corresponding to the logical address to be recovered exists in both the first storage area and the second storage area, read the first data and the corresponding first serial number in the first storage area, and the second data and the corresponding second serial number in the second storage area respectively.

[0121] Step 503: Compare the size of the first serial number and the second serial number;

[0122] Step 504: Identify the data with larger serial numbers as valid data and recover them.

[0123] For ease of understanding, steps 501-504 will be explained uniformly below:

[0124] Data recovery process: The operation of restoring data consistency after a system malfunction (such as a power outage).

[0125] Logical address to be recovered: refers to the logical location of the user data that needs to be recovered.

[0126] Storage status: refers to the multiple physical versions of data that may exist in the HDB.

[0127] During the data writing process, the system assigns a monotonically increasing global sequence number to each piece of data written to the HDB, bound to the current HDB. This sequence number serves as a logical timestamp for the data, uniquely identifying and ordering the sequence of write operations. Based on this mechanism, the first sequence number refers to the global sequence number carried by the data written to the first storage area (multi-replica area); the second sequence number refers to the global sequence number carried by the data written to the second storage area (erasure coding area). Both originate from the same global sequence number allocation mechanism and are used to compare and determine the relationship between different versions of data across storage areas during data recovery.

[0128] In this embodiment, the first step is to locate all possible physical locations of the data corresponding to the logical address to be recovered based on the index information. If the data of the logical address to be recovered is found to exist simultaneously in both the first storage area of ​​the multi-replica storage area and the second storage area of ​​the erasure coding storage area, then each piece of data and its corresponding sequence number (i.e., the first data and the first sequence number, the second data and the second sequence number) are read respectively. By comparing the size of the two sequence numbers (the larger the sequence number, the later the write time), the latest version of the data is determined, and this data is used as the final version for recovery. Data that exists in only one storage area is directly considered as valid data.

[0129] This solution effectively addresses the issue of multi-version data conflicts that may result from the failure to persist index information in a timely manner. By using sequence number-based time-series judgment, it ensures the accuracy and consistency of data recovery, thereby improving the reliability and fault recovery capabilities of the storage system.

[0130] Figure 6 A flowchart illustrating an embodiment of a data processing method in a block storage system provided in this application. Figure 6 The process shown is in Figure 1 Based on the illustrated process, the following steps are included:

[0131] Step 601: Receive a data read request, wherein the data read request carries the logical address of the data to be read;

[0132] Step 602: Query the index information based on the logical address of the data to be read to obtain the corresponding actual physical storage location;

[0133] Step 603: If the actual physical storage location is located within the logical address range allocated to the first storage area, read the data from the first storage area;

[0134] Step 604: If the actual physical storage location is located within the logical address range allocated to the second storage area, read the data from the second storage area.

[0135] Data read request: refers to the operation command initiated by the client to obtain data.

[0136] Logical address of the data to be read: refers to the location of the data to be read in the logical address space from the user's perspective.

[0137] In this embodiment of the application, after receiving a data read request, the index information is first queried to obtain the actual physical storage location corresponding to the data to be read. Then, the specific area of ​​data storage is determined according to the logical address range of the actual physical storage location: if it is located in the first storage area range, the data is read from the multi-replica SDB; if it is located in the second storage area range, the data is read from the EC Block.

[0138] This solution masks the differences in underlying data storage locations through a unified indexing and querying mechanism, providing a transparent data access interface to clients. This enables the system to efficiently and accurately locate and retrieve data within a hybrid storage architecture, providing clients with a consistent data access experience regardless of the storage region where the data is stored. Simultaneously, it fully leverages the respective advantages of low-latency access from multi-replica storage and high space utilization from erasure coding storage.

[0139] In another embodiment, the method may further include the following steps: after writing the data to be written to the target storage area, generating verification information for the written data; assigning a corresponding sequence number to the written data, wherein the sequence number assigned to the data written later is greater than the sequence number assigned to the data written earlier; embedding the sequence number into the verification information to verify data integrity and writing timing using the verification information.

[0140] Verification information: refers to the data integrity verification code calculated by algorithms such as checksum or CRC (Cyclic Redundancy Check), used to verify data integrity and write timing.

[0141] Serial number: A globally unique, monotonically increasing identifier bound to the HDB, used to precisely record the order in which data is written.

[0142] In this embodiment of the application, after the data is successfully written to the storage area, verification information for verifying the integrity of the data is first generated. Then, a globally monotonically increasing sequence number is assigned to the written data, and the sequence number is embedded as metadata into the verification information and stored together.

[0143] This scheme ensures the inseparability and consistency of the sequence number and data content by embedding verification information within the sequence number, providing a reliable timing basis for subsequent data recovery, deduplication, or consistency verification. Simultaneously, this mechanism effectively resolves data version conflicts that may arise from system crashes or untimely index persistence, significantly enhancing the data reliability and fault recovery capabilities of the storage system.

[0144] In another embodiment, the method may further include the following steps: monitoring the status of the first storage area and the second storage area in the target mixed data block; creating a new mixed data block when the status meets preset sealing conditions, wherein the preset sealing conditions include: the first storage area is full, the second storage area is full, the first storage area fails to write, or the second storage area fails to write; and allocating subsequently received data write requests to the new mixed data block for processing.

[0145] Preset sealing conditions: refers to the rules that trigger the HDB to stop accepting new data writes.

[0146] Creating a new hybrid data block: This refers to the system initializing a new HDB instance, containing an empty multi-replica first storage area and an erasure-coded second storage area, and updating the global address mapping relationship.

[0147] In this embodiment, the status of the two storage areas in the currently active HDB is monitored in real time. When either storage area is full or a write failure occurs, it is determined that the sealing conditions are met. The system will automatically create a new HDB instance as the target of subsequent write requests, and at the same time mark the original HDB as read-only to enter the subsequent garbage collection process.

[0148] This solution effectively avoids service interruptions caused by full or damaged storage blocks, ensuring the continuity and reliability of data writing. It also provides clear resource units for background garbage collection and space defragmentation, improving the overall maintainability and resource utilization efficiency of the storage system.

[0149] Figure 7 This is a block diagram illustrating an embodiment of a data processing apparatus in a block storage system provided in this application. Figure 7 As shown, the device includes:

[0150] The receiving module 71 is used to receive a data write request, wherein the data write request carries a target logical address and data to be written;

[0151] The first determining module 72 is used to determine a target mixed data block from multiple mixed data blocks according to the target logical address, wherein the logical address space of each mixed data block includes a first storage area using a multi-replica redundancy method and a second storage area using an erasure coding redundancy method.

[0152] The second determining module 73 is used to determine a target storage area between the first storage area and the second storage area of ​​the target mixed data block according to the data volume specification of the data to be written.

[0153] The writing module 74 is used to write the data to be written to the target storage area and obtain the actual physical storage location of the data to be written within the target mixed data block;

[0154] The update module 75 is used to update the index information according to the target logical address and the actual physical storage location, so as to record the mapping relationship between the target logical address and the actual physical storage location.

[0155] In one possible implementation, the second determining module is specifically used for:

[0156] Compare the data volume specification with a first preset threshold;

[0157] If the data volume is less than the first preset threshold, the first storage area is determined as the target storage area.

[0158] If the data volume specification is greater than or equal to the first preset threshold, the second storage area is determined as the target storage area.

[0159] In one possible implementation, the update module is specifically used for:

[0160] The mapping relationship between the target logical address and the actual physical storage location in the index information is updated to the memory index structure as a target index entry.

[0161] When the number of index entries stored in the memory index structure reaches a second preset threshold, the index entries stored in the memory index structure are asynchronously persisted to the metadata persistence area in the corresponding mixed data block. The metadata persistence area stores data in a multi-replica redundancy manner.

[0162] In one possible implementation, the apparatus further includes a generation module for:

[0163] After the data to be written is written to the target storage area, verification information is generated for the written data;

[0164] Assign a corresponding sequence number to the data that has been written, wherein the sequence number assigned to the data written later is greater than the sequence number assigned to the data written earlier;

[0165] The serial number is embedded in the verification information to verify data integrity and write timing.

[0166] In one possible implementation, the device further includes a recovery module for:

[0167] During the data recovery process, the storage status of the data corresponding to the logical address to be recovered in the target mixed data block is determined based on the index information.

[0168] If it is determined that the data corresponding to the logical address to be recovered exists in both the first storage area and the second storage area, the first data and the corresponding first sequence number in the first storage area, and the second data and the corresponding second sequence number in the second storage area are read respectively.

[0169] Compare the first serial number with the second serial number;

[0170] Data with larger serial numbers are identified as valid data and recovered.

[0171] In one possible implementation, the apparatus further includes a creation module for:

[0172] Monitor the status of the first storage area and the second storage area in the target mixed data block;

[0173] If the state meets the preset sealing conditions, a new hybrid data block is created, wherein the preset sealing conditions include: the first storage area is full, the second storage area is full, the first storage area fails to write, or the second storage area fails to write.

[0174] Subsequent data write requests are allocated to the new mixed data block for processing.

[0175] In one possible implementation, the device further includes a reading module for:

[0176] Receive a data read request, wherein the data read request carries the logical address of the data to be read;

[0177] The corresponding actual physical storage location is obtained by querying the index information based on the logical address of the data to be read.

[0178] If the actual physical storage location is located within the logical address range allocated to the first storage area, data is read from the first storage area;

[0179] If the actual physical storage location is within the logical address range allocated to the second storage area, data is read from the second storage area.

[0180] like Figure 8 As shown in the figure, this application provides a device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0181] Memory 113 is used to store computer programs;

[0182] In one embodiment of this application, when the processor 111 executes a program stored in the memory 113, it implements a data processing method in a block storage system provided in any of the foregoing method embodiments, including:

[0183] Receive a data write request, the data write request carrying a target logical address and the data to be written;

[0184] Based on the target logical address, a target mixed data block is determined from multiple mixed data blocks, wherein the logical address space of each mixed data block includes a first storage area using a multi-replica redundancy method and a second storage area using an erasure coding redundancy method;

[0185] Based on the data volume specifications of the data to be written, a target storage area is determined between the first storage area and the second storage area of ​​the target mixed data block;

[0186] Write the data to be written to the target storage area and obtain the actual physical storage location of the data to be written within the target mixed data block;

[0187] Based on the target logical address and the actual physical storage location, update the index information to record the mapping relationship between the target logical address and the actual physical storage location.

[0188] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data processing method in the block storage system provided in any of the foregoing method embodiments.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0191] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0192] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method in a block storage system, characterized by, The method includes: Receive a data write request, the data write request carrying a target logical address and the data to be written; Based on the target logical address, a target mixed data block is determined from multiple mixed data blocks, wherein the logical address space of each mixed data block includes a first storage area using a multi-replica redundancy method and a second storage area using an erasure coding redundancy method; Based on the data volume specifications of the data to be written, a target storage area is determined between the first storage area and the second storage area of ​​the target mixed data block; Write the data to be written to the target storage area and obtain the actual physical storage location of the data to be written within the target mixed data block; Based on the target logical address and the actual physical storage location, update the index information to record the mapping relationship between the target logical address and the actual physical storage location; The method further includes: Assign a corresponding sequence number to the data that has been written, wherein the sequence number assigned to the data written later is greater than the sequence number assigned to the data written earlier; During the data recovery process, the storage status of the data corresponding to the logical address to be recovered in the target mixed data block is determined based on the index information. If it is determined that the data corresponding to the logical address to be recovered exists in both the first storage area and the second storage area, the first data and the corresponding first sequence number in the first storage area, and the second data and the corresponding second sequence number in the second storage area are read respectively. Compare the first serial number with the second serial number; Data with larger serial numbers are identified as valid data and recovered.

2. The method of claim 1, wherein, The step of determining the target storage area between the first storage area and the second storage area of ​​the target mixed data block according to the data volume specifications of the data to be written includes: Compare the data volume specification with a first preset threshold; If the data volume is less than the first preset threshold, the first storage area is determined as the target storage area. If the data volume specification is greater than or equal to the first preset threshold, the second storage area is determined as the target storage area.

3. The method of claim 1, wherein, The step of updating the index information based on the target logical address and the actual physical storage location includes: The mapping relationship between the target logical address and the actual physical storage location in the index information is updated to the memory index structure as a target index entry. When the number of index entries stored in the memory index structure reaches a second preset threshold, the index entries stored in the memory index structure are asynchronously persisted to the metadata persistence area in the corresponding mixed data block. The metadata persistence area stores data in a multi-replica redundancy manner.

4. The method of claim 1, wherein, The method further includes: After the data to be written is written to the target storage area, verification information is generated for the written data; The serial number is embedded in the verification information to verify data integrity and write timing.

5. The method of claim 1, wherein, The method further includes: Monitor the status of the first storage area and the second storage area in the target mixed data block; If the state meets the preset sealing conditions, a new hybrid data block is created, wherein the preset sealing conditions include: the first storage area is full, the second storage area is full, the first storage area fails to write, or the second storage area fails to write. Subsequent data write requests are allocated to the new mixed data block for processing.

6. The method of claim 1, wherein, The method further includes: Receive a data read request, wherein the data read request carries the logical address of the data to be read; The corresponding actual physical storage location is obtained by querying the index information based on the logical address of the data to be read. If the actual physical storage location is located within the logical address range allocated to the first storage area, data is read from the first storage area; If the actual physical storage location is within the logical address range allocated to the second storage area, data is read from the second storage area.

7. A data processing device in a block storage system, the data processing device comprising: The device includes: A receiving module is used to receive a data write request, wherein the data write request carries a target logical address and data to be written; The first determining module is used to determine a target mixed data block from multiple mixed data blocks according to the target logical address, wherein the logical address space of each mixed data block includes a first storage area using a multi-replica redundancy method and a second storage area using an erasure coding redundancy method. The second determining module is used to determine a target storage area between the first storage area and the second storage area of ​​the target mixed data block according to the data volume specification of the data to be written; The writing module is used to write the data to be written to the target storage area and obtain the actual physical storage location of the data to be written within the target mixed data block; The update module is used to update the index information according to the target logical address and the actual physical storage location, so as to record the mapping relationship between the target logical address and the actual physical storage location; The device further includes a recovery module for: Assign a corresponding sequence number to the data that has been written, wherein the sequence number assigned to the data written later is greater than the sequence number assigned to the data written earlier; During the data recovery process, the storage status of the data corresponding to the logical address to be recovered in the target mixed data block is determined based on the index information. If it is determined that the data corresponding to the logical address to be recovered exists in both the first storage area and the second storage area, the first data and the corresponding first sequence number in the first storage area, and the second data and the corresponding second sequence number in the second storage area are read respectively. Compare the first serial number with the second serial number; Data with larger serial numbers are identified as valid data and recovered.

8. An electronic device, comprising: include: A processor and a memory, the processor being configured to execute a data processing program in a block storage system stored in the memory to implement the data processing method in a block storage system according to any one of claims 1-6.

9. A storage medium, characterized by The storage medium stores one or more programs, which can be executed by one or more processors to implement the data processing method in the block storage system of any one of claims 1-6.

Images