Data processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention relates to a data processing method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the coding operation of an erasure code on first data under the condition of receiving a write-in request for the first data, and obtaining a plurality of logic units and a plurality of fragments; according to the first identifier of the logic unit and the second identifier of each fragment, writing the plurality of fragments into a storage node; and under the condition that a reading request is received, obtaining a target first identifier and a target second identifier of the second data, reading the plurality of target fragments of the second data from the plurality of storage nodes, and carrying out merging processing to obtain the second data. Therefore, the data can be divided into the fragments with the unique identifiers for storage in the writing stage, and the target fragments can be quickly positioned according to the logic addresses and the recovered data can be merged in the reading stage, so that the metadata query overhead is reduced, the reading and writing efficiency is improved, and the storage and network transmission cost is reduced; and the overall performance of the distributed storage system in a high-concurrency scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment and storage medium. Background Technology

[0002] With the widespread adoption of cloud computing and big data, distributed storage systems need to simultaneously cope with the exponential growth in data volume and the continuous improvement in service quality requirements. Among these, ensuring data availability has become a core objective of system design, specifically manifested in data persistence and read-write consistency.

[0003] To ensure data integrity in the event of hardware failure or node downtime, existing distributed storage systems often employ multi-replica technology. Multi-replica strong consistency write schemes typically use a three-replica mechanism, ensuring data reliability by synchronously writing to multiple replicas. However, this approach can lead to low storage utilization, high write latency, and significant network overhead during data writing and reading. Therefore, optimizing the data read / write process in distributed storage systems has become a pressing issue. Summary of the Invention

[0004] In view of this, in order to solve the above-mentioned technical problems or some of the technical problems, the present invention provides a data processing method, apparatus, computer equipment and storage medium.

[0005] In a first aspect, embodiments of the present invention provide a data processing method, including: Upon receiving a write request for the first data, erasure coding is performed on the first data to obtain multiple logical units, and each logical unit is divided to obtain multiple fragments. Based on the first identifier of the logical unit and the second identifier of each of the fragments, the multiple fragments are written to the corresponding storage nodes respectively; Upon receiving a read request for the second data, obtain the first target identifier and the second target identifier corresponding to the logical address of the second data; Multiple target fragments of the second data are read from multiple storage nodes based on the first target identifier and the second target identifier; The second data is obtained by merging multiple target fragments.

[0006] In one possible implementation, before performing erasure coding on the first data, the method further includes: If a write record for the first data is found in the cache, it is determined that the first data has been written to the storage node; Alternatively, if no write record of the first data is found in the cache, determine whether the metadata of the first data is stored in the management node; When the management node stores the metadata of the first data, it is determined that the first data has been written to the storage node; When the management node does not store the metadata of the first data, it is determined that the first data has not been written to the storage node.

[0007] In one possible implementation, when the first data has not been written to the storage node, the erasure coding operation on the first data includes: The first data is divided into multiple logical units according to a preset size; For each of the logical units, it is divided into multiple data fragments according to erasure coding rules, and corresponding encoded fragments are generated for the multiple data fragments; Each of the data fragments and each of the encoded fragments are referred to as the fragments.

[0008] In one possible implementation, writing the plurality of fragments to their corresponding storage nodes according to the first identifier of the logical unit and the second identifier of each fragment includes: Assign a first identifier to the logical unit; A second identifier is assigned to each of the aforementioned shards, and a corresponding storage location in the storage node is assigned to each of the aforementioned shards; The metadata of the first data is stored in the management node, and each shard is written to the corresponding storage location; An index relationship is generated for each of the storage locations corresponding to the fragments, the first identifier, and the second identifier, and the index relationship is stored in the storage node.

[0009] In one possible implementation, the method further includes: If the first data has been written to the storage node, obtain multiple historical fragments corresponding to the historical first data; After merging multiple fragments and multiple historical fragments, the merged fragments are written to their respective storage nodes.

[0010] In one possible implementation, the step of reading multiple target fragments of the second data from multiple storage nodes based on the first target identifier and the second target identifier includes: The storage locations of multiple storage nodes corresponding to the second data are determined based on the first target identifier and the second target identifier. Multiple target fragments are read in parallel from the storage locations of multiple storage nodes.

[0011] In one possible implementation, the target fragment comprises: multiple target data fragments and target encoding fragments, wherein merging the multiple target fragments to obtain the second data includes: If all target data fragments corresponding to the second data are successfully read, the target data fragments are merged to obtain the second data. If not all target data fragments corresponding to the second data are successfully read, the unread target data fragments are recovered based on the currently successfully read target data fragments and target encoding fragments. After recovery, they are merged to obtain the second data.

[0012] In a second aspect, embodiments of the present invention provide a data processing apparatus, comprising: The processing module is configured to, upon receiving a write request for first data, perform erasure coding on the first data to obtain multiple logical units, and divide each of the logical units to obtain multiple fragments. The write module is used to write multiple fragments to corresponding storage nodes according to the first identifier of the logical unit and the second identifier of each fragment; The acquisition module is used to acquire the first target identifier and the second target identifier corresponding to the logical address of the second data when a read request for the second data is received. A read module is configured to read multiple target fragments of the second data from multiple storage nodes based on the first target identifier and the second target identifier; The merging module is used to merge multiple target fragments to obtain the second data.

[0013] Thirdly, embodiments of the present invention provide a computer device, including: a processor and a memory, wherein the processor is configured to execute a data processing program stored in the memory to implement the data processing method described in any one of the first aspects above.

[0014] Fourthly, embodiments of the present invention provide a storage medium storing one or more programs, which can be executed by one or more processors to implement the data processing method described in any one of the first aspects.

[0015] The data processing scheme provided in this invention involves, upon receiving a write request for first data, performing erasure coding on the first data to obtain multiple logical units, and dividing each logical unit into multiple fragments; writing the multiple fragments to their corresponding storage nodes based on the first identifier of the logical unit and the second identifier of each fragment; upon receiving a read request for second data, obtaining the target first identifier and target second identifier corresponding to the logical address of the second data; reading the multiple target fragments of the second data from the multiple storage nodes based on the target first identifier and target second identifier; and merging the multiple target fragments to obtain the second data. Therefore, data can be divided into fragments with unique identifiers during the write phase, and the target fragments can be quickly located and merged to restore the data during the read phase based on the logical address. This reduces metadata query overhead, improves read / write efficiency, lowers storage and network transmission costs, and enhances the overall performance of the distributed storage system in high-concurrency scenarios. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating another data processing method provided in an embodiment of the present invention; Figure 3 A schematic diagram of a data storage structure in data processing is provided in an embodiment of the present invention; Figure 4 A flowchart illustrating another data processing method provided in an embodiment of the present invention; Figure 5 A flowchart illustrating a data writing method provided in an embodiment of the present invention; Figure 6 A flowchart illustrating a data reading method provided in an embodiment of the present invention; Figure 7 A schematic diagram of the structure of a data processing system provided in an embodiment of the present invention; Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention; Figure 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0019] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method specifically includes: S11. Upon receiving a write request for the first data, perform erasure coding on the first data to obtain multiple logical units, and divide each logical unit to obtain multiple fragments.

[0020] The data processing method provided in this invention is applied to computer devices, including but not limited to servers, desktop computers, and tablets. For example, it can be applied to distributed storage systems for business scenarios such as log analysis, video-on-demand, financial transactions, and online backup. These businesses typically face high-concurrency read / write demands on massive amounts of data, as well as data reliability issues under node failures. This solution encodes the written data into multiple logical units and further divides it into sharded storage, enabling rapid data location and recovery based on logical addresses and the identifiers of each shard during reading. This satisfies both high availability and high performance requirements while reducing storage and transmission costs.

[0021] In this embodiment, the first data is the data block that needs to be written to the distributed storage system. When a user initiates a write request, the following parameters need to be passed: block_id: a globally unique identifier for the data block (first data), used to distinguish different data blocks in the storage system. offset: the offset of the write position, representing the starting point of the data write relative to the beginning position of the data block. length: the length of the data to be written, representing the size of the data being written. data: the content of the first data that the user actually wants to write. After receiving these parameters, the client (Data Client) performs erasure coding on the first data, dividing the first data into multiple logical units. Each logical unit is a stripe, and each logical unit is further divided into multiple fixed-size fragments. These fragments include multiple data fragments and encoded fragments (parities) obtained from encoding the data fragments for verification.

[0022] Specifically, erasure coding (EC) introduces redundant encoding into data, allowing for the recovery of lost data after partial loss. A stripe is the smallest logical unit of EC encoding (e.g., 256KB), divided into multiple fragments and parities (e.g., data fragments + encoding fragments). In distributed systems, RS(k, m) erasure coding is commonly used. Reed-Solomon (RS) coding is a frequently used erasure coding method in storage systems, with two parameters k and m, denoted as RS(k, m). k data blocks form a vector D, which is multiplied by a generator matrix B to obtain a data vector consisting of k data blocks and m parity blocks. If a data block is lost, it can be recovered through a series of calculations. RS(k, m) can tolerate the loss of up to m blocks (including data blocks and parity blocks). That is, the original data slice consists of k fragments, encoded into m parities, and stored in k+m nodes. Erasure coding overwrite requires the participation of at least m+1 nodes, including 1 data node and m parity block nodes. As the range of data to be overwritten increases, the number of nodes required also increases. RS(k, m) erasure coding tolerates at most m node loss, requiring k nodes to decode the original data. Therefore, overwrite updates must maintain data version consistency across at least k nodes to ensure that the update does not result in data loss. When a user writes an overwrite, the logical offset and size (offset and length) carried by the write request are divided into fixed-size logical units (Stripes). Then, each Stripe is split into smaller fixed-size data fragments (Fragments) according to the configured EC encoding rules. The EC encoded fragments (Parity) are calculated and used to write the Fragment and Parity data to the storage node cluster in the future.

[0023] As an example, such as Figure 2 The diagram above shows a data processing method provided by an embodiment of the present invention. The diagram shows a data block divided into multiple stripes, each stripe being 256KB in size.

[0024] The diagram shows a data block divided into four consecutive stripes from left to right. Each 256KB stripe is further divided into multiple smaller fragments, each 64KB in size, resulting in four fragments. These fragments are then erasure encoded to generate two additional parity fragments. The free spaces on storage nodes 0-3 store the original data fragments; the free spaces on storage nodes 4 and 5 store the parity fragments.

[0025] S12. Based on the first identifier of the logical unit and the second identifier of each fragment, write the multiple fragments to the corresponding storage nodes respectively.

[0026] In this embodiment, a globally unique version number is assigned to each logical unit as a first identifier, which is incremented during writing. A unique second identifier is also assigned to each shard, and each shard is written to a different storage node (Dataserver). During writing, the free physical location of each storage node is obtained (non-in-situ overwrite) to avoid partial write failures and contamination of old data in the event of a system crash.

[0027] The metadata of the first data is stored on the management node (Data master), and the metadata is managed through the Raft algorithm to ensure the consistency and high availability of the metadata; the reliability and fault tolerance of the data are ensured through a multi-replica mechanism, and the status of all storage nodes is monitored to form a cluster status.

[0028] Simultaneously, the memory of the storage node also stores a metadata structure (Direct Map), which records the index relationship between logical units and their corresponding metadata. This map represents the mapping relationship between the logical address, physical location (storage node + disk offset (offset and length)), and version number (first identifier and second identifier) ​​of each storage segment corresponding to each logical unit. In other words, the Direct Map records the mapping from the data offset to the disk offset. The metadata structure is stored in the memory of the storage nodes in the form of Direct Map blocks. Each Direct Map block represents a persistent copy of the Direct Map, employing a 3-replica strong consistency protocol. That is, for each logical unit, its corresponding shards are stored on different storage nodes, while the index relationship corresponding to the logical unit is stored in the memory of the storage nodes. The index relationship can be managed through the management node.

[0029] As an example, such as Figure 3 The diagram shown illustrates a data storage structure in data processing according to an embodiment of the present invention. After processing in step S11, the erasure code (EC Block) of a logical unit consists of an index relationship (DirectMap block) and multiple data blocks, supporting random read and write operations. Figure 3 In the EC block, "Replica" refers to an erasure coding replica, representing each shard obtained after the erasure coding operation, and "Disk" refers to the disk used for storage on the storage node. The Direct Mapblock is a 3-replica strongly consistent write block used for the persistence of strip metadata management for this EC block. The Data Block is a block composed of multiple Replicas (erasure coding replicas, including multiple data shards and coding shards). Each EC Replica stores corresponding data and is distinguished by an identifier. Taking RS 4+2 as an example, 0-3 are data shards written by the user, and 4 and 5 are coding shards, stored on different storage nodes. Simultaneously, the three index relationships are stored on different storage nodes.

[0030] S13. Upon receiving a read request for the second data, obtain the first target identifier and the second target identifier corresponding to the logical address of the second data.

[0031] In this embodiment, the second data is the data that needs to be read. The first target identifier represents the unique identifier of the logical unit contained in the second data, used to locate which logical unit the second data belongs to. The second target identifier represents the unique identifier of the fragment obtained after erasure coding of each logical unit, used to determine the specific fragment under that logical unit. Based on the first target identifier and the second target identifier, the system can determine which storage nodes the multiple fragments corresponding to the second data are stored on.

[0032] Specifically, the user initiates a data read request through the Data Client, carrying the logical address of the second data. The system converts the logical address into a target first identifier (Stripe) + target second identifier (Fragment and Parity) format by reading the index relationship. In other words, the second data request is split into read requests for multiple fragments (Fragment and Parity), and each request is retrieved from different storage nodes.

[0033] In this context, user read and write requests are based on logical addresses (e.g., "read 10MB of data starting from 100MB"). Address translation maps logical requests to specific storage stripes, fragments, and parities. Based on index relationships and metadata in the management node, the target first identifier corresponding to the logical unit contained in the current second data, and the target second identifier corresponding to the fragments in each logical unit, are obtained.

[0034] S14. Read multiple target fragments of the second data from multiple storage nodes based on the first target identifier and the second target identifier.

[0035] S15. Merge multiple target fragments to obtain the second data.

[0036] In this embodiment, steps S14 and S15 are described simultaneously. The client determines the storage nodes where multiple target fragments of the second data are currently located based on the determined first target identifier and second target identifier, and sends read requests to these storage nodes to read all target fragments. If all data fragments can be read normally, they are directly merged according to the fragment order to splice together the complete second data.

[0037] As an example, assuming no Parity is required, each Stripe is 1MB in size, consisting of 4 Fragments (256KB each), stored on 4 Data Servers. A user request reads a logical address offset of 2.5MB, with a length of 512KB. The logical address 2.5MB falls on the 3rd Stripe. The data in the 3rd Stripe is split into 4 Fragments: Fragment0: 0–256KB, Fragment1: 256–512KB, Fragment2: 512–768KB, and Fragment3: 768–1024KB. The logical address 2.5MB corresponds to the 512KB offset of Stripe3, falling within Fragment2.

[0038] The system breaks down user logic requests into multiple sub-requests. For example, it might retrieve 0–256KB of Fragment2 from DataServer2 and 0–256KB of Fragment3 from DataServer3. Ultimately, the client can assemble the required 512KB of continuous data.

[0039] The data processing method provided in this invention involves, upon receiving a write request for first data, performing erasure coding on the first data to obtain multiple logical units, and dividing each logical unit into multiple fragments; writing the multiple fragments to their corresponding storage nodes according to the first identifier of the logical unit and the second identifier of each fragment; upon receiving a read request for second data, obtaining the target first identifier and target second identifier corresponding to the logical address of the second data; reading the multiple target fragments of the second data from the multiple storage nodes according to the target first identifier and target second identifier; and merging the multiple target fragments to obtain the second data. Therefore, data can be divided into fragments with unique identifiers during the write phase, and the target fragments can be quickly located and merged to restore the data during the read phase, thereby reducing metadata query overhead, improving read and write efficiency, reducing storage and network transmission costs, and improving the overall performance of the distributed storage system in high-concurrency scenarios.

[0040] Figure 4 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the method specifically includes: S21. If a write record of the first data is found in the cache, determine that the first data has been written to the storage node; or, if a write record of the first data is not found in the cache, determine whether the metadata of the first data is stored in the management node; if the metadata of the first data is stored in the management node, determine that the first data has been written to the storage node; if the metadata of the first data is not stored in the management node, determine that the first data has not been written to the storage node.

[0041] In this embodiment, in this distributed storage system, each data block, when written, has its metadata recorded and managed on the management node (Master). Specifically, the Master maintains a fragment mapping table with the following format: fragment_id: i, representing the second identifier of the i-th fragment after the data block is split; server_id: j, indicating that the fragment is stored on the storage node (Data Server) identified by j. It also records Stripe_id, representing the first identifier of the stripe into which the written data is split. Thus, the storage location of each fragment within each stripe can be clearly recorded. Furthermore, a corresponding index relationship can be built based on the metadata of each stripe containing the written data for storage on the storage nodes (e.g., building an index based on offset / length to store user data split into stripes).

[0042] When a Data Client receives a write request, it first accesses the Master to obtain the metadata of the block. This metadata is then cached locally based on the block_id to avoid accessing the Master for every write, thereby reducing latency.

[0043] When a user initiates the first data write request, the parameters passed in can include block_id, offset, length, and data. The client first queries the local cache for the write record of the first data to be written based on block_id: if the data corresponding to block_id exists in the cache, it is determined that the write record exists, and it is determined that the history of the first data has been written to the storage node, which is an overwrite write scenario; If the data corresponding to the block_id does not exist in the cache, the client will further query the management node (Master): if the Master stores the metadata corresponding to the first data, it means that the first data has been written to the storage node; if the Master does not store the metadata of the first data, it is determined that the first data has not been written to the storage node in the past, which is the first write scenario.

[0044] S22. If the first data is not written to the storage node, the first data is divided into multiple logical units according to a preset size; for each logical unit, it is divided into multiple data fragments according to erasure coding rules, and corresponding encoded fragments are generated for the multiple data fragments; each data fragment and each encoded fragment are used as fragments.

[0045] In this embodiment, the first data is not written to the storage node, indicating that the first data is being written for the first time and needs to be split according to erasure coding to obtain multiple fragments for storage. At the same time, it indicates that the index relationship is empty and a corresponding index relationship needs to be generated for the first data for storage.

[0046] Specifically, the client divides the first data into multiple logical units according to a preset partition size. For each logical unit, erasure coding is performed: the logical unit is divided into multiple data fragments, and based on a preset erasure coding rule (e.g., RS(k, m)), a preset number of coded fragments (parities) are generated for each data fragment. All data fragments and coded fragments together constitute a complete redundant storage unit for that logical unit, resulting in multiple fragments. Subsequently, according to the mapping relationship between fragments and storage nodes, these fragments are distributed to different storage nodes for storage, achieving high data reliability and fault tolerance.

[0047] S23. Assign a first identifier to the logical unit; assign a second identifier to each shard and assign a storage location in the corresponding storage node to each shard; store the metadata of the first data to the management node and write each shard to the corresponding storage location; generate an index relationship between the storage location, the first identifier and the second identifier corresponding to each shard and store the index relationship in the storage node.

[0048] In this embodiment, a globally unique first identifier is assigned to each logical unit for quick location during subsequent storage and retrieval. A second identifier is assigned to each shard under each logical unit, uniquely corresponding to a shard within that logical unit. The management node maintains the mapping relationship between shards and storage nodes, used to allocate free storage locations within the storage nodes corresponding to each shard. Based on the allocation strategy returned by the management node, the client specifies a target storage node for each shard and determines the storage location of the shard on that node (e.g., physical offset or storage path). A correspondence is formed between the first identifier of the logical unit, the second identifier of each shard, and the storage location of the storage node where the shard is stored. Before writing to a shard, the client reports the information of the first identifier of the logical unit + the second identifier of each shard + the storage location as metadata to the management node and persists it, ensuring that the system can quickly resolve the location of the shard through the logical address in subsequent read / write requests.

[0049] The client sends each shard to the corresponding storage location on the storage node. The storage node receives the shards and writes them to disk. The client waits for the storage node to return a successful write response. After a shard is successfully written, the system generates an index for each shard. This index includes: the first identifier of the logical unit, the second identifier of the shard, and the storage location corresponding to the shard. The index is stored in the corresponding storage node as a DirectMap structure, serving as a local index structure for easy local shard location. In other words, the content written to the storage node includes: the index (DirectMap block) and multiple shards (Data Block).

[0050] Through a three-level mapping mechanism of first identifier, second identifier, and storage location, efficient data location and retrieval are achieved; centralized metadata management is combined with local indexing; and even in the event of node failure or partial fragment loss, complete data can still be recovered through indexing and erasure coding mechanisms.

[0051] S24. If the first data has been written to the storage node, obtain multiple historical fragments corresponding to the first historical data; merge the multiple fragments and the multiple historical fragments, and write the merged fragments to the corresponding storage nodes respectively.

[0052] In this embodiment, if the first data history has already been written to the storage nodes, the current first data to be written is determined to be an overwrite write process. The index relationship is read and loaded. Each piece of first data that has been written corresponds to an index relationship (Direct Map block). The client first reads the Direct Map block and loads the DirectMap data structure from it. The DirectMap represents the index relationship and can be a mapping table that records the correspondence between the logical address and physical location of each shard storage location. Through the DirectMap, the client can quickly locate the storage location of the historically stored shards. Based on the index relationship of the DirectMap, the client reads the historical shards from each storage node. The historical shards represent the storage state of the first data in previous versions.

[0053] Furthermore, based on the data offset and length of the first data being written, the client splits the newly written data into fragments and merges them with the historical fragments read: for ranges requiring updates, the new fragments overwrite the historical fragment content; for ranges not yet updated, the historical fragment data is retained. The result of the merging is a new set of fragments (including data fragments and checksum fragments). The client then rewrites the merged new fragments to the corresponding free storage locations on the storage nodes. After the write is complete, the client updates the DirectMap and Direct Mapblock, and synchronizes the new index relationships to the management node and the local memory of each storage node to ensure the correctness of data location.

[0054] This invention achieves efficient and secure data management in distributed storage by combining erasure coding overwrite mechanism with DirectMap indexing, versioned striping, and insert write strategy. During writes, data loss due to partial write failures is avoided; during reads, erasure coding recovery is supported, enabling data reconstruction even if some nodes fail; overwrite ensures consistency through append or redundant write mechanisms. By implementing EC processing logic on the client side, the amount of data transmitted over the network is reduced. Compared with traditional multi-replica and Raft append write, while maintaining high reliability, it significantly improves storage utilization, write security, and read / write performance, and can quickly locate the position of written data, providing an efficient and scalable solution for large-scale distributed storage.

[0055] As an example Figure 5 This is a flowchart illustrating a data writing method provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the method includes: The interaction nodes include: client (Data Client, DC), management node (Data Master, DM), and storage node (Data Server, DS). Data writing is implemented in four stages: "Block allocation → Data preparation → Data writing → Confirmation and persistence". The logic of each step is broken down below: Block Allocation Phase (Resource Initialization) 1. Block Request (EC Configuration): Initiated by the Data Center (DC), the DC requests storage locations from the Data Manager (DM) and specifies the EC (Erasure Coding) configuration for the block (such as the number of redundant fragments, coding algorithm, etc., for data fault tolerance). Purpose: To pre-allocate logical storage resources for write operations and agree on data redundancy strategies. 2. Block ID and DS List Allocation: Upon receiving the request, the DM assigns a unique Block ID to the block (for subsequent location and management); selects a set of Data Servers (DS list) as the actual storage nodes for the block. The Block ID and DS list are returned to the DC. Direct Map Loading into Memory: After obtaining the DS list, the DC requests the corresponding DS to load the Direct Map (logical address to physical address mapping table, i.e., index relationship) associated with the block into memory. During subsequent writes, the DC can quickly find the physical storage location through the Direct Map, avoiding disk access for each address lookup and improving efficiency.

[0056] Data preparation phase (processing the written content) ("loop" means that steps 3 and 4 are executed for each write request, and are executed cyclically during batch writes) 3. Locate the Stripe by logical address and merge the old and new data. The DC finds the corresponding Stripe in the Direct Map based on the logical address of the written data (a Stripe contains several data fragments and encoding fragments). If the Stripe has been written in the past (overwrite scenario), the DC will first read the old data and merge it with the newly written data (to ensure data integrity and avoid logical errors caused by writing only part of the fragments). If the past has not been written, then proceed to step 4. Split the data into multiple fragments through erasure coding. That is, based on the EC configuration agreed upon during initialization, the DC performs erasure coding on the merged data to generate several data fragments and redundant encoding fragments.

[0057] During the data writing phase (disk storage): 5. Send the Fragment (containing data fragments and encoded fragments) and a new version number (containing the first identifier and the second identifier). The DC sends the generated fragment, along with the new version number, to the previously determined DS. 6. Find a free location and write the data. After receiving the fragment, the DS searches for free physical space on its local disk (finding the location based on the Direct Map or disk management policy) and writes the fragment to the disk. 7. Update the Direct Map (in memory). After successful writing, the DS updates the Direct Map in memory (recording the physical address corresponding to the fragment) so that subsequent reads and writes can quickly address the data using the latest mapping, ensuring consistency between the memory index and physical storage.

[0058] During the confirmation and persistence phase (ensuring data reliability), 8. Confirm successful write (including Direct Map update): After completing the local write and Direct Map memory update, the DS returns a successful write confirmation to the DC. The DC receives confirmations from all DSs to confirm the write logic is complete. 9. Asynchronously persist the Direct Map to the Direct Map Block: To prevent data loss in memory (e.g., due to node restart), the DM asynchronously persists the Direct Map in memory to the Direct Map Block (metadata block, specifically storing mapping, version, and other metadata information) on disk. This allows the DM to manage metadata and resource allocation, the DS to manage actual data storage, and the DC to manage client logic and data processing, each performing its specific function to improve cluster scalability.

[0059] In one embodiment, reading multiple target fragments of second data from multiple storage nodes based on a first target identifier and a second target identifier includes: The storage locations of multiple storage nodes corresponding to the storage of the second data are determined based on the first target identifier and the second target identifier; multiple target fragments are read in parallel from the storage locations of the multiple storage nodes.

[0060] In this embodiment, when a user initiates a read request for the second data, the client first resolves the target first identifier (logical unit ID) and target second identifier (shard ID) corresponding to the second data based on the input logical address. The client then retrieves the metadata information of the second data from the local cache or management node based on these two identifiers. The metadata records the storage location of each shard within the storage nodes (including the storage node ID and physical location, such as offset or path). This allows the client to accurately determine which storage nodes and locations contain the multiple shards corresponding to the second data. Based on the resolved shard storage location information, the client sends read requests to multiple storage nodes in parallel. Upon receiving the request, each storage node reads the shard from its corresponding storage location and returns the result to the client. Parallel reading significantly improves read efficiency and reduces overall latency.

[0061] In one embodiment, the target fragment includes: multiple target data fragments and target encoding fragments. The multiple target fragments are merged to obtain second data, including: If all target data fragments corresponding to the second data are successfully read, the all target data fragments are merged to obtain the second data. If not all target data fragments corresponding to the second data are successfully read, the unread target data fragments are recovered based on the currently successfully read target data fragments and target encoding fragments. After recovery, they are merged to obtain the second data.

[0062] In this embodiment, the fragments returned by the storage node are divided into two categories: target data fragments: storing the user's actual data; and target encoded fragments: redundant fragments generated based on erasure coding, used for data recovery. If all data fragments corresponding to the target data are successfully read from the storage node: the client directly merges all target data fragments according to the fragment order, splicing them into complete second data; at this time, there is no need to use encoded fragments, which is the most efficient.

[0063] If, during the reading process, some target data fragments fail to be retrieved due to storage node failure, disk failure, or network anomalies, the client will trigger the erasure coding recovery mechanism using the successfully retrieved data fragments plus the corresponding encoded fragments. By performing matrix operations on the remaining fragments (such as inverting the Vandermonde matrix in RS coding), the missing target data fragments are reconstructed. After recovery, the client merges the recovered fragments with the remaining retrieved target data fragments to obtain the complete second data. This ensures that even if some storage nodes are unavailable, complete data can still be recovered through encoded fragments; it ensures that the recovered fragments maintain logical consistency with the original data, without affecting the user's reading experience; and by combining parallel reading and erasure coding recovery, it improves the system's tolerance to node failures.

[0064] As an example Figure 6 This is a flowchart illustrating a data reading method provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the method includes: The Data Client handles read and write requests initiated by users. The Data Master manages and synchronizes metadata information (such as the distribution location and version number of data shards). The Data Server stores the actual data shards and encoded shards; multiple servers form a storage cluster.

[0065] When upper-layer users need to read data through the Data Client, the Data Client splits the read IO into EC Stripe sizes based on a fixed size, then splits it into Fragments, and distributes them to different Data Servers to read the data. Once all data has been read, it can return the result to the upper layer.

[0066] Specifically, users initiate data read requests through the Data Client. The system converts the logical address into a "Stripe (logical unit) + Fragment" format, splitting the data read request into read requests for multiple fragments, and then reading the fragments from different Data Servers.

[0067] The Data Client sends read requests in parallel to multiple Data Server nodes (e.g., Data Server1, 2, and 3). Each Data Server returns its stored Fragment to the Data Client.

[0068] If all fragments return successfully, proceed to the next step; if some fail, the EC (erasure coding) recovery process is triggered: the lost fragments are recalculated based on the coded fragments with the same first identifier (version number) and the successfully returned data fragments; version consistency is verified; and missing data is recovered. The Data Client assembles all received data fragments into complete data and returns it to the user. Therefore, if a Data Server node fails, the system uses EC erasure coding to recover lost data, improving the integrity and accuracy of data reading. Data read and write operations improve data processing efficiency and performance through parallel fragmentation.

[0069] In one embodiment, direct overwrite writing of EC data in a distributed storage system poses a significant risk: if a node crashes, experiences network jitter, or suffers disk failure during the write process, some shard writes may fail. In this case, the new data has not been fully written to disk while the old data has been overwritten, potentially leading to data loss and compromising system durability and consistency. To provide a secure overwrite logic for upper-layer applications, the underlying system typically introduces two mechanisms: **Insertion write mechanism:** This mechanism avoids directly overwriting old data with new data. Instead, it finds an available physical location to write to. The Direct Map records the logical and physical correspondence and identifies available physical locations. This allows the system to revert to the old version even if some writes fail, ensuring no data loss. **Strip redundancy write mechanism (Stripe Replication):** During the overwrite process, newly generated shards are first written to multiple redundant replicas (e.g., different storage nodes within the same stripe). Once a majority of replicas have been successfully written, the metadata is updated to ensure data consistency across the stripe. This mechanism further reduces the risk of overwrite failures due to single points of failure.

[0070] Through the above mechanisms, the system can provide logically equivalent overwrite capabilities to upper-layer applications while ensuring data reliability and consistency, thereby enabling efficient and secure overwrite operations even in EC storage environments.

[0071] Figure 7 This is a schematic diagram of the structure of a data processing system provided in an embodiment of the present invention, such as... Figure 7 As shown, the system includes: The distributed storage system comprises: a client (Data Client), a management node (Data Master), and a storage node cluster (Data Server1, 2, 3) consisting of multiple storage nodes. The client receives requests from upper-layer applications and, based on erasure coding configuration policies, translates these requests into read / write requests for stored objects on the backend storage nodes. It is also responsible for requesting metadata from the Data Master and performing data read / write operations with the Data Servers. The storage node cluster includes multiple storage nodes used to store actual data shards and receive and respond to read / write requests for encoded shards. The management node manages the metadata of written data using the Raft algorithm, ensuring metadata consistency and high availability. A multi-replica mechanism ensures data reliability and fault tolerance, monitors the status of all storage service nodes to form the cluster status, and the Raft algorithm guarantees that the system can still function normally even when a few nodes fail. One node acts as the leader, and the other two as followers.

[0072] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the device includes: Processing module 31 is configured to, upon receiving a write request for first data, perform erasure coding on the first data to obtain multiple logical units, and divide each logical unit to obtain multiple fragments. The writing module 32 is used to write the multiple fragments to the corresponding storage nodes according to the first identifier of the logical unit and the second identifier of each fragment; The acquisition module 33 is used to acquire the first target identifier and the second target identifier corresponding to the logical address of the second data when a read request for the second data is received; Reading module 34 is used to read multiple target fragments of the second data from multiple storage nodes according to the first target identifier and the second target identifier; The merging module 35 is used to merge multiple target fragments to obtain the second data.

[0073] In one possible implementation, the processing module is further configured to determine that the first data has been written to the storage node if a write record of the first data is found in the cache; Alternatively, if no write record of the first data is found in the cache, determine whether the metadata of the first data is stored in the management node; When the management node stores the metadata of the first data, it is determined that the first data has been written to the storage node; When the management node does not store the metadata of the first data, it is determined that the first data has not been written to the storage node.

[0074] In one possible implementation, the processing module is specifically used to divide the first data into multiple logical units according to a preset size; For each of the logical units, it is divided into multiple data fragments according to erasure coding rules, and corresponding encoded fragments are generated for the multiple data fragments; Each of the data fragments and each of the encoded fragments are referred to as the fragments.

[0075] In one possible implementation, the writing module is specifically used to assign a first identifier to the logical unit; A second identifier is assigned to each of the aforementioned shards, and a corresponding storage location in the storage node is assigned to each of the aforementioned shards; The metadata of the first data is stored in the management node, and each shard is written to the corresponding storage location; An index relationship is generated for each of the storage locations corresponding to the fragments, the first identifier, and the second identifier, and the index relationship is stored in the storage node.

[0076] In one possible implementation, the acquisition module is further configured to acquire multiple historical fragments corresponding to the historical first data when the first data has been written to the storage node; The writing module is further configured to merge multiple fragments and multiple historical fragments, and then write the merged fragments to their respective storage nodes.

[0077] In one possible implementation, the reading module is specifically used to determine the storage location of multiple storage nodes corresponding to the second data based on the first target identifier and the second target identifier; Multiple target fragments are read in parallel from the storage locations of multiple storage nodes.

[0078] In one possible implementation, the merging module is specifically used to merge all the target data fragments to obtain the second data when all the target data fragments corresponding to the second data have been successfully read. If not all target data fragments corresponding to the second data are successfully read, the unread target data fragments are recovered based on the currently successfully read target data fragments and target encoding fragments. After recovery, they are merged to obtain the second data.

[0079] The data processing device provided in this embodiment may be as follows: Figure 8 The apparatus shown can perform all the steps of the above data processing method, thereby achieving the technical effect of the above data processing method. For details, please refer to the relevant descriptions in the above embodiments. For the sake of brevity, these will not be elaborated here.

[0080] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 9 The computer device 400 shown includes at least one processor 401, a memory 402, at least one network interface 404, and other user interfaces 403. The various components in the computer device 400 are coupled together via a bus system 405. It is understood that the bus system 405 is used to implement communication between these components. In addition to a data bus, the bus system 405 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 4 The general designated all buses as Bus System 405.

[0081] The user interface 403 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0082] It is understood that the memory 402 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 402 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0083] In some implementations, memory 402 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 4021 and application program 4022.

[0084] The operating system 4021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 4022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 4022.

[0085] In this embodiment of the invention, by calling the program or instructions stored in the memory 402, specifically the program or instructions stored in the application program 4022, the processor 401 executes the method steps provided in each method embodiment, including, for example: Upon receiving a write request for the first data, erasure coding is performed on the first data to obtain multiple logical units, and each logical unit is divided to obtain multiple fragments. Based on the first identifier of the logical unit and the second identifier of each of the fragments, the multiple fragments are written to the corresponding storage nodes respectively; Upon receiving a read request for the second data, obtain the first target identifier and the second target identifier corresponding to the logical address of the second data; Multiple target fragments of the second data are read from multiple storage nodes based on the first target identifier and the second target identifier; The second data is obtained by merging multiple target fragments.

[0086] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 402. Processor 401 reads the information in memory 402 and, in conjunction with its hardware, completes the steps of the above method.

[0087] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0088] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0089] The computer device provided in this embodiment may be as follows: Figure 9 The device shown can perform all the steps of the above data processing method, thereby achieving the technical effect of the above data processing method. For details, please refer to the relevant descriptions in the above embodiments. For the sake of brevity, these will not be elaborated here.

[0090] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0091] When one or more programs in the storage medium can be executed by one or more processors to implement the data processing method described above that is executed on the device side.

[0092] The processor is used to execute a data processing program stored in the memory to implement the following steps of a data processing method executed on the device side: Upon receiving a write request for the first data, erasure coding is performed on the first data to obtain multiple logical units, and each logical unit is divided to obtain multiple fragments. Based on the first identifier of the logical unit and the second identifier of each of the fragments, the multiple fragments are written to the corresponding storage nodes respectively; Upon receiving a read request for the second data, obtain the first target identifier and the second target identifier corresponding to the logical address of the second data; Multiple target fragments of the second data are read from multiple storage nodes based on the first target identifier and the second target identifier; The second data is obtained by merging multiple target fragments.

[0093] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0094] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0095] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, include: Upon receiving a write request for the first data, erasure coding is performed on the first data to obtain multiple logical units, and each logical unit is divided to obtain multiple fragments. Based on the first identifier of the logical unit and the second identifier of each of the fragments, the multiple fragments are written to the corresponding storage nodes respectively; Upon receiving a read request for the second data, obtain the first target identifier and the second target identifier corresponding to the logical address of the second data; Multiple target fragments of the second data are read from multiple storage nodes based on the first target identifier and the second target identifier; The second data is obtained by merging multiple target fragments.

2. The method according to claim 1, characterized in that, Before performing erasure coding on the first data, the method further includes: If a write record for the first data is found in the cache, it is determined that the first data has been written to the storage node; Alternatively, if no write record of the first data is found in the cache, determine whether the metadata of the first data is stored in the management node; When the management node stores the metadata of the first data, it is determined that the first data has been written to the storage node; When the management node does not store the metadata of the first data, it is determined that the first data has not been written to the storage node.

3. The method according to claim 2, characterized in that, If the first data has not been written to the storage node, the erasure coding operation on the first data includes: The first data is divided into multiple logical units according to a preset size; For each of the logical units, it is divided into multiple data fragments according to erasure coding rules, and corresponding encoded fragments are generated for the multiple data fragments; Each of the data fragments and each of the encoded fragments are referred to as the fragments.

4. The method according to claim 3, characterized in that, The step of writing multiple fragments to their corresponding storage nodes according to the first identifier of the logical unit and the second identifier of each fragment includes: Assign a first identifier to the logical unit; A second identifier is assigned to each of the aforementioned shards, and a corresponding storage location in the storage node is assigned to each of the aforementioned shards; The metadata of the first data is stored in the management node, and each shard is written to the corresponding storage location; An index relationship is generated for each of the storage locations corresponding to the fragments, the first identifier, and the second identifier, and the index relationship is stored in the storage node.

5. The method according to claim 3, characterized in that, The method further includes: If the first data has been written to the storage node, obtain multiple historical fragments corresponding to the historical first data; After merging multiple fragments and multiple historical fragments, the merged fragments are written to their respective storage nodes.

6. The method according to claim 1, characterized in that, The step of reading multiple target fragments of the second data from multiple storage nodes based on the first target identifier and the second target identifier includes: The storage locations of multiple storage nodes corresponding to the second data are determined based on the first target identifier and the second target identifier. Multiple target fragments are read in parallel from the storage locations of multiple storage nodes.

7. The method according to claim 3, characterized in that, The target fragment comprises: multiple target data fragments and target encoding fragments. The process of merging the multiple target fragments to obtain the second data includes: If all target data fragments corresponding to the second data are successfully read, the target data fragments are merged to obtain the second data. If not all target data fragments corresponding to the second data are successfully read, the unread target data fragments are recovered based on the currently successfully read target data fragments and target encoding fragments. After recovery, they are merged to obtain the second data.

8. A data processing apparatus, characterized in that, include: The processing module is configured to, upon receiving a write request for first data, perform erasure coding on the first data to obtain multiple logical units, and divide each of the logical units to obtain multiple fragments. The write module is used to write multiple fragments to corresponding storage nodes according to the first identifier of the logical unit and the second identifier of each fragment; The acquisition module is used to acquire the first target identifier and the second target identifier corresponding to the logical address of the second data when a read request for the second data is received. A read module is configured to read multiple target fragments of the second data from multiple storage nodes based on the first target identifier and the second target identifier; The merging module is used to merge multiple target fragments to obtain the second data.

9. A computer device, characterized in that, include: A processor and a memory, the processor being configured to execute a data processing program stored in the memory to implement the data processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the data processing method according to any one of claims 1 to 7.

Images