Data processing method, apparatus, device, storage medium, and program product

Abstract

The application discloses a data processing method, device and equipment, a storage medium and a program product. When data shards to be written into an append write system are received, an expected write offset of the data shards is determined. The append write system comprises at least two replicas, and each replica is configured with a storage space segment and a temporary data segment belonging to a same addressing space. A tail pointer value of the storage space segment of each replica is acquired. If the tail pointer value is continuous with the expected write offset, the data shards are written into the storage space segment of the replica, and the tail pointer is updated. If the tail pointer value is not continuous with the expected write offset, the data shards are written into the temporary data segment of the replica. By adopting the embodiment of the application, the long tail problem of the append write system in a weak network environment can be effectively solved, and the weak network adaptability and data read-write stability are improved while the advantages of system simplicity, high efficiency and strong consistency are retained.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] An append-only system is a storage system in which data can only be appended sequentially to the end of the storage space and cannot be randomly modified. Its core advantage lies in its sparse indexing mechanism based on the first data address and the offset within the segment, which significantly reduces indexing overhead. At the same time, it adapts to the physical characteristics of flash memory media, achieving efficient addressing and high throughput. It has become the core paradigm of high-performance storage in scenarios such as cloud computing, big data, and edge computing.

[0003] In traditional append-write systems, data storage strictly relies on continuity constraints. When deploying data redundancy mechanisms across multiple nodes, it is essential to ensure that the storage space segments of each replica maintain a complete and continuous data sequence. If some nodes experience data loss or write delays due to factors such as network jitter, high disk load, or unstable cross-availability zone links, it directly disrupts data continuity, causing the system to fail to meet the core requirements of a strong consistency model. This phenomenon is known as the long-tail problem, which is particularly frequent in weak network environments such as cross-availability zone redundancy and multi-site active-active deployments. In such cases, the system must wait for all nodes to complete data synchronization before confirming a successful write. The long-tail latency of some nodes can lead to overall write blocking, significantly reducing system throughput and response speed, and severely limiting the applicability of append-write systems in weak network scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a data processing method, apparatus, device, storage medium, and program product that can effectively solve the long-tail problem of append-write systems in weak network environments, while retaining the advantages of system simplicity, efficiency, and strong consistency, and improving the adaptability to weak networks and the stability of data read and write.

[0005] To achieve the above objectives, embodiments of the present invention provide a data processing method, comprising:

[0006] Receive data fragments to be written to the append write system; wherein the append write system includes at least two replicas, and each replica is configured with a storage space segment and a temporary data segment belonging to the same address space;

[0007] Determine the expected write offset of the data shard and obtain the tail pointer value of the storage space segment of each replica;

[0008] If the tail pointer value is continuous with the expected write offset, the data is written to the storage space segment of the replica, and the tail pointer is updated.

[0009] If the tail pointer value is not continuous with the expected write offset, the data is fragmented and written to the temporary data segment of the replica.

[0010] As an improvement to the above scheme, determining the expected write offset of the data shard includes:

[0011] Obtain the tail length of all replicas in the append write system;

[0012] The maximum value among the tail lengths of all the replicas is determined as the expected write offset of the data shard.

[0013] As an improvement to the above solution, the method further includes:

[0014] Get the target offset of the data to be read;

[0015] When the target offset is located within the storage space segment of the replica, data is read from the storage space segment;

[0016] When the target offset is not within the storage space segment of the replica, query the temporary data segment of the replica. If the target offset exists in the temporary data segment, read the data from the temporary data segment.

[0017] As an improvement to the above scheme, the storage space segment is configured with a sparse sharding index, and the temporary data segment is configured with a dense sharding index; wherein, the sparse sharding index is used to locate the contiguously stored data shards in the storage space segment, and the dense sharding index is used to locate the sparsely stored data shards in the temporary data segment.

[0018] As an improvement to the above solution, the method further includes:

[0019] Obtain the tail length of the copy in the append write system;

[0020] The target copy containing missing data is determined based on the tail length;

[0021] The missing data of the target copy is retrieved from other copies to perform data repair on the target copy.

[0022] As an improvement to the above scheme, the tail length is the maximum value of the first offset value and the second offset value in the copy; wherein, the first offset value is the tail pointer value of the storage space segment, and the second offset value is the tail offset of the temporary data segment.

[0023] To achieve the above objectives, embodiments of the present invention also provide a data processing apparatus, comprising:

[0024] A data fragment receiving module is used to receive data fragments to be written to the append write system; wherein, the append write system includes at least two replicas, and each replica is configured with a storage space segment and a temporary data segment, wherein the storage space segment and the temporary data segment belong to the same address space;

[0025] An expected write offset determination module is used to determine the expected write offset of the data fragment;

[0026] The tail pointer value acquisition module is used to obtain the tail pointer value of the storage space segment of each copy;

[0027] The data writing module is configured to write the data fragments into the storage space segment of the replica and update the tail pointer if the tail pointer value is continuous with the expected write offset; and to write the data fragments into the temporary data segment of the replica if the tail pointer value is not continuous with the expected write offset.

[0028] To achieve the above objectives, embodiments of the present invention also provide a data processing device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the data processing method as described in any of the above embodiments.

[0029] To achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the data processing method as described in any of the above embodiments.

[0030] To achieve the above objectives, embodiments of the present invention also provide a computer program product, including computer instructions, which, when executed by a processor, implement the data processing method as described in any of the above embodiments.

[0031] Compared to existing technologies, the data processing method, apparatus, device, storage medium, and program product disclosed in this invention configures storage space segments and temporary data segments within the same address space for replicas of the append-write system. It achieves differentiated writing by combining the continuity judgment of the expected write offset and the tail pointer value. When continuous, it directly writes to the storage space segment to ensure efficient appending; when discontinuous, it writes to the temporary data segment to avoid write blocking. This approach addresses the high-throughput write requirements of append-write scenarios while resolving data write anomalies caused by offset breaks through the temporary storage mechanism. Furthermore, it enhances data reliability through a multi-replica architecture, achieving efficient, stable, and reliable data processing. This invention effectively solves the long-tail problem of append-write systems in weak network environments, improving weak network adaptability and data read / write stability while retaining the system's advantages of simplicity, efficiency, and strong consistency. Attached Figure Description

[0032] Figure 1 This is a diagram comparing the data indexes of append-only and random-write systems provided by existing technologies;

[0033] Figure 2 This is a schematic diagram illustrating the writing of different tail position states under the multi-replica data redundancy provided by existing technology;

[0034] Figure 3 This is a functional diagram of the storage space segment and temporary data segment provided in an embodiment of the present invention;

[0035] Figure 4 This is a flowchart of data writing in the data processing method provided in the embodiments of the present invention;

[0036] Figure 5 This is a schematic diagram of the initial state of each replica node before writing data fragments, provided in an embodiment of the present invention.

[0037] Figure 6 This is a schematic diagram of the status of each replica node after the first shard is written, provided in an embodiment of the present invention.

[0038] Figure 7 This is a schematic diagram of the status of each replica node after writing to the second shard, provided in an embodiment of the present invention.

[0039] Figure 8 This is a schematic diagram of the system state after a write operation is performed on a data missing node, as provided in an embodiment of the present invention.

[0040] Figure 9 This is a schematic diagram of the state of a multi-segment system after multi-node long-tail jitter provided in an embodiment of the present invention;

[0041] Figure 10 This is a flowchart of data reading in the data processing method provided in the embodiments of the present invention;

[0042] Figure 11 This is a flowchart of data repair in the data processing method provided in the embodiments of the present invention;

[0043] Figure 12 This is a schematic diagram of data repair provided in an embodiment of the present invention;

[0044] Figure 13 This is a structural block diagram of a data processing device provided in an embodiment of the present invention;

[0045] Figure 14 This is a structural block diagram of a data processing device provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] In existing technologies, the main solution to the long-tail problem in storage systems is based on quorum (a quorum / arbitration mechanism). In distributed systems, quorum is a core protocol for ensuring data consistency and availability. Its core logic is to balance data consistency, availability, and fault tolerance in multi-replica deployment scenarios by defining a minimum number of replicas required for read and write operations. In principle, quorum allows the system to achieve strong data consistency when R+W>N, where R is the number of nodes in the redundancy group that the system needs to access during a read, W is the minimum number of nodes in the redundancy group that the system needs to complete a write operation, and N is the total number of nodes in the redundancy group. This means that the system can still maintain overall data consistency even when some nodes in the redundancy group are unavailable or unavailable due to long-tail latency. However, quorum technology typically relies on complex consistency algorithms (such as Paxos and Raft), and their complexity and overhead, such as synchronized logs, limit their implementation in high-performance systems.

[0048] One variation of the quorum approach is to use erasure coding to address the problems of the aforementioned strong consistency systems. Because erasure coding uses an N+M sharding design, it essentially allows normal read and write operations even when M shards are unavailable (or the long tail is unavailable). However, due to the stripe size limitation of erasure coding, it cannot meet latency requirements in scenarios with small data shards, and in some scenarios, it may even fail to gather enough data, leading to write blocking. In contrast, the replica approach supports writes of any size and does not have the problem of handling differences between large and small shards. Meanwhile, the append-only write approach based on a multi-replica master-slave strong consistency model possesses the simplicity and high performance lacking in the quorum system; however, its strong consistency model limits its tolerance for the loss of some nodes, making it unable to solve the long tail problem in weak networks.

[0049] Common storage systems typically categorize data organization into append-only and random-write systems. In append-only systems, multiple data fragments are continuously written to a contiguous segment of storage space (usually contiguous physical addresses). Because data storage is contiguous, data addressing can be easily performed using the address of the first fragment and its offset within the segment, significantly reducing the sparse index required for addressing data fragments. However, in random-write systems, multiple contiguous fragments may be allocated to random, non-contiguous storage spaces, making data location impossible using the append-only indexing method. The fragment index must record addressing information for each fragment in a fine-grained manner, resulting in a dense index. This significantly impacts both the index size and indexing efficiency.

[0050] See Figure 1 , Figure 1 This is a diagram comparing the data indexes of append-only and random-write systems provided by existing technologies. Figure 1 In the diagram, "ABC" represents the contiguous data fragments to be written to the front end, and "abc" represents the fragments actually written to by the storage system. This correspondence illustrates the storage differences between append-only and random-write systems. In the append-only system on the left, the contiguous fragments A, B, and C at the front end are written to contiguous storage space in the storage layer (corresponding to a, b, and c), and the order is consistent with the front end. This reflects the characteristics of contiguous storage, so addressing only requires recording the starting address plus the offset using a sparse index. In the random-write system on the right, the contiguous fragments A, B, and C at the front end are allocated to non-contiguous space in the storage layer (the actual writing order becomes b, a, and c). The position of each fragment is random, therefore, a dense index needs to be recorded for each fragment to achieve data location.

[0051] Therefore, append-only systems have advantages in data index management, which not only simplify the access process and improve performance, but also match the physical characteristics of flash NAND-based flash memory media, making them an efficient system paradigm for high-performance storage systems.

[0052] However, when implementing multi-replica data redundancy based on append-only systems, the data continuity model cannot tolerate gaps caused by missing data within storage segments. Therefore, only a strong consistency model can be used, where the write succeeds when all append writes are continuous. If a replica experiences data loss or latency, causing the current write to become discontinuous, the write operation fails.

[0053] See Figure 2 , Figure 2This is a schematic diagram illustrating the writing of data to different tail positions under multiple copy data redundancy provided by existing technology. Figure 2 This diagram illustrates the impact of replica tail position continuity on write results in a multi-replica append-write system. The left-hand diagram represents a successful write, where the tail pointer states of replicas 1, 2, and 3 are consistent (all marked with "√"), indicating that the current write offset of all replicas is continuous with the expected offset. In this case, the system determines that the multiple replicas are synchronized and can successfully write the data to be written to the storage segments of all replicas, maintaining the continuity of the append write. The right-hand diagram represents a failed write, where the tail pointer states of replicas 1 and 2 are normal ("√"), but the tail pointer state of replica 3 is abnormal ("×"). This indicates that replica 3, due to network jitter, high disk load, etc., has a discontinuous current write offset from the expected offset, resulting in data gaps. Because the append-write system requires strict continuity and consistency of data across multiple replicas, this write operation will fail, and the data append cannot be completed.

[0054] Typically, the placement of multiple replicas varies depending on the system's redundancy requirements. For example, in scenarios with single availability zone tolerance, multiple racks / cabinets with physical topology redundancy can usually be selected within a single availability zone for data placement. In scenarios with multi-availability zone tolerance, data can typically be placed independently within multiple availability zones. Regardless of the redundancy topology, multiple replicas may experience long-tail writes due to factors such as network jitter and high disk load. With a fixed long tail, the system can still achieve strong consistency for the remaining replicas by removing related slow nodes. However, due to the randomness of the long-tail phenomenon, the system cannot effectively circumvent this problem by frequently removing nodes.

[0055] This invention provides a data processing method that addresses the problem of discontinuous writes to individual replicas causing overall write failures in multi-replica append-write systems by adding an auxiliary area called a temporary data segment outside the main storage area (the storage space segment). Each replica's storage space segment is configured with an independent temporary data segment. The storage space segment still only supports append writes and is configured with a sparse sharding index. Data within the temporary data segment is also append-only, but it is configured with a dense sharding index, thus allowing for sparse data sharding. Furthermore, the temporary data segment and the storage space segment belong to the same address space; therefore, the dense sharding index uses the address of the first data shard within the storage space segment as a relative address to address the data within the temporary data segment it records.

[0056] For example, see Figure 3 , Figure 3This is a functional diagram of the storage space segment and temporary data segment provided in an embodiment of the present invention. This embodiment configures a dual-region architecture for each replica: a storage space segment (primary storage) + a temporary data segment (auxiliary storage). Both belong to the same address space, and differentiated storage and indexing strategies ensure compatibility with discontinuous data scenarios. The storage space segment is the primary storage area of ​​the replica, supporting only continuous append writes. It is configured with a sparse shard index, using the base addr (the address of the first data shard) as a reference. Data can be located by the offset at the end of the append write, ensuring the efficiency of regular continuous writes. The temporary data segment is the auxiliary storage area of ​​the replica, also supporting append writes, but allowing sparse data shards. It is configured with a dense shard index, whose addressing is relative to the base addr of the storage space segment, such as base+10 and base+50 in the diagram. The offset position of each shard is recorded separately to accommodate data when the storage space segment writes are discontinuous.

[0057] When a copy's write offset becomes discontinuous due to network or disk issues, the data is written to a temporary data segment instead of failing directly. The relative address is recorded using a dense index. If the storage space segment subsequently becomes contiguous, the fragments of the temporary data segment can be merged into the main segment. This solves the failure problem caused by discontinuous writes, maintains the uniformity of the address space, and balances the efficiency of append writes with system fault tolerance.

[0058] Based on the two data storage areas mentioned above, this invention allows newly written data to be directly written to the end of the storage space segment if it is contiguous, and the append write tail pointer is updated, returning a write success response to the requesting end. If the newly written data is not contiguous on the storage space segment, the data is fragmented and written to a temporary data segment, and the relevant data addressing information is updated in the dense fragment index, while the append write tail pointer of the storage space segment remains unchanged.

[0059] See Figure 4 , Figure 4 This is a flowchart of data writing in the data processing method provided in the embodiment of the present invention, wherein the data processing method includes steps S11 to S14.

[0060] S11. Receive data fragments to be written to the append write system; wherein the append write system includes at least two replicas, and each replica is configured with a storage space segment and a temporary data segment belonging to the same address space.

[0061] For example, the system first determines the list of writable nodes within the current replica group by monitoring replica status (such as tail pointer continuity, node load, etc.). Assume that three writable nodes that meet the criteria are selected this time: replica 1, replica 2, and replica 3. Simultaneously, the system's consistency model is configured as: R=2, W=2, N=3, meaning that a write operation must succeed in at least two replicas before a write success is announced to the business side. Initially, its storage space segment and temporary data segment are empty, containing no data. Figure 5 As shown, Figure 5 This is a schematic diagram of the initial state of each replica node before writing data fragments according to an embodiment of the present invention. Before the write operation is initiated, there is no data in the storage space segment and temporary data segment of each replica. The base address of the storage space segment of replicas 1 to 3 has been initialized, and the offset of the append write tail is 0 (indicating no data). The temporary data segment is also empty, and the corresponding dense fragment index has not yet recorded any content, and the state is NUL.

[0062] S12. Determine the expected write offset of the data shard and obtain the tail pointer value of the storage space segment of each replica.

[0063] For example, the expected write offset refers to the target contiguous address range within the storage area of ​​the data fragment to be written in an append-only write system. The tail pointer value is an identifier of the end of the written data in the replica storage segment. For example, after the preceding write is completed, the tail pointer value is updated to 8KB, indicating that the current storage segment has been used up to the 8KB offset position.

[0064] Further, determining the expected write offset of the data shard includes: obtaining the tail length of all replicas in the append write system; and determining the maximum value among the tail lengths of all replicas as the expected write offset of the data shard.

[0065] For example, the tail length is the farthest offset of written data in a single replica. The tail length is the maximum of a first offset value and a second offset value in the replica; wherein the first offset value is the tail pointer value of the storage space segment, i.e., the end offset of the written data in the main storage area, and the second offset value is the tail offset of the temporary data segment, i.e., the end offset of the written data in the auxiliary storage area.

[0066] For example, if the tail pointer value (first offset) of the storage space segment in replica 1 is 100, and the offset of the last fragment of the temporary data segment (second offset) is 150, then the tail length of replica 1 is 150. If the first offset value of replica 2 is 120, and the second offset value is 120, then the tail length of replica 2 is 120. If the first offset value of replica 3 is 90, and the second offset value is 130, then the tail length of replica 3 is 130. Determining the maximum value of the tail lengths of all replicas as the expected write offset for the data fragment means using the furthest offset of already written data among all replicas as a reference to determine the target starting position of the data to be written. Taking the above three replicas as an example, the maximum tail length of all replicas is 150. Therefore, the expected write offset for this data fragment is the range corresponding to "150 + data fragment size". For example, if the data fragment size is 50, then the expected write offset is 150~200.

[0067] In this embodiment of the invention, the farthest written position of the entire replica group is used as the writing benchmark for new data, ensuring that the new data can connect with the farthest existing data in the replica group, avoiding address overlap or gaps between the new data and existing data due to the lag in writing some replicas, while also being compatible with the writing status of storage space segments and temporary data segments, allowing the expected offset to adapt to the dual-region storage architecture.

[0068] S13. If the tail pointer value is continuous with the expected write offset, the data fragments are written to the storage space segment of the replica, and the tail pointer is updated.

[0069] For example, the continuity between the tail pointer value and the expected write offset means that the tail pointer value of the replica storage segment (i.e., the end offset of the data already written in the main storage) is exactly equal to the starting value of the expected write offset for this data fragment. For instance, if the tail pointer value is 150 and the starting value of the expected write offset is also 150, it indicates that the data already written in the main storage area and the target position of the new data are seamlessly connected without gaps. In this case, the system will directly write the data fragment to the storage segment of the replica, completing the continuous append. After successful writing, the tail pointer value of the replica is updated to the end value of the expected write offset. For example, if the data fragment size is 50, the tail pointer value is updated from 150 to 200, thus recording the latest written end position in the main storage area and providing a benchmark for the offset calculation of the next write. This operation not only ensures the continuous storage characteristics of the storage segment but also maintains the orderliness of append writes by updating the tail pointer. At the same time, subsequent addressing can be achieved by relying on sparse indexes, balancing write efficiency and retrieval efficiency.

[0070] S14. If the tail pointer value is not continuous with the expected write offset, the data is fragmented and written to the temporary data segment of the replica.

[0071] For example, a discontinuity between the tail pointer value and the expected write offset means that the tail pointer value (e.g., 120) of the replica storage segment is inconsistent with the starting value of the expected write offset (e.g., 150), resulting in address gaps (i.e., data holes). This situation is often caused by network jitter, high disk load leading to replica write lag, or missing preceding data, failing to meet the strict requirements of append-write systems for continuous storage. In this case, the system writes data fragments to temporary data segments. This area supports non-contiguous data storage and is configured with dense indexes, temporarily storing data that cannot be written continuously to the main segment. This step avoids write failures caused by storage discontinuity, ensuring data integrity, and maintains the logical order of data according to the expected offset through temporary storage, thus being compatible with the characteristics and fault tolerance requirements of append-write systems.

[0072] To better explain the above steps S11-S14, the present invention provides the following specific embodiments:

[0073] 1) Assume a client initiates a write operation on data shard A. The client determines the write offset to be 0 based on the system state. Replicas 1, 2, and 3 have all completed the write operation. At this point, the system state is as follows: Figure 6 As shown, Figure 6 This is a schematic diagram of the status of each replica node after writing the first shard, provided by an embodiment of the present invention. The base_addr of storage space segment A is the starting address. The appended write tail pointer shows that the current write has reached offset 100, indicating that data shard A has been successfully written to the main storage area and the tail pointer has been updated. The temporary data segment currently has no data (arrow points to NUL), and the dense shard index is not associated with any shard information, indicating that this write did not trigger the temporary storage mechanism because the tail pointer value is continuous with the expected offset. All replicas complete the continuous appending of data shard A through the storage space segment.

[0074] 2) The client continues to initiate the writing of data shard B. Based on the system status, the client determines the write offset to be 100. Replicas 1 and 2 have completed the write operation. Replica 3, due to a temporary network failure, did not perform the write operation for this data shard. At this point, the system statuses are as follows: Figure 7 As shown, Figure 7This is a schematic diagram illustrating the state of each replica node after writing the second shard, as provided in this embodiment of the invention. The storage segments of replicas 1 and 2 already contain data shards A and B, and the append write tail pointer is updated to 130. The temporary data segment and dense shard index are empty. The storage segment of replica 3 only retains data shard A, and the append write tail pointer remains at 100. Data shard B has not been written, and the temporary data segment and dense shard index are also empty. It should be noted that at this point, the append write tail pointers of replicas 1 and 2 have been updated, but replica 3 has not. Under the consistency model, the write to data shard B is successful, meaning it tolerates the unavailability of one long-tail node. However, this operation also causes a difference in the length of the storage segments of each replica. At this point, the storage segment length of replicas 1 and 2 is 130, while the storage segment length of replica 3 is 100.

[0075] 3) Since replica 3 is a temporary network jitter, the system will not remove it from the replica group. Therefore, subsequent write operations will still be sent to all replicas 1, 2, and 3. The client continues to initiate the write of data shard C, and the write offset should be 130 according to the system status. According to the aforementioned rules, the write offsets of replicas 1 and 2 are contiguous with the append write tail of the storage space segment, so the shard to be written can be directly written to the storage space segment. However, for replica 3, because the tail is not contiguous, the shard to be written will be written to the temporary data segment.

[0076] See Figure 8 , Figure 8 This is a schematic diagram of the system state after a write operation is performed on a missing data node according to an embodiment of the present invention. The append write tail (currently 130) and write offset (130) of the storage space segments of replicas 1 and 2 are continuous, so data fragment C is directly written to the storage space segments of both. After the write is completed, the storage space segments of replicas 1 and 2 contain A, B, and C, and the append write tail is updated to 180. The temporary data segment and dense shard index are empty. The append write tail (currently 100) and write offset (130) of the storage space segment of replica 3 are not continuous, so data fragment C is written to its temporary data segment. The dense shard index records the offset (offset=130) and size (size=50) of the fragment. At this time, the storage space segment of replica 3 only retains A, and the append write tail is still 100. Thus, all three replicas have completed the persistence of the data to be written, so this write operation returns successful. It can be seen that, based on this method, the system can continue to write data in a state of data inconsistency caused by long-tail networks and ensure strong data consistency.

[0077] It should be noted that subsequent data writing may also result in various combinations of states. For example, replica 1 and replica 2 may experience temporary jitter leading to data loss. This invention will not list them all, such as... Figure 9 As shown, Figure 9This is a schematic diagram of the state of a multi-sharded system after long-tail jitter in a multi-node system, as provided in an embodiment of the present invention. Replica 1 experiences no jitter, and A, B, C, D, and E are continuously written to its storage space segment. Replica 2, due to jitter, only writes A, B, and C to its storage space segment, with E subsequently written to a temporary segment (index record offset=190, size=10). Replica 3 experiences continuous jitter, with only A remaining in its storage space segment, while C, D, and E are all written to a temporary segment (index record C's offset=130, size=50, and E's offset=190, size=10). Thus, it can be seen that due to varying degrees of jitter, the distribution and combination of data in the storage space segment and temporary data segment differ among replicas, but all achieve data persistence.

[0078] In this embodiment of the invention, by configuring storage space segments and temporary data segments within the same address space for replicas of the append-write system, and combining the continuity judgment of the expected write offset and the tail pointer value, differentiated writing is achieved. When the offsets are continuous, the data is directly written to the storage space segment to ensure efficient appending; when the offsets are discontinuous, the data is written to the temporary data segment to avoid write blocking. This approach addresses the high-throughput write requirements of append-write scenarios while resolving data write anomalies caused by offset breaks through the temporary storage mechanism. Furthermore, the multi-replica architecture enhances data reliability, achieving efficient, stable, and reliable data processing. This invention effectively solves the long-tail problem of append-write systems in weak network environments, improving weak network adaptability and data read / write stability while retaining the system's advantages of simplicity, efficiency, and strong consistency.

[0079] See Figure 10 , Figure 10 This is a flowchart of data reading in the data processing method provided in the embodiment of the present invention, wherein the data processing method includes steps S21 to S23.

[0080] S21. Obtain the target offset of the data to be read.

[0081] For example, the target offset refers to the specific address identifier of the data fragment to be read within the unified address space in a data storage system. It corresponds to the starting offset position and length range of the data fragment and is the core basis for the system to locate and extract the target data. For instance, in a log query scenario, when maintenance personnel need to retrieve system logs for a certain period, the system will map the log generation time to the corresponding target offset. For example, an offset of 1500~2000 corresponds to log data from 10:00 to 10:30, quickly locating and returning the target log fragment. In a video fragment loading scenario, when a user drags the progress bar to the 10-minute mark, the client will convert that playback position into the target offset of the video data. For example, an offset of 50MB~55MB corresponds to the video fragment at the 10-minute mark, requesting data within that offset range from the storage system to achieve smooth video transitions.

[0082] S22. When the target offset is located within the storage space segment of the copy, read data from the storage space segment.

[0083] It should be noted that the storage space segment is configured with a sparse shard index, which is used to locate contiguous data shards within the storage space segment. A sparse shard index refers to a lightweight index structure that only records the base address (e.g., base addr) and append tail pointer of the contiguous data within the storage space segment, eliminating the need to create a separate index for each shard. Its core principle is to leverage the contiguous storage characteristics of the storage space segment to locate data using a base address plus an offset.

[0084] For example, the system first obtains the target offset of the data to be read (e.g., 130~180), and simultaneously obtains the append write tail pointer of the target replica storage space segment (e.g., the tail pointer of the replica is 180). It then checks whether the target offset is within the storage space segment. By comparing the range of the target offset (130~180) with the valid range of the storage space segment (0~180), it confirms that the target offset falls entirely within the segment. Next, it calls the sparse shard index, and based on the base address of the storage space segment recorded in the index (e.g., base addr=0), calculates the relative address of the target offset, satisfying: target offset (130) - base address (0) = relative offset, i.e., the relative offset is 130. Based on the calculated relative offset, the system directly reads the data from the corresponding position in the storage space segment, completing the read operation. This process utilizes the append write tail pointer to quickly determine the data location, and combined with the base address + offset logic of the sparse shard index, achieves efficient reading of data within the storage space segment, reducing index overhead while ensuring read efficiency through contiguous storage.

[0085] S23. When the target offset is not within the storage space segment of the replica, query the temporary data segment of the replica. If the target offset exists in the temporary data segment, read the data from the temporary data segment.

[0086] It should be noted that the temporary data segment is configured with a dense sharding index, which is used to locate sparsely stored data shards within the temporary data segment. A dense sharding index refers to an index structure where a separate index entry is created for each sparsely stored data shard in the temporary data segment. Each entry records information such as the shard's target offset, relative address, and size. This allows for precise location of individual data shards scattered throughout the temporary segment, compensating for the addressing limitations caused by the discontinuous nature of the temporary data.

[0087] For example, the system first obtains the target offset of the data to be read (e.g., 190~200) and simultaneously obtains the append write tail pointer of the target replica's storage space segment (e.g., the replica's tail pointer is 180). Upon inspection, it finds that the target offset (190~200) exceeds the storage space segment range (0~180). Then, the system queries the dense shard index corresponding to the replica's temporary data segment to check if there is an entry matching the target offset. For example, if the index records an entry with "offset=190, size=10", it matches the target offset. If a matching entry exists in the index, the system locates the storage location corresponding to the target offset within the temporary data segment based on the relative address recorded in the entry (an offset based on the storage space segment's base address). Finally, it reads the data from the corresponding location in the temporary data segment, completing the read operation. This process accurately locates the scattered shards within the temporary segment using the dense shard index, solving the addressing problem of data outside the storage space segment, while also being compatible with the logic of a unified addressing space, ensuring the integrity of the data read.

[0088] Furthermore, after performing step S23, the method further includes: if the target offset does not exist in the temporary data segment, returning a response message indicating that the data does not exist to the requester.

[0089] For example, if the target offset does not fall within the valid address range of the storage space segment, and the dense shard index of the temporary data segment also has no corresponding matching entry, it means that neither storage area contains the target data, and the system will directly return a response message indicating that the data does not exist. This judgment logic clarifies the boundaries of data retrieval, avoids meaningless storage traversal and invalid queries, ensures the accuracy of the system response, reduces redundant overhead, and improves retrieval efficiency.

[0090] In this embodiment of the invention, the data storage area is quickly defined by the append write tail pointer of the storage space segment. Combined with the differentiated positioning capabilities of sparse and dense shard indexes, accurate and efficient data reading is achieved. Continuous data located in the storage space segment leverages the lightweight advantage of the sparse index, using a base address + relative offset for fast retrieval, reducing index overhead and read latency. Sparse data falling into the temporary data segment is precisely mapped by the dense index, preventing data loss. This mechanism is compatible with the architecture characteristics of dual storage areas and balances the read efficiency and data integrity of the storage system through layered read logic. It effectively supports the high-reliability data access requirements in multi-replica append write scenarios and improves the read / write stability of the system in long-tail jitter scenarios.

[0091] Furthermore, to address the issue of missing data fragments on some nodes due to network jitter or other factors, this invention provides a data repair and completion scheme. This scheme retains the simplicity of repairing data from a pure append-only write model and eliminates the full data lookup and comparison operation required for repairing data from a pure random write system. See also Figure 11 , Figure 11 This is a flowchart of data repair in the data processing method provided in the embodiment of the present invention. At this time, the data processing method includes steps S31 to S33.

[0092] S31. Obtain the tail length of the copy in the append write system.

[0093] For example, the definition of the tail length can be referred to in step S12 above, and will not be repeated here. Specifically, the tail length of each replica node (peer) is synchronized through the gossip protocol or other distributed protocols. This length is the greater of the write tail pointer value appended to the storage space segment of each replica and the offset of the last fragment of the temporary data segment, which is used to characterize the farthest logical position of the data written to a single replica.

[0094] S32. Determine the target copy with missing data based on the tail length.

[0095] For example, see Figure 12 , Figure 12 This is a data repair diagram provided by an embodiment of the present invention. A target replica has missing data; its storage space segment has only written data A, and the append write tail pointer is stuck at the end of A, indicating a missing data interval. The dense shard index is not yet associated with the temporary data segment content. During the identification of the target replica, the system needs to summarize the tail lengths of all replicas in the replica group (e.g., replicas 1 and 2) and determine the maximum tail length (i.e., the furthest data offset completed by replicas 1 and 2). The tail length of the target replica is compared with this maximum value. Because the append write tail pointer of the target replica is less than the maximum value, it is determined that it has missing data. The target replica, through a missing data filter, scans backwards from the append write tail pointer of its own storage space segment, and simultaneously, combined with the dense shard index of the temporary data segment, identifies the offset interval corresponding to the missing flag, forming a set of missing data.

[0096] S33. Retrieve the missing data of the target copy from other copies to perform data repair on the target copy.

[0097] For example, the target replica's data repair engine initiates a data retrieval request to the healthy replicas (replicas 1 and 2) within the replica group via Peering (inter-node communication) based on the missing data set, specifying the offset range of the missing data. Upon receiving the request, the healthy replica (e.g., replica 1) locates the corresponding data from its own storage space segment (or temporary data segment) and returns the missing data to the target replica. After receiving the data, the target replica's data repair engine writes it to its own storage space segment, filling the missing range, and synchronously updates the appended tail pointer to the same position as the healthy replica. Once the data writing is complete, the tail length of the target replica is consistent with the replica group, completing the repair, and it can then participate in normal read and write operations.

[0098] In this embodiment of the invention, the tail length is synchronized via the gossip protocol, and the missing replicas and data ranges are quickly located by combining maximum value comparison and local scanning, avoiding the redundant overhead of full data verification. Furthermore, missing data is retrieved using peering communication from healthy replicas, and accurate data transmission is achieved by relying on the indexing capabilities of the storage area. This balances repair efficiency and resource consumption, allowing the system to automatically complete data alignment while tolerating long-tail node jitter, thus improving the reliability and stability of distributed storage.

[0099] Compared to existing technologies, the data processing method disclosed in this invention introduces a temporary data area that supports random writes as a supplement to the append-write data area, thus overcoming the problem that traditional pure append-write systems cannot achieve strong consistency writes when data is inconsistent across multiple nodes. Furthermore, since the temporary data area supports local random writes, the main data of the replica nodes remains within the append-write area. The sparse index size is small and temporary, and in the long run, it will revert to the zero-sparse index overhead advantage of the pure append-write mode. Simultaneously, because the temporary data area supports local random writes, the main data of the replica nodes remains within the append-write area, allowing the reuse of data repair methods from append-write systems. Combined with sparse index scanning, missing data can be quickly constructed. This avoids the inefficiency of a full table index scan for determining missing data in pure random write data.

[0100] See Figure 13 , Figure 13 This is a structural block diagram of a data processing device 100 provided in an embodiment of the present invention. The data processing device 100 includes:

[0101] The data fragment receiving module 11 is used to receive data fragments to be written to the append write system; wherein the append write system includes at least two replicas, and each replica is configured with a storage space segment and a temporary data segment, wherein the storage space segment and the temporary data segment belong to the same address space;

[0102] The expected write offset determination module 12 is used to determine the expected write offset of the data fragment;

[0103] The tail pointer value acquisition module 13 is used to acquire the tail pointer value of the storage space segment of each copy;

[0104] The data writing module 14 is used to write the data fragments into the storage space segment of the replica and update the tail pointer if the tail pointer value is continuous with the expected write offset; and to write the data fragments into the temporary data segment of the replica if the tail pointer value is not continuous with the expected write offset.

[0105] Specifically, the data processing device 100 further includes:

[0106] The data reading module is used to obtain the target offset of the data to be read; when the target offset is located within the storage space segment of the replica, the data is read from the storage space segment; when the target offset is not located within the storage space segment of the replica, the temporary data segment of the replica is queried, and if the target offset exists in the temporary data segment, the data is read from the temporary data segment.

[0107] Specifically, the data processing device 100 further includes:

[0108] The data repair module is used to obtain the tail length of the replica in the append write system; determine the target replica with missing data based on the tail length; and pull the missing data of the target replica from other replicas to repair the data of the target replica.

[0109] It is worth noting that the working process of each module in the data processing device 100 described in the embodiments of the present invention can refer to the working process of the data processing method described in the above embodiments, and will not be repeated here.

[0110] See Figure 14 , Figure 14 This is a structural block diagram of a data processing device 200 provided in an embodiment of the present invention. The data processing device 200 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the various data processing method embodiments described above.

[0111] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the data processing device 200.

[0112] The data processing device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the data processing device 200 and does not constitute a limitation on the data processing device 200. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the data processing device 200 may also include input / output devices, network access devices, buses, etc.

[0113] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the data processing device 200, connecting all parts of the data processing device 200 via various interfaces and lines.

[0114] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the data processing device 200 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0115] If the modules / units integrated in the data processing device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0116] Furthermore, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the data processing method as described in any of the above embodiments.

[0117] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, include: Receive data fragments to be written to the append write system; wherein the append write system includes at least two replicas, and each replica is configured with a storage space segment and a temporary data segment belonging to the same address space; Determine the expected write offset of the data shard and obtain the tail pointer value of the storage space segment of each replica; If the tail pointer value is continuous with the expected write offset, the data is written to the storage space segment of the replica, and the tail pointer is updated. If the tail pointer value is not continuous with the expected write offset, the data is fragmented and written to the temporary data segment of the replica; The step of determining the expected write offset of the data shard includes: Obtain the tail length of all replicas in the append write system; wherein the tail length is the maximum value of the first offset value and the second offset value in the replica, the first offset value is the tail pointer value of the storage space segment, and the second offset value is the tail offset of the temporary data segment; The maximum value among the tail lengths of all the replicas is determined as the expected write offset of the data shard.

2. The data processing method as described in claim 1, characterized in that, The method further includes: Get the target offset of the data to be read; When the target offset is located within the storage space segment of the replica, data is read from the storage space segment; When the target offset is not within the storage space segment of the replica, query the temporary data segment of the replica. If the target offset exists in the temporary data segment, read the data from the temporary data segment.

3. The data processing method according to any one of claims 1 to 2, characterized in that, The storage space segment is configured with a sparse sharding index, and the temporary data segment is configured with a dense sharding index; wherein, the sparse sharding index is used to locate contiguous data shards in the storage space segment, and the dense sharding index is used to locate sparsely stored data shards in the temporary data segment.

4. The data processing method as described in claim 1, characterized in that, The method further includes: The target copy containing missing data is determined based on the tail length; The missing data of the target copy is retrieved from other copies to perform data repair on the target copy.

5. A data processing apparatus, characterized in that, include: A data fragment receiving module is used to receive data fragments to be written to the append write system; wherein, the append write system includes at least two replicas, and each replica is configured with a storage space segment and a temporary data segment, wherein the storage space segment and the temporary data segment belong to the same address space; An expected write offset determination module is used to determine the expected write offset of the data fragment; The tail pointer value acquisition module is used to obtain the tail pointer value of the storage space segment of each copy; The data writing module is configured to write the data fragments into the storage space segment of the replica and update the tail pointer if the tail pointer value is continuous with the expected write offset; and to write the data fragments into the temporary data segment of the replica if the tail pointer value is not continuous with the expected write offset. Specifically, the expected write offset determination module is used for: Obtain the tail length of all replicas in the append write system; wherein the tail length is the maximum value of the first offset value and the second offset value in the replica, the first offset value is the tail pointer value of the storage space segment, and the second offset value is the tail offset of the temporary data segment; The maximum value among the tail lengths of all the replicas is determined as the expected write offset of the data shard.

6. A data processing device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the data processing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the data processing method as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the data processing method as described in any one of claims 1 to 4.

Images