Distributed state consistency processing method based on sequence number arbitration and hierarchical event traceability

Abstract

The application discloses a distributed state consistency processing method and system based on sequence number arbitration and hierarchical event traceability. The method comprises the following steps: receiving a client request and executing a multi-level query, determining a new request when a hit is not found; writing the new request into an original input stream, performing exclusive arbitration based on a partition-level lease by a state arbitration subsystem, assigning a continuous sequence number to a legal event and updating an atomic state register, and processing illegal requests in stages; in response to a state backfill event or sequence number hole detection, reconstructing the state by using a silent replay mode; monitoring storage engine performance and dynamically adjusting reconstruction parameters to realize resource isolation. The application eliminates the multi-copy consistency risk from the architecture level by using the read-write separation arbitration, physical isolation authority log, event-driven backfill and self-adaptive flow control mechanism, and constructs a single trusted hot view, so that the absolute state consistency, high-precision fault recovery and resource isolation high-availability state management are realized.

Description

Technical Field

[0001] This application relates to the field of data consistency assurance technology, and in particular to a distributed state consistency processing method, system, computer-readable storage medium, and electronic device based on sequence number arbitration and hierarchical event tracing. Background Technology

[0002] In a distributed microservice architecture, ensuring state consistency across components (gateways, message queues, and business services) is a core challenge in guaranteeing data accuracy. Current mainstream solutions include database optimistic locking, checkpointing mechanisms in stream processing engines (such as Apache Flink), and traditional distributed locks.

[0003] Defects and shortcomings of existing technology:

[0004] 1. Concurrent write conflicts among multiple components: The lack of a unified arbiter can easily lead to state overwriting or dirty data.

[0005] 2. Coarse-grained fault recovery and data window risk: Mainstream stream processing solutions use a periodic snapshot mechanism. When recovering from a fault, it can only roll back to the most recent snapshot point. State changes during the interval between two snapshots (RPO > 0) may be lost, making it impossible to achieve accurate reconstruction at any point in time.

[0006] 3. State storage coupling and lack of auditing: The state data of the existing solution is usually stored in the engine’s private binary format, forming a “state black box” that is not readable from the outside; and only the success status is recorded, losing the details of rejected illegal requests, which cannot meet the financial-grade end-to-end auditing requirements.

[0007] 4. Difficulty in maintaining state-view coupling and consistency: In existing solutions, state storage is often tightly coupled with the computing engine (such as Flink's state backend), making it difficult for external systems to query the state directly and efficiently, forming a "state black box". Some solutions, in order to solve query performance, have adopted a multi-replica asynchronous replication strategy (such as an independent caching layer), resulting in non-unique data sources. This requires reliance on complex eventual consistency reconciliation mechanisms, which can easily lead to state divergence and data inconsistency in failure scenarios.

[0008] 5. Low reconstruction efficiency and long-tail blocking: Existing event tracing solutions usually store all events together, and a large number of gaps need to be processed during reconstruction; and there is a lack of flow control mechanism, the reconstruction of a single long historical ID may block the entire storage engine and affect real-time services. Summary of the Invention

[0009] To address the aforementioned problems in existing technologies, this application proposes a novel distributed state consistency processing method and system based on sequence number arbitration and hierarchical event sourcing. This invention constructs an atomic state registry as a unique hot view, uses the original input stream as a write-alive log (WAL), and the authoritative event log as a clean reconstruction source, utilizing logical sequence numbers to eliminate clock drift.

[0010] This solution eliminates the independent final state cache and adopts a multi-level query and decoupled recovery mechanism to eradicate the risk of dual-write consistency at the architectural level; it ensures the absolute continuity of sequence numbers through heartbeat physical isolation; it retains extreme recovery capabilities through graded handling of illegal requests; and it achieves highly available, strongly consistent and resource-isolated state management through adaptive batch replay and lease circuit breaker mechanisms.

[0011] The table below visually illustrates the solutions proposed in this plan to address the shortcomings of existing technologies:

[0012]

[0013] To achieve the above objectives, the present invention employs the following technical strategies:

[0014] 1. Read / Write Separation State Arbitration Method Based on WAL:

[0015] The state coordinator is split into a request persistence subsystem (producer) and a state arbitration subsystem (consumer). The two are completely decoupled through a persistent write-alive log (WAL), supporting independent scaling; they naturally support fault replay and idempotent recovery, avoiding the consistency problems caused by direct dual-write in traditional solutions.

[0016] 2. The purity and physical isolation mechanism of authoritative event logs:

[0017] The authoritative log stores only valid business events that have been arbitrated and assigned sequence numbers; heartbeats and illegal events are physically isolated through independent topics or mandatory partition routing based on ACLs. This ensures that sequence numbers in the reconstruction source are strictly continuous and without gaps, greatly simplifying the replay logic and improving reconstruction efficiency.

[0018] 3. Event-driven state recovery mechanism:

[0019] Decoupling query and repair responsibilities. The entry controller only executes read-only queries and returns status codes, while a separate compensation task triggers state reconstruction by listening for query result events or periodically checking for differences. This avoids query path blocking, supports batch aggregation and compensation, and achieves low-latency response and eventual consistency self-healing under high concurrency.

[0020] 4. Adaptive batch replay method based on continuous Seq intervals:

[0021] By leveraging the sequential sequence number characteristic of authoritative logs, reconstruction tasks are divided into deterministic Seq value ranges. The range span and concurrency are dynamically adjusted by monitoring the combined backpressure signals of storage engine P99 latency, reconstruction queue depth, and slow query frequency. This achieves physical resource isolation between reconstruction traffic and real-time services, preventing long-tail reconstructions from blocking the system and ensuring stability under high load.

[0022] 5. Illegal request tiered processing and cold backup recovery mechanism:

[0023] Distinguish between retryable and non-retryable illegal requests. For non-retryable requests, write the original cold backup stream (e.g., error-events topic) with the same structure to the audit log. This ensures the progress of the main consumption chain while retaining the ability to manually extract and re-inject data for repair in extreme scenarios, thus achieving a closed-loop operation and maintenance system.

[0024] 6. Key affinity mapping mechanism based on explicit sharding tags:

[0025] By using explicit shard tags (such as hash tags), all keys associated with the same BusinessID are forced to be mapped to the same physical storage shard. This ensures the feasibility of atomic operations across keys (such as Lua scripts) while optimizing cluster load balancing and avoiding hotspot skew.

[0026] 7. Zonal-level lease management and idempotency guarantee mechanism:

[0027] Fine-grained leases are adopted, corresponding one-to-one with message partitions. After a new node acquires a lease, it automatically skips processed events using sequence number verification (LastSeq comparison). During node failover, no complex state synchronization is required; idempotent replay alone can guarantee eventual consistency, completely eliminating the risk of split-brain.

[0028] Specifically, this application provides the following technical solutions:

[0029] The first aspect of this application provides a distributed state consistency processing method based on sequence number arbitration and hierarchical event sourcing, such as... Figure 1 As shown, the method includes:

[0030] S1, Multi-level Query and Request Admission: Receive client requests and execute multi-level query strategies; prioritize querying the atomic state registry, and return the status if a match is found; if no match is found, query the state index layer, and return the result if a final state is found, or return an asynchronous processing status code and trigger a status recovery event if an intermediate state is found; if no match is found, it is determined to be a new request and allowed to proceed.

[0031] S2. Request Persistence and Arbitration: New requests are written to the original input stream as a pre-write log. The state arbitration subsystem consumes the original input stream and performs exclusive state arbitration based on partition-level leases. Continuous logical sequence numbers are assigned to legitimate business events, the atomic state registry is atomically updated, and the data is written to the authoritative event log. Illegal requests are processed in a tiered manner, and non-retryable illegal requests are written to the audit log while the consumption offset is submitted.

[0032] S3. State restoration and reconstruction: In response to a state restoration event or sequence number hole detection, read historical events from the authoritative event log, and rebuild the atomic state registry state using a silent replay mode. During the reconstruction process, business side effects are prohibited.

[0033] S4, Adaptive Flow Control and Resource Isolation: During the state reconstruction process, the storage engine performance metrics and reconstruction queue depth are monitored, and the reconstruction batch interval span and concurrency are dynamically adjusted to achieve resource isolation between reconstruction traffic and real-time services.

[0034] Furthermore, in the method of this application, after the state restoration event is triggered in step S1, the state reconstruction is performed asynchronously by a compensation task independent of the entry controller, thus decoupling the query path and the state repair path.

[0035] Furthermore, in the method of this application, the partition-level lease mentioned in step S2 corresponds one-to-one with the partition of the original input stream. The lease holder renews the lease periodically. When the lease expires, a new node acquires the lease and continues consumption from the consumption offset submitted last time. Processed events are automatically skipped through sequence number verification.

[0036] Furthermore, in the method of this application, the hierarchical processing of illegal requests described in step S2 includes:

[0037] For retryable illegal requests, the submission of consumption offsets is suspended to trigger message re-delivery. If the retry failure reaches the threshold, the request is transferred to the dead letter queue and asynchronously written to the audit log.

[0038] For illegal requests that cannot be retried, the consumption offset is immediately submitted, and the complete original message is written to the cold backup original stream. The message structure of the cold backup original stream is consistent with the original input stream, and re-injection is supported for data repair.

[0039] Furthermore, in the method of this application, step S2 also includes heartbeat physical isolation: writing heartbeat events into a heartbeat isolation area independent of the authoritative event log, or using an access control list to force heartbeat events to enter a designated partition, ensuring that the sequence numbers of business events in the authoritative event log are strictly continuous and without gaps.

[0040] Furthermore, in the method of this application, the sequence number hole detection in step S3 includes: the state arbitration subsystem compares the sequence number carried by the event with the last sequence number of the corresponding business identifier in the atomic state registry during the consumption process; when the sequence number is detected to be discontinuous, the current consumption is paused and on-demand reconstruction is triggered.

[0041] The silent replay adopts an adaptive batch replay based on continuous sequence number intervals: utilizing the continuous sequence number characteristics of the authoritative event log, the reconstruction task is divided into a defined sequence number value range interval, and the interval span and concurrency are dynamically adjusted according to the joint backpressure signal.

[0042] Furthermore, in the method of this application, the joint backpressure signal includes: the P99 response latency of the target storage engine, the reconstruction task queue depth, and the slow query frequency of the storage engine; when the load is detected to exceed the threshold, an exponential backoff strategy is executed to extend the inter-batch pause time.

[0043] Furthermore, in the method of this application, the atomic state registry described in step S1 adopts a key affinity mapping mechanism based on explicit shard tags to force all keys related to the same business identifier to the same physical storage shard, so as to support cross-key atomic operations.

[0044] Furthermore, in the method of this application, the atomic state registry implements a differentiated lifetime management strategy: a lifetime dependent on heartbeat continuation is set for intermediate states, and a short lifetime is set for final states to save storage space.

[0045] The second aspect of this application provides a distributed state consistency processing system based on sequence number arbitration and hierarchical event sourcing, such as... Figure 7 As shown, the system includes:

[0046] The idempotent entry controller is used to receive client requests, execute multi-level query strategies, and decouple query and repair responsibilities.

[0047] The event bus and raw input stream are used as a log for persisting the full raw request before writing;

[0048] The state coordinator cluster includes a request persistence subsystem and a state arbitration subsystem. The request persistence subsystem is used to append requests to the original input stream, and the state arbitration subsystem is used to consume the original input stream, perform state arbitration, assign sequence numbers, update the atomic state registry, and write to the authoritative event log.

[0049] An atomic state registry is used to store a unique hot state view of the system, supporting high-performance read / write and cross-key atomic operations;

[0050] The authoritative event log is used to store valid business events that have been arbitrated and assigned sequence numbers, serving as the sole source of truth for state reconstruction.

[0051] Audit log storage is used to record full event details to support compliance auditing;

[0052] The lease manager is used to implement partition-level lease management based on a distributed coordination service, ensuring exclusive write access for a single node.

[0053] The system implements the steps of the aforementioned distributed state consistency processing method based on sequence number arbitration and hierarchical event sourcing during runtime.

[0054] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0055] Memory: Used to store computer programs;

[0056] Processor: Used to execute the computer program to implement the steps of the aforementioned distributed state consistency processing method based on sequence number arbitration and hierarchical event sourcing.

[0057] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned distributed state consistency processing method based on sequence number arbitration and hierarchical event sourcing.

[0058] In summary, compared with the prior art, the solution of the present invention has the following advantages:

[0059] (1) Eliminate the risk of multi-replica consistency at the architecture level: Construct a single trusted hot view based on an atomic state registry, and achieve complete separation of query path and state calculation path through multi-level query and decoupling backfill mechanism. This eliminates data discrepancies caused by asynchronous writing of multiple replicas from the root of the architecture, and strong consistency can be guaranteed without complicated dual-write reconciliation.

[0060] (2) Absolute state consistency: By using “raw stream WAL + authoritative log clean source + sequence number arbitration”, we ensure that the sequence number is strictly continuous without gaps and that there is zero data loss.

[0061] (3) High-precision recovery and self-healing: Tiered storage supports RPO=0 reconstruction; event-driven recovery mechanism realizes automatic repair under fault conditions without business awareness; cold backup stream supports extreme manual intervention.

[0062] (4) Audit compliance and cost optimization: authoritative logs ensure reconstruction efficiency, and audit logs (S3+Parquet) ensure full-chain compliance and low cost, and support dead letter replay.

[0063] (5) Resource isolation and high availability: Adaptive flow control based on joint backpressure signal prevents long-tail reconstruction from blocking the system; lease circuit breaker and idempotent mechanism prevent split brain and ensure cluster stability.

[0064] (6) Flexible scaling: The read-write separation architecture and dynamic lease mechanism enable the production and consumption subsystems to scale up and down independently according to the load. Attached Figure Description

[0065] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of the invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0066] Figure 1 This is a flowchart illustrating the overall implementation of the distributed state consistency processing method based on sequence number arbitration and hierarchical event sourcing of the present invention.

[0067] Figure 2 This is a flowchart of the read / write separation state arbitration process based on WAL in the present invention.

[0068] Figure 3 This is a flowchart of the multi-level query and adaptive completion process in the present invention.

[0069] Figure 4 This is a detailed internal processing logic diagram of the state arbitration subsystem in the present invention.

[0070] Figure 5 This is a sequence diagram of event-driven multi-level query and automatic completion in the scheme of this invention.

[0071] Figure 6 This is the logic diagram of adaptive batch replay based on continuous Seq intervals in the present invention.

[0072] Figure 7 This is a structural diagram of the distributed state consistency processing system based on sequence number arbitration and hierarchical event sourcing according to the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0074] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0075] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0076] Example: A distributed state consistency processing method and system based on sequence number arbitration and hierarchical event sourcing

[0077] (I) Overall Architecture and Core Component Definition

[0078] This system adopts a dual-channel architecture that separates business flow and state flow. To avoid confusion, the core storage components and their responsibilities are defined first:

[0079] Raw Input Stream: The system's unique Write-Ahead Log (WAL) ensures no requests are lost. It persists all raw requests (including business events, duplicates, invalid requests, and heartbeats). A typical implementation is Kafka Topic.

[0080] Authoritative Event Log: The sole source of truth for state reconstruction, with strictly consecutive sequence numbers. Only business events (with sequence numbers) that have been arbitrated and verified are persisted. A typical implementation is a standalone Kafka Topic.

[0081] Audit Log: A compliance audit storage system that supports manual replay. It records full event details (including rejection reasons, error stack traces, and dead letters). A typical implementation is S3 + Parquet.

[0082] Cold Backup Stream: Optional component. Its structure is identical to the original input stream (e.g., a separate error-events topic), specifically designed to store the original messages of illegal requests deemed "non-retryable." Separated from the audit log, it supports direct re-injection into the original input stream for replay after repair.

[0083] State Index Layer: A persistent complementary layer to the atomic state registry, supporting L2 queries. It stores the latest state snapshot. A typical implementation is Elasticsearch.

[0084] Atomic State Registry: A unique hot state view of the system, enabling high-performance read and write operations. It stores hot data at the memory level and implements differentiated TTL management. A typical implementation is Redis Cluster.

[0085] Based on the above definitions, this system includes the following core components:

[0086] 1. Idempotent Entry Controller:

[0087] 1.1 Responsibilities: Stateless component. Receives client requests and generates a globally unique business identifier (BusinessID).

[0088] 1.2 Multi-level query strategy (read-only operation):

[0089] (1) L1 query: Directly query the atomic state registry. If a match is found, return the state.

[0090] (2) L2 query: If L1 is not found (Key does not exist / TTL expired), query the status index layer.

[0091] If a final state is hit: return the result directly;

[0092] If an intermediate state is hit: a specific status code (such as 202 Accepted) is returned, and a status recovery event (asynchronous notification or marking difference) is triggered, which is handled by an independent compensation task.

[0093] (3) L3 decision: If neither L1 nor L2 is hit, it is determined to be a new request and allowed to pass through the event bus.

[0094] 1.3 Core Features: Decoupling query and repair responsibilities, the entry controller does not perform any write operations or synchronous recovery, avoiding blocking the query path.

[0095] 2. Event Bus & Raw Input Stream:

[0096] 2.1 Responsibility: The system's Write Log (WAL).

[0097] 2.2 Content: Receive all raw requests coming in from upstream.

[0098] 2.3 Typical Implementation: Apache Kafka Topic.

[0099] 3. State Coordinator Cluster:

[0100] 3.1 Responsibility: The sole writer of state changes. A read-write separation architecture is adopted, split into two logical subsystems (which can be deployed on the same node or independently):

[0101] (1) Request persistence subsystem (producer): It is only responsible for appending the requests from the ingress controller to the raw input stream.

[0102] (2) State Arbitration Subsystem (Consumer): Responsible for consuming requests from the original input stream, performing state machine verification, assigning sequence numbers, updating state storage, and writing valid events to the authoritative event log.

[0103] 3.2 Stateless Architecture Design (Key Features):

[0104] Design principle: The state coordinator node adopts a stateless design. The node itself does not persist any business data or critical state information.

[0105] External state: All business states are persisted in the atomic state registry (Redis) and the authoritative event log (Kafka); all consumption progress (Offset) is persisted in the message queue; and all lock and lease information is persisted in the distributed coordination service (Etcd / ZooKeeper).

[0106] Beneficial effects: Based on this design, the failure or restart of any coordinator node can be seamlessly taken over by simply competing for a lease and reloading the context from external storage, without the need for complex state data migration or synchronization, thereby achieving elastic scaling and high availability of the system.

[0107] 3.3 Lease Manager:

[0108] Definition: A partition-level lease mechanism implemented based on distributed coordination services (such as Etcd / ZooKeeper).

[0109] Granularity: A lease corresponds one-to-one with a partition of the original input stream.

[0110] Mechanism: A coordinator node can dynamically hold multiple partition leases, with the number of leases adaptively adjusted based on node load. Nodes need to renew leases periodically; if a lease expires, consumption of that partition is suspended, and a new node will acquire a lease and continue consumption from the previously submitted offset.

[0111] Idempotency guarantee: After a new node acquires a lease, it continues consumption from the Last Committed Offset. It uses a sequence number verification mechanism (comparing LastSeq) to automatically skip processed events, ensuring eventual consistency during lease transfer.

[0112] 4. Atomic State Registry:

[0113] 4.1 Definition: The only reliable thermal state view of the entire system.

[0114] 4.2 Key Affinity Mapping Mechanism Based on Explicit Fragmentation Tags:

[0115] Physical Key format: Namespace:{ShardTag}:BusinessID.

[0116] {ShardTag}: Explicit shard tag (can be a hash fragment of BusinessID, logical shard ID, or consistent hash slot).

[0117] Function: Ensure that all keys related to the same business identifier are mapped to the same physical storage shard, and support atomic operations across keys.

[0118] 4.3 Differentiated TTL Strategy: Set different time-to-live (TTL) values ​​based on business state (intermediate state / final state). Intermediate states rely on heartbeat renewal to maintain activity; final states allow rapid expiration to save storage, and can be recovered through L2 / L3 mechanisms after expiration.

[0119] 5. Layered Event Storage:

[0120] 5.1 Authoritative Event Log:

[0121] Source: Written by the state arbitration subsystem after a successful atomic state update.

[0122] Content: Only valid business events that have been verified and assigned a logical sequence number (Seq) are persisted.

[0123] Heartbeat physical isolation strategy:

[0124] (1) Strategy A (Recommended): Use an independent Topic (such as heartbeat-events) that is physically separate from the original input stream to store the heartbeat signal.

[0125] (2) Strategy B (Strong Constraint): If it must be in the same cluster, use access control list (ACL) based or fixed partition routing to force heartbeat events to only enter the specified partition (such as Partition 0), while business events cannot enter the partition, ensuring that the state arbitration subsystem can never read the heartbeat when consuming the business partition.

[0126] Effect: Ensures that the Seq of business events in the authoritative log strictly corresponds to the storage order, with no gaps.

[0127] Typical implementation: standalone Kafka Topic.

[0128] 5.2 Audit Log Storage:

[0129] Content: Records all original requests and processing results (including rejection reasons and dead letter queue).

[0130] Typical implementation: Object storage (S3 / OSS) + columnar file (Parquet).

[0131] 5.3 State Index Layer:

[0132] Content: A structured snapshot containing the complete business status.

[0133] Typical implementation: Elasticsearch / OpenSearch.

[0134] (II) Core Mechanism of this Plan

[0135] Mechanism 1: Read-Write Separation State Arbitration Method Based on WAL

[0136] Adopting a "store first, modify later, read-write decoupled" model:

[0137] 1. Phase 1: Request Persistence (Producer)

[0138] The request persistence subsystem receives the request, appends it to the original input stream, and waits for an ACK. This step does not verify the business logic.

[0139] 2. Phase Two: Status Arbitration and Application (Consumer)

[0140] 2.1 The state arbitration subsystem pulls requests from the original input stream.

[0141] 2.2 Lease self-check: Checks whether the current node holds a valid lease for the partition.

[0142] 2.3 Execute the atomic script:

[0143] (1) For abnormal requests encountered during consumption, the system adopts a hierarchical processing and dual-path archiving strategy to ensure high availability of the main link and recovery in extreme scenarios.

[0144] (2) Retryable Errors

[0145] Typical scenarios: transient failures such as temporary state conflicts, downstream service timeouts, and network jitter.

[0146] Processing flow:

[0147] (a) Pause Offset Commit: The coordinator does not commit the current message offset, but uses the automatic rebalancing or manual retry mechanism of Kafka consumer groups to trigger the in-place re-delivery of the message.

[0148] (b) Dead letter transfer: If the message fails to retry N times (with a configurable threshold), it is identified as a "poison pill message". The system will stop retrying and forward the message to a standard dead letter queue (Standard DLQ Topic, such as dlq-events).

[0149] Note: Messages transferred to the dead letter queue usually carry additional error context metadata for use by subsequent automated repair scripts or manual analysis.

[0150] (c) Audit Retention: Regardless of whether the retry is successful or not, the final failure details (including the complete error stack, number of retries, and timestamp) are asynchronously written to the audit log storage.

[0151] (d) Progress advance: After the message is transferred to the dead letter queue, the coordinator submits the offset to ensure that the main consumption stream is not blocked.

[0152] (3) Non-Retryable Errors

[0153] Typical scenarios include deterministic errors such as message format parsing errors, signature verification failures, and serious violations of business rules (such as illegal state machine transitions).

[0154] Processing flow:

[0155] (a) Rejecting state change: The coordinator directly refuses to perform any state update operation and does not assign a sequence number (Seq).

[0156] (b) Immediate Offset Commit: To avoid a single bad message causing the entire partition's consumption to stall (Head-of-Line Blocking), the coordinator immediately commits the message's offset, skipping the main processing logic and ensuring the smooth operation of the main link.

[0157] (c) Dual-path archiving:

[0158] Path A (Audit Compliance): The reasons for rejection, the original message summary, and the basis for the decision are written into the audit log for storage, and used for daily problem investigation and compliance audit.

[0159] Path B (Extreme Recovery - Core Innovation): Write the complete raw payload into the cold backup stream (such as the cold-backup-events topic) without loss.

[0160] Key features: The message structure of the cold backup raw stream is completely identical to that of the raw input stream (raw-events), and does not contain any error wrapper metadata.

[0161] Core value: When business rules are corrected or misjudgments are discovered, operations and maintenance personnel can directly re-inject the messages in the stream into the original input stream for replay without complex data cleaning or unpacking, achieving lossless data repair in extreme scenarios.

[0162] 2.4 Legitimate Business Events:

[0163] Calculate NewSeq = LastSeq + 1.

[0164] Atomic update of State and NewSeq in the registry.

[0165] 2.5 Cardiac bypass treatment:

[0166] This solution prioritizes a physical isolation strategy for heartbeats: heartbeat events are written to a heartbeat topic (such as heartbeat-events) that is independent of the original input stream, and consumed by an independent heartbeat keep-alive component that only updates LastActiveTime and resets TTL. The state arbitration subsystem does not consume heartbeat data at all, thereby ensuring that there are no non-business events in the authoritative event log and that the sequence numbers are strictly continuous.

[0167] 2.6 Write to authoritative event log: Only after a legitimate business event is successfully processed will a valid event with NewSeq be appended to the authoritative event log.

[0168] 3. Phase Three: Committing Offset and Asynchronous / Synchronous Operations

[0169] 3.1 Submission conditions: The consumption offset is submitted to the original input stream only after the authoritative event log is successfully written and the atomic state registry is successfully updated (for illegal requests that cannot be retried, only the audit log / cold backup stream needs to be successfully written).

[0170] 3.2 Asynchronous Weak Consistency: Updates to the Elasticsearch (ES) state index layer are asynchronous operations, allowing for short delays and not blocking offset commits. If an ES update fails, a background compensation task will eventually fix the issue.

[0171] Mechanism 2: Event-driven multi-level query and automatic completion

[0172] The L1→L2→L3 system is constructed, adopting a dual mode of event triggering + lightweight inspection.

[0173] 1. L1 / L2 queries and event triggering:

[0174] 1.1 When the L2 entry controller queries and finds an intermediate state, it returns a 202 Accepted response and triggers a state recovery event.

[0175] 1.2 Asynchronous State Backfill: An independent compensation task listens for backfill events (or periodically checks for differences), calls the on-demand reconstruction sub-process, retrieves historical events for that ID from the authoritative event log, and rebuilds the registry state. This mechanism decouples query and repair responsibilities, avoiding query path blocking.

[0176] 2. Lightweight inspection and on-demand reconstruction:

[0177] During the consumption process, the state arbitration subsystem compares the expected Seq carried by each legitimate event with the LastSeq of that ID in the atomic state registry:

[0178] (1) If event.Seq = registry.LastSeq + 1: proceed normally;

[0179] (2) If event.Seq > registry.LastSeq + 1: it is determined to be a sequence number gap, the current consumption thread is paused, the on-demand reconstruction of the ID is immediately triggered, the missing interval (registry.LastSeq+1) ~ (event.Seq-1) of the authoritative log is pulled from the log and replayed, and consumption is resumed after completion;

[0180] (3) If event.Seq <= registry.LastSeq: it is determined to be a duplicate event and is skipped idempotently.

[0181] 3. Partition-level serial number hole detection and batch repair:

[0182] 3.1 Detection and Judgment: When consuming messages, the coordinator node discovers a discrepancy between the current message's Seq and the registry's LastSeq.

[0183] 3.2 If the partition lease has just been acquired by this node, and a large number of different BusinessIDs with discontinuous sequence numbers are detected during the initial consumption phase (indicating that hot state data is missing on a large scale due to faults or TTL expiration), it is determined to be a partition-level state lag. At this time, in order to avoid triggering massive concurrent single-point reconstruction requests that could overload the system, the system will suspend the regular consumption stream and instead execute a controlled partition batch recovery process: actively scan the list of active IDs for the partition, pull the missing state from the authoritative event log in batches and write it back to the registry, and resume regular consumption after the hot data has been restored to a safe level.

[0184] 3.3 Execution: Pause consumption in this partition, retrieve the latest status of all active IDs in this partition from the authoritative event log, write the data back to the registry in batches, and resume consumption after completion. This process strictly follows the adaptive flow control strategy of Mechanism 3.

[0185] Mechanism 3: Silent Replay and Adaptive Batch Replay

[0186] 1. Silent replay mode: Only update the state during reconstruction, and prohibit side effects.

[0187] 2. Adaptive batch replay based on continuous Seq intervals:

[0188] 2.1 Deterministic Segmentation Basis: Utilizing the continuous sequence number characteristic of authoritative event logs, the division of reconstruction tasks is based on a clear Seq value range (such as Seq 1-1000), rather than fuzzy segmentation based on timestamps or offsets, ensuring that the range of each reconstruction is precisely known.

[0189] 2.2 Joint backpressure monitoring mechanism:

[0190] (1) Monitoring dimensions include: the P99 response latency of the target storage engine, the depth of the reconstruction task queue, and the frequency of occurrence of slow query logs of the storage engine.

[0191] (2) When the depth of the ID queue to be rebuilt exceeds the threshold, or when the frequency of slow query logs in Redis Cluster is abnormal, it is considered a high load signal.

[0192] 2.3 Adaptive Adjustment: Based on the aforementioned combined backpressure signal, dynamically adjust the Seq interval span and reconstruction concurrency for each batch. When the load is low, expand the interval (e.g., 2000 records / batch) and increase concurrency; when the load is high, shrink the interval (e.g., 500 records / batch) and decrease concurrency.

[0193] 2.4 Exponential backoff: If the load is detected to exceed the threshold, the pause time is automatically extended between batches to achieve adaptive matching and physical isolation between reconstruction granularity and system resources.

[0194] Mechanism 4: TTL-based lifecycle management

[0195] 1. Zero-scan heartbeat renewal: Utilizes the storage engine's native TTL. The heartbeat event resets the key's TTL. The key automatically expires in the event of a failure.

[0196] 2. Differentiated TTL strategy: intermediate states rely on heartbeats to maintain activity, while final states allow for rapid expiration to save storage.

[0197] Example description:

[0198] Raw input stream: Kafka Topic raw-events (including all requests).

[0199] Authoritative event log: Kafka Topic auth-events (only valid business events).

[0200] Heartbeat physical isolation: Heartbeat events are written to a separate heartbeat-events topic to ensure that there are no heartbeats in auth-events.

[0201] Cold backup raw stream: Kafka Topic error-events, with a structure completely identical to raw-events, is used to store illegal requests that cannot be retried.

[0202] Lease Management: Etcd implements partition-level leases (64 partitions correspond to 64 leases). Nodes compete for leases and dynamically hold multiple leases. When a lease expires, a new node continues consumption from the Last Committed Offset, automatically skipping processed events based on a sequence number verification mechanism (LastSeq comparison) to ensure consistency.

[0203] Workflow:

[0204] 1. Normal flow: Request → raw-events (ACK) → Coordinator consumes → Verify / update Redis → Write auth-events → Commit raw-events Offset → Asynchronously update ES.

[0205] 2. Exception handling:

[0206] 2.1 Illegal requests can be retried (e.g., temporary state conflicts, downstream service timeouts):

[0207] Instead of committing the offset, it relies on the Kafka mechanism to automatically resubmit the offset.

[0208] If the retries fail after N attempts, the message is forwarded to a dead letter queue (such as a separate dlq-events topic) for subsequent automatic repair or manual replay.

[0209] At the same time, the detailed context of the failure (including the error stack) is asynchronously written to the audit log for retention.

[0210] 2.2 Illegal events that cannot be retried (such as format errors or serious violations of business rules):

[0211] Refuse to update the state and do not assign a Seq.

[0212] Submit the offset to advance the main consumption progress and avoid blocking.

[0213] Write the complete original message and the reason for rejection to the audit log.

[0214] Furthermore, this system writes the complete original message into the cold backup original stream (an independent Topic with the same structure as the original input stream) as a "secondary dead letter," supporting manual extraction and re-injection in extreme scenarios.

[0215] 3. Fault Recovery:

[0216] 3.1 Redis crashed, Pay_001 was lost.

[0217] 3.2 User retry: The entry point checks Redis for empty data, checks Elasticsearch for PROCESSING, returns 202 Accepted, and triggers a backup event.

[0218] 3.3 The compensation task pulls all valid events (Seq=1, 2...) from auth-events for Pay_001.

[0219] 3.4 Silent Replay: Execute in batches according to Seq intervals (monitor P99 latency, queue depth and slow query logs, and dynamically adjust batch size), without sending any goods or SMS messages, only restoring the Redis state.

[0220] 4. Extreme scenario recovery (human intervention):

[0221] 4.1 An illegal request was rejected due to a misjudgment by the business rules. This was written to the audit log and error-events, and the offset has been committed.

[0222] 4.2 After 3 days, the business side confirms that the request is legitimate. The operations and maintenance personnel extract the original request message from error-events.

[0223] 4.3 Re-inject the message into raw-events.

[0224] 4.4 When the status arbitration subsystem consumes data, if the ID already has a final state, it is determined to be in conflict by comparing the serial number; however, under special operation and maintenance permissions, the state can be forcibly overwritten or a specific repair process can be triggered to achieve data repair in extreme scenarios.

[0225] Figure 2 This is a flowchart of the read / write separation state arbitration process based on WAL in the present invention.

[0226] The diagram illustrates the core process of state writing and arbitration. The specific steps are shown in the table below:

[0227]

[0228] Figure 3 The flowchart of the multi-level query and adaptive completion process in the present invention is shown in the table below:

[0229]

[0230] Figure 4 The detailed internal processing logic diagram of the state arbitration subsystem in this invention is shown in the table below, illustrating the steps:

[0231]

[0232] Figure 5 This is a sequence diagram of event-driven multi-level query and automatic completion in the scheme of this invention.

[0233] Figure 6 This is the logic diagram of adaptive batch replay based on continuous Seq intervals in the present invention.

[0234] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of systems, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function.

[0235] Embodiments of this application also disclose an electronic device, including: a processor, a communication interface, a memory for storing a processor-executable computer program, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor executes the executable computer program to implement the steps of the aforementioned distributed state consistency processing method based on sequence number arbitration and hierarchical event sourcing.

[0236] Furthermore, this application also discloses a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the various steps of the distributed state consistency processing method based on serial number arbitration and hierarchical event sourcing disclosed in this application.

[0237] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the distributed state consistency processing method based on sequence number arbitration and hierarchical event sourcing disclosed in this application. When this computer program is executed by a processing system, it can achieve the functions defined in the embodiments of this application.

[0238] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0239] Those skilled in the art should also understand that they can modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features without departing from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A distributed state consistency processing method based on sequence number arbitration and hierarchical event sourcing, characterized in that, The method includes: S1, Multi-level Query and Request Admission: Receive client requests and execute multi-level query strategies; prioritize querying the atomic state registry, and return the status if a match is found; if no match is found, query the state index layer, and return the result if a final state is found, or return an asynchronous processing status code and trigger a status recovery event if an intermediate state is found; if no match is found, it is determined to be a new request and allowed to proceed. S2. Request Persistence and Arbitration: New requests are written to the original input stream as a pre-write log. The state arbitration subsystem consumes the original input stream and performs exclusive state arbitration based on partition-level leases. Continuous logical sequence numbers are assigned to legitimate business events, the atomic state registry is atomically updated, and the data is written to the authoritative event log. Illegal requests are processed in a tiered manner, and non-retryable illegal requests are written to the audit log while the consumption offset is submitted. S3. State restoration and reconstruction: In response to a state restoration event or sequence number hole detection, read historical events from the authoritative event log, and rebuild the atomic state registry state using a silent replay mode. During the reconstruction process, business side effects are prohibited. S4, Adaptive Flow Control and Resource Isolation: During the state reconstruction process, the storage engine performance metrics and reconstruction queue depth are monitored, and the reconstruction batch interval span and concurrency are dynamically adjusted to achieve resource isolation between reconstruction traffic and real-time services.

2. The method according to claim 1, characterized in that, After the state restoration event is triggered as described in step S1, the state reconstruction is performed asynchronously by a compensation task independent of the entry controller, thus decoupling the query path from the state repair path.

3. The method according to claim 1, characterized in that, The partition-level leases mentioned in step S2 correspond one-to-one with the partitions of the original input stream. The lease holder renews the lease periodically. When the lease expires, a new node acquires the lease and continues consumption from the consumption offset submitted last time. Processed events are automatically skipped through sequence number verification.

4. The method according to claim 1, characterized in that, The step S2, which involves performing tiered processing on illegal requests, includes: For retryable illegal requests, the submission of consumption offsets is suspended to trigger message re-delivery. If the retry failure reaches the threshold, the request is transferred to the dead letter queue and asynchronously written to the audit log. For illegal requests that cannot be retried, the consumption offset is immediately submitted, and the complete original message is written to the cold backup original stream. The message structure of the cold backup original stream is consistent with the original input stream, and re-injection is supported for data repair.

5. The method according to claim 1, characterized in that, Step S2 also includes heartbeat physical isolation: writing heartbeat events to a heartbeat isolation area independent of the authoritative event log, or using an access control list to force heartbeat events to enter a designated partition, ensuring that the sequence numbers of business events in the authoritative event log are strictly continuous and without gaps.

6. The method according to claim 1, characterized in that, The sequence number hole detection in step S3 includes: during the consumption process, the state arbitration subsystem compares the sequence number carried by the event with the last sequence number of the corresponding business identifier in the atomic state registry. When a discontinuous sequence number is detected, the current consumption is paused and on-demand reconstruction is triggered. The silent replay adopts an adaptive batch replay based on continuous sequence number intervals: utilizing the continuous sequence number characteristics of the authoritative event log, the reconstruction task is divided into a defined sequence number value range interval, and the interval span and concurrency are dynamically adjusted according to the joint backpressure signal.

7. The method according to claim 6, characterized in that, The combined backpressure signals include: the target storage engine's P99 response latency, the rebuild task queue depth, and the storage engine's slow query frequency; when the load exceeds the threshold, an exponential backoff strategy is executed to extend the inter-batch pause time.

8. The method according to claim 1, characterized in that, The atomic state registry described in step S1 adopts a key affinity mapping mechanism based on explicit shard tags to force all keys related to the same business identifier to be mapped to the same physical storage shard in order to support cross-key atomic operations.

9. The method according to claim 1, characterized in that, The atomic state registry implements a differentiated lifetime management strategy: it sets a lifetime for intermediate states that depends on heartbeat renewal, and sets a short lifetime for final states to save storage space.

10. A distributed state consistency processing system based on sequence number arbitration and hierarchical event sourcing, characterized in that, The system implements the distributed state consistency processing method based on sequence number arbitration and hierarchical event sourcing as described in any one of claims 1-9 during operation, and the system includes: The idempotent entry controller is used to receive client requests, execute multi-level query strategies, and decouple query and repair responsibilities. The event bus and raw input stream are used as a log for persisting the full raw request before writing; The state coordinator cluster includes a request persistence subsystem and a state arbitration subsystem. The request persistence subsystem is used to append requests to the original input stream, and the state arbitration subsystem is used to consume the original input stream, perform state arbitration, assign sequence numbers, update the atomic state registry, and write to the authoritative event log. An atomic state registry is used to store a unique hot state view of the system, supporting high-performance read / write and cross-key atomic operations; The authoritative event log is used to store valid business events that have been arbitrated and assigned sequence numbers, serving as the sole source of truth for state reconstruction. Audit log storage is used to record full event details to support compliance auditing; The lease manager is used to implement partition-level lease management based on a distributed coordination service, ensuring exclusive write access for a single node.

Images