A distributed idempotent state updating and expired write shielding method and system based on epoch lock interval and a storage medium

Abstract

The application discloses a system, method and storage medium for distributed idempotent state update and expired write shielding. The system collects and processes node operation indexes to generate a node operation feature vector E, and a global coordination center assigns a strictly increasing and non-overlapping epoch lock interval [Epoch_low, Epoch_high] to the processing node according to the feature vector. The processing node establishes a placeholder record in a memory type storage with an idempotent key and selects an epoch lock Epoch_lock within the interval; after the placeholder is successful, a row-level exclusive lock is applied to the target record in the same transaction in the business database, and a double predicate condition update is performed in the order of interval switching and then interval writing; when the upper interval boundary field epoch_hi_field is less than Epoch_low, the upper interval boundary is updated to Epoch_high; when (epoch_field is less than Epoch_lock) and (Epoch_lock is less than or equal to epoch_hi_field), the business state is written and the epoch_field is updated to Epoch_lock, thereby rejecting expired writes. Placeholder timeout triggers takeover competition, and a script-based atomic conditional update writes takeover_low to determine a unique takeoverer to continue writing. Optionally, a transaction log drilling index module follows to parse the transaction log to establish a drilling index and maintain a watermark_lsn consistency boundary, to support playback, verification or compensation.

Description

[0001] [Technical Field] This invention relates to the fields of distributed consistency control, idempotent state update, database transaction concurrency control, and transaction log indexing. In particular, it relates to a system, method, and storage medium for epoch lock interval allocation based on node running feature vectors, script-based atomic takeover of memory key-value storage, interception of expired writes for double predicate condition updates within business database transactions, and drill-down indexing and consistency boundary output of transaction logs.

[0002] [Background Technology] Asynchronous distributed systems often employ at least-once delivery semantics. Network partitions, latency jitter, and node failures can lead to duplicate event processing and late commits. Late write requests may overwrite new writes, causing expired writes and lost updates.

[0003] Existing solutions often rely on application-layer deduplication, distributed lock leases, or single token / version number verification. Such solutions are prone to race conditions in scenarios such as "lease expires but the original node commits late", "concurrent takeover competition", and "interleaving of interval switching and write timing", leading to double writes or overwriting of expired writes.

[0004] Meanwhile, drill-down queries in transaction logs typically lack verifiable consistency boundaries and index structures, resulting in high drill-down costs and difficulty in expressing result consistency boundaries, making it difficult to meet the stability requirements of auditing and tracing scenarios.

[0005]

Invention Content

Technical Problems to be Solved by the Invention

[0006] [Technical Solution] The technical solution of this invention, as described in claims 1-10, mainly includes: A) collecting node running status indicators and generating a node running feature vector E; B) allocating strictly increasing and non-overlapping epoch lock intervals according to E and performing atomic placeholders on idempotent keys in memory key-value storage to establish placeholder records, wherein the placeholder record contains a state field state and the state is initially PROCESSING, and ttl1 and ttl2 are set; C) after successful placeholder, applying a row-level exclusive lock to the target record within the same transaction in the business database and performing double predicate condition updates in the order of "first interval switching, then writing within the interval" to intercept expired writes; D) triggering a placeholder timeout. A new interval application is issued and the takeover_low is updated using a script-based atomic condition to determine the unique takeover and complete the database write; after detecting that the database transaction has been successfully committed, the state of the placeholder record is atomically migrated to COMMITTED and TTL2 is set to suppress the duplicate processing of the same idempotent key; E) The transaction log drill-down index module builds a drill-down index by following and parsing the transaction log according to LSN and maintains the watermark_lsn consistency boundary output; F) The drill-down query returns the sorted change sequence and watermark_lsn by entity_id and returns the original log fragment based on log_offset when needed.

[0007] [Beneficial Effects] The present invention has at least the following effects: (1) It rejects expired writes by updating the database side with double predicate conditions, thereby reducing the risk of state disorder caused by race conditions; (2) It ensures the uniqueness of takeover by using the memory key-value storage script-based atomic takeover decision and reduces the overhead of double writes and retry caused by concurrent takeovers; (3) It reduces the frequency of coordination center interactions by allocating interval-based epoch locks; (4) It provides a stable and verifiable expression of drill-down results and reduces the cost of drill-down queries by using the drill-down index and the consistent boundary output of watermark_lsn.

[0008] [Attached Image Description] Figure 1 This is a schematic diagram of the overall system architecture.

[0009] Figure 2 This is a schematic diagram of a double-predicate conditional update process within a transaction, which involves first switching between ranges and then writing within those ranges.

[0010] Figure 3 This is a timing diagram illustrating script-based atomic takeover and late submission conflict interception.

[0011] Figure 4 This is a timing diagram of a script-based atomic takeover competition decision.

[0012]

Detailed Implementation

Explanation of Figure Symbols

[0013]

Variable, Function, and Symbol Explanation

[0014] clamp(x,a,b) represents the clamp function with upper and lower bounds; round(y) represents rounding y to the nearest integer; exp(z) represents the exponential function with the natural constant e as the base; pow(x,g) represents x raised to the power of g; sum(.) represents summation; min(x,y) represents taking the smaller value; ceil(x) represents the rounding function.

[0015]

Parameter Acquisition Method Explanation

[0016] [Example 1: Generation of Node Running Feature Vector E] For example... Figure 1 As shown, the node metric collector 30 collects metrics including CPU utilization u_cpu, memory pressure m_pressure, storage IO load iops_load, network round-trip time jitter j_rtt, packet loss rate p_loss, garbage collection pause time gc_pause, transaction rollback count rollback_count, total number of transactions tx_count, coordination center request latency t_coord, and coordination center request failure rate p_fail to form an metric set M; the feature vector generator 40 normalizes at least two different types of metrics to [0,1] and performs sliding statistics within the statistical window W to obtain the rollback rate r and Epoch, where r can be calculated as r=rollback_count / tx_count and tx_count>=1.

[0017] In one implementation, EWMA is performed on the rollback rate time series to obtain a smoothed rollback rate: r = r_t * alpha + (1 - alpha) * r_prev and alpha = 1 - exp(-delta_t / tau_decay).

[0018] An example of I_cap is: calculate c=sum(w_i*m_i) for the normalized index vector m_i, and let I_cap=1 / (1+c) or discretize it according to the mapping table, where w_i is a preset constant or determined by regression based on historical monitoring data.

[0019] [Example 2: Epoch Interval Allocation and Placeholder Record Establishment] For example... Figure 1 As shown, the distributed coordination center 50 maintains a globally increasing counter C. When receiving an interval request, it returns [Epoch_low=C+1,Epoch_high=C+step(E)] and updates C=Epoch_high to ensure that the intervals are strictly increasing and do not overlap.

[0020] In one embodiment, step(E) = round(step_min+(step_max-step_min)*(1-health_score)), where health_score is clamped to [0,1] and is obtained from the maximum value or weighted sum of the ratios of at least two metrics in the feature vector E to the threshold; in one embodiment, the larger the health_score, the higher the node load or the worse the health, so that step(E) decreases as health_score increases to shorten the span of the allocated epoch lock interval.

[0021] The status update execution unit 60 performs an atomic conditional write on the idempotent_key in the in-memory key-value database 70 to create a placeholder record, writing state=PROCESSING, Epoch_low, Epoch_high, Epoch_lock, takeover_low, create_ts and setting ttl1 and ttl2, where Epoch_lock satisfies Epoch_low <= Epoch_lock <= Epoch_high and is locally incrementally selected within the interval, and Epoch_high = Epoch_low + step(E) - 1; after detecting that the business database transaction is committed successfully, the state of the placeholder record is atomically migrated to COMMITTED and ttl2 is set to suppress duplicate processing of the same idempotent_key.

[0022]

Embodiment 3: Dual predicate interception in the business database (order within a transaction)

[0023] The SQL examples for illustration are as follows: Interval switching: UPDATE resource_entity_table SET epoch_hi_field=? WHERE entity_id=? AND epoch_hi_field<?;; Writing within the interval: UPDATE resource_entity_table SET state_value=LEAST(state_value+?, state_max), epoch_field=? WHERE entity_id=? AND epoch_field<? AND?<=epoch_hi_field;.

[0024] To further ensure idempotency under extreme failures, within the same database transaction where the dual-predicate update is executed, a deduplication table based on the idempotent_key can be queried or maintained, or the last_idempotent_key field of the target record can be verified to ensure that the same request is not executed repeatedly.

[0025]

Example 4: Scripted atomic takeover (including priority competition)

[0026] In the takeover competition, further compare takeover_priority, and when takeover_priority is the same, the smaller Epoch_low2 wins to determine the unique takeover者, and the node that returns success is used as the unique takeover者 to continue executing the database transaction write described in Example 3.

[0027]

Example 5: Lock-free snapshot read with early filtering

[0028]

Example 6: Resource Consumption Analysis (Exemplary)

[0029] [Example 7: Drill-down Index in Transaction Log (CDC, Index and Watermark)] For example... Figure 1 As shown, the transaction log drill-down index module 90 reads the transaction log in LSN order and parses it to obtain commit_lsn, tx_id, row_change, and log_offset. After detecting the transaction commit marker, it outputs a mirror event containing commit_lsn. The log mirror storage 100 persists the mirror event and builds a drill-down index with (entity_id, Epoch_lock, commit_lsn) as the key and containing log_offset. It also deduplicates the data using (commit_lsn, tx_id) as the deduplication key and maintains the watermark_lsn corresponding to each entity_id to indicate the maximum commit_lsn of the consecutive commit sequence. The drill-down query service 110 sorts the data by commit_lsn and returns the row_change sequence and watermark_lsn. When needed, it returns the original log fragment based on log_offset.

[0030] In one implementation, the log mirror storage is partitioned by time range or by LSN range and maintains a Bloom filter for each partition to skip partitions that do not contain the target entity_id.

[0031] In one implementation, watermark_lsn employs low-water mark semantics and maintains a set to be completed when it arrives out of order, so that watermark_lsn and its preceding mirror events satisfy persistence completeness.

[0032] In an enhanced embodiment, the system of the present invention further includes a transaction log drill-down index module, which is used to perform trailing parsing and mirror storage of committed changes generated by the state update execution unit 60 in the business database 80, so as to provide positioning capabilities for the replay, verification or compensation of idempotent state updates.

[0033] The transaction log drill-down index module includes a transaction log trailing read unit, a log parsing unit, and a log mirror storage unit. The transaction log trailing read unit continuously trails the transaction log stream of the business database 80. The log parsing unit parses the logs in LSN order to obtain the commit sequence number commit_lsn, transaction identifier tx_id, row-level change record row_change, and original log location information log_offset. It only outputs the mirror event after detecting a transaction commit flag to avoid uncommitted transactions affecting downstream processes. The log mirror storage unit persists the mirror event and builds a drill-down index. The drill-down index includes at least one index entry with (entity_id, Epoch_lock, commit_lsn) as the key and containing log_offset, where Epoch_lock is the epoch_field value or an equivalent epoch lock value written to the record corresponding to entity_id in the business database 80.

[0034] In one implementation, the log parsing unit uses (commit_lsn, tx_id) as the deduplication key to avoid duplicate consumption of the same transaction.

[0035] In one implementation, the drill-down query unit 110 receives a query request containing an entity_id, sorts the matched mirrored events by commit_lsn, and returns the corresponding log_offset. The drill-down query unit 110 can also maintain a low-watermark_lsn, indicating that all transactions with commit_lsn <= watermark_lsn have been fully consumed and persisted to the log mirroring storage, thus providing consistency boundary semantics to the caller.

[0036] The above are merely preferred embodiments of the present invention. Those skilled in the art can make various modifications or substitutions without departing from the spirit of the present invention: for example, replacing the memory key-value store with other distributed storage that supports atomic conditional writing and script execution environments; replacing the epoch interval allocation unit with other coordination services that support globally incrementing counters; replacing the SQL form of double-predicate conditional updates with equivalent stored procedures or application-layer conditional updates; replacing transaction log tailing with other change data capture mechanisms, etc., all of which should fall within the protection scope of the present invention.

Claims

1. A distributed idempotent state update and expired write shielding device configured on a processing node, characterized in that, include: The interval acquisition module is used to initiate an interval request to an external global coordination center to obtain a strictly increasing and non-overlapping epoch lock interval [Epoch_low, Epoch_high], and the interval step size step(E) is determined according to the node running feature vector E of the processing node; the memory placeholder module is used to send an atomic conditional write instruction to external memory-type storage to establish a placeholder record with the idempotent_key as the primary key. The placeholder record contains at least node_id, [Epoch_low, Epoch_high], state field, create_ts, and takeover_low field, and sets the placeholder lifetime time ttl1; the state field is initially PROCESSING, and when the placeholder record already exists and state is COMMITTED, it is judged as a duplicate request and rejected; the epoch lock selection module is used to select an epoch lock Epoch_lock within the epoch lock interval, satisfying Epoch_low <= Epoch_lock <= Epoch_high, where Epoch_lock is selected sequentially within the interval using a node-local incrementing method; Database transaction update module, used to initiate database transaction instructions to the external business database to apply row-level exclusive locks to the target record within the same database transaction and execute them sequentially: (i) Interval switching update: when the upper bound field epoch_hi_field of the target record is less than Epoch_low, update epoch_hi_field to Epoch_high; (ii) In-interval write update: when both epoch_field and Epoch_lock are satisfied, update the business status field of the target record and update epoch_field to Epoch_lock; when the in-interval write update condition is not met, refuse to write to shield expired writes; Status confirmation module, used to send a status update instruction to the memory storage after detecting that the database transaction has been successfully committed, update the state of the corresponding placeholder record from PROCESSING to COMMITTED, and the status update operation carries Epoch_lock or the epoch lock interval identifier to verify the validity of the placeholder record; The takeover competition module is used to initiate a takeover competition when a placeholder record is detected, the state is PROCESSING, and the difference between the current time and create_ts exceeds the ttl1. The competing node requests a second interval [Epoch_low2, Epoch_high2] and performs a takeover decision update on the placeholder record through an atomic script: updating takeover_low from null to Epoch_low2 or updating it to Epoch_low2 if Epoch_low2 is less than the already recorded takeover_low. The node that successfully updates obtains the sole takeover right and continues to execute the database transaction update module using its own epoch lock interval. The state update request processed by the device carries an event timestamp event_ts and evidence metadata evidence_meta. The event timestamp is used to identify the time when the event that triggered this update occurred, and the evidence metadata is used to record the event source and processing context information.

2. The apparatus according to claim 1, characterized in that, The global coordination center maintains a globally incrementing counter C and returns the epoch lock interval [Epoch_low, Epoch_high] when responding to the interval request. Each allocation sets Epoch_low = C + 1 and Epoch_high = C + step(E), and updates the counter to C = Epoch_high to ensure that the intervals are strictly incrementing and do not overlap. Furthermore, step(E) satisfies the constraints: step_min <= step(E) <= step_max, and step(E) is a positive integer. Step(E) = round(step_min + (step_max - step_min) × (1 - health_score)), where health_score is calculated by the node's running feature vector E and constrained to [0,1] by clamp. The device dynamically adjusts the interval step size step(E) to regulate the frequency of requesting epoch lock intervals from the global coordination center, thereby reducing the coordination center's interaction overhead.

3. The apparatus according to claim 1, characterized in that, The lifetime ttl1 is determined based on the node's running feature vector E in the following way: ttl1(E) = clamp(ttl_base × (1 + β × reliability_score(E)), ttl_min, ttl_max), where ttl_base is the base lifetime, β is an adjustment coefficient, and reliability_score(E) is the node reliability score calculated based on the feature vector E. The clamp function constrains the result within the range of [ttl_min, ttl_max]. The reliability score is negatively correlated with the number of historical node failures and positively correlated with the request success rate and the node's runtime. The device sets the expiration time of the placeholder record based on the calculated ttl1(E) to shorten the duration of the placeholder record and trigger takeover when the node's reliability score decreases, thereby reducing the retry and waiting overhead caused by placeholder hanging.

4. The apparatus according to claim 1, characterized in that, The takeover competition module also includes a takeover priority comparison when initiating a takeover competition: the placeholder record also includes a takeover_priority field, and the atomic script compares the priority of the competing node while updating takeover_low. The update is only allowed when the priority of the competing node is higher than the recorded priority or when takeover_priority is empty; the priority is determined according to the stability index in the node running feature vector E, and the higher the stability, the higher the priority.

5. The apparatus according to claim 1, characterized in that, Before applying a row-level exclusive lock to the target record, the device initiates a lock-free snapshot read to the business database to obtain the current values ​​of epoch_field and epoch_hi_field of the target record. When it detects that epoch_field >= Epoch_lock, it determines the current write request as an expired write and skips subsequent locking and update operations.

6. The apparatus according to claim 1, characterized in that, It also includes a database layer idempotent verification module, which is used to maintain a deduplication record table in the database or maintain the last_idempotent_key field in the business table. Before performing write updates within the specified interval, it first verifies whether the idempotent_key has been processed. If it has been processed, it directly returns an idempotent response, serving as a supplementary guarantee layer to the memory placeholder module.

7. The apparatus according to claim 1, characterized in that, It also includes a transaction log drill-down index module, which is used to follow up by reading the transaction log of the business database and parse it to obtain commit_lsn, tx_id, row_change and log_offset. After detecting the transaction commit flag, it generates a mirror event and persists it. The following drill-down index is established to support audit traceability: an index item with (entity_id, Epoch_lock, commit_lsn) as the key and containing log_offset, and maintains the watermark_lsn consistency boundary corresponding to each entity_id to indicate the maximum commit_lsn of the consecutive commit sequence.

8. A data structure for resource entity records used to implement distributed idempotent state updates and expired write shielding, stored in a computer-readable storage medium of a business database, characterized in that, The data structure includes at least: a resource entity identifier (entity_id), a state value field (state_value), a state upper limit field (state_max), an epoch field (epoch_field), and an interval upper limit field (epoch_hi_field). Furthermore, the data structure is configured to satisfy the following constraints to shield expired writes when performing update transactions in the business database: during the interval switching phase, updating epoch_hi_field to epoch_high is allowed only if epoch_hi_field < epoch_low; during the interval write phase, updating state_value and epoch_field is allowed only if both (epoch_field < epoch_lock) and (epoch_lock <= epoch_hi_field) are simultaneously satisfied, where epoch_low, epoch_high, and epoch_lock are generated by the processing node and carried with the update transaction.

9. A distributed idempotent state update and expired write masking method executed by a processing node, characterized in that, Includes the following steps: S1: Initiate a range request to the external global coordination center to obtain a strictly increasing and non-overlapping epoch lock range [Epoch_low, Epoch_high]; S2: Send an atomic conditional write instruction to the external memory-based storage, establish a placeholder record with the idempotent_key as the primary key and set ttl1, and initially set the state field to PROCESSING; if the placeholder record already exists and the state is COMMITTED, it is determined to be a duplicate request and the processing is terminated; S3: Select Epoch_lock using a node-local incrementing method within the epoch lock range; S4: Initiate a database transaction instruction to the external business database to apply a row-level exclusive lock to the target record within the same transaction and execute it sequentially: when epoch_hi_field is less than Epoch_low, update epoch_hi_field=E. poch_high; when epoch_field is less than epoch_lock and epoch_lock is less than or equal to epoch_hi_field, update the business status and write back epoch_field=epoch_lock; otherwise, refuse to write; S5: after the database transaction is successfully committed, send a status update instruction to the memory storage to update the status of the corresponding placeholder record to COMMITTED; S6: when the state of the placeholder record is detected to be PROCESSING and the difference between the current time and create_ts exceeds ttl1, apply for a second interval and update takeover_low through an atomic script to determine the sole takeover node, and the takeover node returns to step S4 to continue execution.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the distributed idempotent state update and expired write masking method described in claim 9, which is executed by a processing node.

Images