Hierarchical progressive periodic fee increment processing system and method based on shard collaboration

Abstract

The application relates to a hierarchical progressive periodic fee increment processing method and system based on sharding cooperation, which comprises the following steps: starting a fee calculation task and configuring scheduling parameters, and distributing a task trigger signal containing the fee calculation task and the scheduling parameters to each working node; traversing data of the fee calculation task in a multi-level progressive structure, wherein a two-level sharding mechanism is enabled to realize data sharding cooperation during the data traversal process; performing single-account fee increment calculation and persistence, and simultaneously performing increment change chain evidence; and after all levels are traversed, batch closing and tail difference balance are performed.

Description

Technical Field

[0001] This application relates to the field of periodic cost data processing, and more specifically, to a hierarchical progressive incremental processing system and method for periodic costs based on fragmented collaboration. Background Technology

[0002] Currently, in the field of batch calculation of periodic costs (such as late payment penalties, late fees, and interest) and the broader area of ​​periodic numerical increments, the mainstream technical solutions mainly include the following: Option 1 Single-machine full-scan solution This solution uses a single server to perform a one-time scan and calculation of all overdue data at a fixed time. The system is triggered by a scheduled task, iterating through all eligible bill records, calculating the penalty for each, and updating the database accordingly. This solution is simple and intuitive to implement, making it suitable for scenarios with relatively small data volumes.

[0003] This solution has the following drawbacks: 1) When the data volume reaches millions or more, the single-machine processing time is too long, and it may not be possible to complete the full calculation within the scheduled maintenance window; 2) As the amount of data increases, the system cannot improve its processing capacity by simply adding more machines, and there is a clear performance ceiling; 3) Single point of failure risk is high. Once the process is interrupted, it is necessary to resume from the breakpoint or recalculate the entire data.

[0004] Option 2 Batch processing solution based on OFFSET pagination

[0005] This scheme uses the traditional OFFSET / LIMIT pagination method to traverse the data to be processed. The system queries data page by page according to a fixed page size, processes each page, and then queries the next page until all data has been processed. Some systems have introduced multi-threading or distributed scheduling to improve parallelism.

[0006] This solution has the following drawbacks: 1) OFFSET pagination has serious performance issues when paginating deeply. The database needs to scan and discard the first N records before returning the target page, and the query time increases linearly with the page number. 2) If data is inserted or deleted during the traversal process, it may result in data omission or duplicate processing; 3) Offset paging in a multi-threaded environment is prone to concurrency conflicts and data inconsistencies.

[0007] Option 3 Real-time per-transaction calculation scheme

[0008] This solution calculates late payment penalties in real time each time a user makes a query or payment, rather than processing them through scheduled batch tasks. When a user initiates a payment request, the system calculates the number of overdue days and the penalty amount in real time based on the bill's due date and the current date.

[0009] This solution has the following drawbacks: 1) Compound interest calculation involves exponentiation, which incurs high computational costs when the overdue period is large, affecting the user's payment response speed; 2) The inability to provide complete records of daily changes in penalty fees hinders financial reconciliation and audit traceability; 3) During peak periods with a large number of concurrent queries, real-time computing can put significant pressure on the database and application server; 4) It cannot automatically accumulate and display penalty information when the user does not actively query it.

[0010] Option 4 Distributed solutions for application-layer sharding

[0011] This solution performs data sharding and routing at the application layer (e.g., dividing by merchant ID range), allocating data from different ranges to different computing nodes for processing. The application layer maintains a sharding mapping table to find the corresponding processing node based on the merchant ID.

[0012] This solution has the following drawbacks: 1) Application-layer sharding requires maintaining an additional sharding mapping table, which increases system complexity and consistency maintenance costs; 2) Static range sharding is prone to severe load skew when data distribution is uneven; 3) Updating the sharding mapping table (such as scaling up or down nodes) requires coordination of all nodes, which poses a risk of service interruption; 4) Application layer sharding is coupled with data query logic, and changes to sharding require modifications to the application code.

[0013] In summary, in actual operation, the system needs to perform incremental calculations on hundreds of thousands or even millions of records daily, ensuring the accuracy and consistency of the calculation results, in order to handle the batch calculations of periodic costs and numerical increments. However, existing mainstream implementation solutions suffer from the following prominent problems: Question 1 The existing solutions suffer from insufficient horizontal scalability and data skew. Most solutions rely on single-machine serial processing or static sharding at the application layer. When the amount of managed data grows to a certain scale (e.g., exceeding 500,000 active records), single-machine processing time may exceed 2 hours. Traditional sharding strategies are typically based on static data range division. When data distribution is uneven—for example, a top merchant manages tens of thousands of records while a regular merchant only manages a few dozen—the load difference between shards is huge, causing some nodes to complete prematurely while others remain overloaded for extended periods.

[0014] Question 2 Deep pagination leads to performance degradation. Existing solutions generally use OFFSET pagination to traverse data. When processing the Nth page, the database needs to scan the first N × pageSize records before returning the target page data. In a real production environment, when processing tens of thousands of pages, the time taken for a single query can degrade from milliseconds to seconds or even slower, with the overall processing time increasing non-linearly.

[0015] Question 3 Concurrency safety issues. In distributed, multi-node parallel processing scenarios, multiple nodes may modify the same record simultaneously. Existing solutions typically use pessimistic locking (SELECT FOR UPDATE) to ensure data consistency, but pessimistic locking can lead to lock waiting and deadlock risks, severely impacting system throughput in high-concurrency scenarios. Some solutions use standard optimistic locking (version field), but this requires additional modifications to the table structure and lacks effective backoff strategies in high-conflict scenarios, leading to retry storms.

[0016] Question 4 The existing solutions suffer from several drawbacks: lack of idempotency guarantees and coarse-grained mechanics. When a scheduled task is retried after an abnormal interruption, existing solutions often fail to identify which data has already been processed, leading to the same record being calculated repeatedly on the same day. While some solutions introduce batch-level idempotency, the granularity is too coarse—if a batch processes 1000 records but fails at the 800th record, the entire batch is skipped during retry, and the remaining 200 unprocessed records are permanently missed.

[0017] Question 5 The initial historical data recalculation is inefficient and lacks retrospective capabilities. When the system is first launched or new incremental functions are added, all overdue records need to be recalculated. However, the existing solution does not differentiate between simple interest and compound interest models. Under the compound interest model, it is impossible to directly solve using mathematical formulas, resulting in extremely high computational overhead for the initial recalculation. More importantly, when monitoring policies change or historical data entry errors are discovered, the existing solution completely lacks the ability to retrospectively correct them, making it impossible to accurately recalculate settled historical batches.

[0018] Question 6 The existing solution suffers from insufficient financial compliance and audit traceability capabilities. It only rounds individual records to the nearest whole number, failing to account for discrepancies after batch aggregation. The aggregated amount after rounding tens of thousands of records individually deviates from the theoretical amount, thus failing to meet financial audit requirements. Furthermore, it lacks an immutable calculation audit chain, making it unsuitable for compliance in subsequent evidence-gathering scenarios.

[0019] Therefore, there is a need to provide an improved solution for handling periodic costs to address these issues. Summary of the Invention

[0020] This application relates to a system and method for periodic incremental changes and persistence of structured master-slave hierarchical data in a distributed environment. This application is not dependent on a specific hardware architecture and can run on any general-purpose server cluster, serverless function computing platform, or hybrid cloud environment that supports distributed scheduling.

[0021] According to a first aspect of this application, a hierarchical progressive incremental periodic cost processing method based on fragmented collaboration is provided, comprising: Start the cost calculation task and configure the scheduling parameters, and distribute the task trigger signal containing the cost calculation task and the scheduling parameters to each working node; The data of the cost calculation task is traversed in a multi-level progressive structure, wherein a two-level sharding mechanism is enabled during the data traversal process to achieve data sharding collaboration. Perform incremental calculation and persistence of single-invoice expenses, and simultaneously perform incremental change chaining; and After all levels have been traversed, batch closing and reconciliation of tail differences are performed.

[0022] According to a second aspect of this application, a hierarchical progressive periodic cost incremental processing system based on fragmented collaboration is provided, comprising: A distributed scheduling center is configured to initiate a cost calculation task, configure scheduling parameters, and distribute a task trigger signal containing the cost calculation task and the scheduling parameters to each working node. Multiple worker nodes are configured to traverse the data of the cost calculation task in a multi-level progressive structure, wherein during the data traversal process, a two-level sharding mechanism is enabled to achieve data sharding collaboration and to perform single bill cost incremental calculation and persistence. A shared persistent memory is configured to persistently store the computation results output by the worker node; and A chain-linked evidence storage system is configured to perform incremental change chain-linked evidence storage. The distributed scheduling center is also configured to perform batch closing and tail reconciliation after all levels have been traversed.

[0023] This overview is provided to introduce, in a simplified form, some of the concepts further described in the detailed description below. This overview is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0024] To describe how the above and other advantages and features of the invention are obtained, a more detailed description of the invention, which has been briefly described above, will be presented with reference to specific embodiments of the invention shown in the accompanying drawings. It will be understood that these drawings depict only exemplary embodiments of the invention and are therefore not intended to limit its scope. The invention will be described and explained using the drawings and with the aid of additional features and details, in which: Figure 1 An example flowchart of a hierarchical progressive periodic cost increment processing method based on fragmented collaboration according to an embodiment of this application is shown.

[0025] Figure 2 The specific execution flow of the hierarchical progressive data traversal stage in the hierarchical progressive periodic cost increment processing method is shown.

[0026] Figure 3 The specific execution flow of the single-bill cost increment calculation and persistence stage in the hierarchical progressive periodic cost increment processing method is shown.

[0027] Figure 4 It shows Figure 3 The specific details of the "two-dimensional idempotency check" in the text.

[0028] Figure 5 The specific execution flow of incremental change chain storage in the hierarchical progressive periodic cost incremental processing method is shown.

[0029] Figure 6 An example structural diagram of a hierarchical progressive periodic cost incremental processing system based on fragmented collaboration according to an embodiment of this application is shown. Detailed Implementation

[0030] To address the aforementioned existing technical problems, the core improvements in this application include the following aspects: 1) Decentralized horizontal scaling is achieved through deterministic mapping functions at the SQL layer, and second-level nested sharding is introduced to solve data skew. 2) Eliminate deep page-turning performance issues by replacing OFFSET pagination with cursor pagination; 3) Improve concurrency throughput and stability by combining an optimistic locking mechanism based on the original values ​​of business fields with an exponential backoff retry strategy; 4) It prevents duplicate calculations and supports precise continuation of the process through dual-dimensional idempotency protection and record-level state marking; 5) The dual-mode calculation strategy of initial historical accumulation and subsequent daily increments balances efficiency and accuracy; 6) Supports accurate recalculation of historical batches through time-backtracking incremental correction capabilities; 7) Meet audit compliance requirements through financial-level tail difference collection and chain-based evidence storage.

[0031] To achieve the above improvements, Figure 1 The present application discloses an example flow stage of a hierarchical progressive periodic cost increment processing method based on fragmented collaboration according to an embodiment of the present application. As shown in the figure, the method, in the order of execution from beginning to end, includes the following five main stages: S1 Task scheduling and data fragmentation configuration → S2 Hierarchical progressive data traversal → S3 Single bill cost increment calculation and persistence → S4 Incremental change chain storage → S5 Batch closing and tail difference reconciliation.

[0032] Phase S1: Task scheduling and data sharding configuration.

[0033] First, in step S1-1, the distributed scheduling center starts the cost calculation task at a preset trigger time.

[0034] In step S1-2, the scheduling parameters are configured, including: 1) Generate batch identification information corresponding to this task—batch business date (batchBizDate) and globally unique batch serial number (batchSeqNo). The batch business date is a logical date rather than a physical date, used to solve the idempotency determination problem when running batches across midnight.

[0035] 2) Set the total number of shards (shardingTotalCount, e.g., N) and sharding item (shardingItem, e.g., k) for the cost calculation task. The total number of shards and the sharding item are used to split the cost calculation task into N independent subtasks in the first-level sharding of subsequent stages. Each subtask carries a unique sharding item k (k = 0, 1, ..., N-1) and the batch serial number.

[0036] In steps S1-3, the distributed scheduling center distributes the data containing the cost calculation task and the above scheduling parameters to each working node, so that each working node obtains a shard number k and batch identification information, and the process enters stage S2.

[0037] Phase S2: Hierarchical progressive data traversal.

[0038] Figure 2The process of the hierarchical, progressive data traversal phase S2 is demonstrated: starting with breakpoint resume checks, it proceeds through multiple nested loops including first-level merchant cursor queries (which can be further expanded to include second-level nested shard evaluations if necessary), payment item cursor traversal, hot and cold data splitting, and overdue bill cursor traversal, ultimately outputting a single overdue bill for processing in phase S3. The process incorporates key optimization mechanisms such as cursor position persistence, master-slave routing strategies, and local hotspot configuration caching strategies.

[0039] As shown in the figure, within each working node, the system traverses the data in a multi-level progressive structure.

[0040] The default hierarchy is three levels: the first level iterates through the merchant set, the second level iterates through the payment items set under the current merchant, and the third level iterates through the overdue bills set under the current payment item. The number of levels can be configured to more levels according to business needs (such as adding a "community / property management" level to form a four-level structure).

[0041] Each working node is nested and traversed layer by layer according to the parent-child hierarchy of the business entities (merchant → payment item → overdue bill) using cursor pagination based on primary key ID (id > lastId).

[0042] Specifically, in step S2-1, the worker node parses the scheduling parameters and checks the breakpoint resume flag. If it exists, the cursor's persistent position is loaded; otherwise, the cursor starts from its initial value.

[0043] In step S2-2, first-level sharding is enabled, that is, the SQL layer sharding filter condition MOD(row_id,shardingTotalCount) = shardingItem, combined with the cursor pagination condition id > lastMchntId (i.e., first-level sharding), queries the list of merchants responsible for the current shard.

[0044] The purpose of first-level sharding (node-level) is to ensure that each worker node independently executes the sharding of the merchant data it is responsible for. The distributed scheduling center can use a deterministic mapping function (MOD modulo) to distribute all the data to be processed to N worker nodes without overlap, thus achieving data sharding coordination. Therefore, this level of sharding rule can be programmed as follows: add WHERE MOD(merchant.row_id, N) = k to the SQL query condition, where merchant.row_id is the primary key ID of the merchant table, N is the total number of shards, and k is the current shard number. This rule ensures that each merchant is processed by only one worker node, achieving non-overlapping data partitioning. No communication or coordination is required between worker nodes; they execute their respective computational tasks completely independently.

[0045] In step S2-3: Evaluate whether to enable second-level nested sharding for each merchant in the merchant list.

[0046] To address the issue of excessive load on a single shard caused by a large number of overdue records under the jurisdiction of top merchants, a two-level nested sharding mechanism is enabled within the working node (within the merchant, the shards are divided into two segments according to the time dimension) to achieve two-layer load balancing.

[0047] When the number of pending records for a single merchant exceeds a preset threshold (e.g., 5000 records), the system automatically enables secondary sharding: within the current merchant, sharding is performed based on the bill due date, dividing a large batch of records into several time sub-intervals (e.g., by quarter), with each sub-interval serving as an independent processing unit. Secondary sharding does not change node allocation, but only optimizes the traversal granularity within a single node, ensuring that the page size of cursor pagination matches the data volume, thus avoiding a decrease in cursor traversal efficiency due to excessive data volume for a single merchant.

[0048] Therefore, the two-level data sharding mechanism described in this application can be programmatically expressed as follows: First-level sharding: WHERE MOD(merchant.row_id, N) = k -- Node-level allocation, where merchant.row_id is the primary key ID of the merchant table; Second-level segmentation: WHERE bill.due_date BETWEEN rangeStart AND rangeEnd -- Intra-merchant segmentation, where bill.due_date is the bill due date, rangeStart is the start time range, and rangeEnd is the end time range.

[0049] In steps S2-4, the list of payment items under the current merchant is queried by pagination using the cursor.

[0050] In steps S2-5, during the traversal process, a hot and cold data hierarchical scheduling strategy is executed, that is, hot data is split into hot and cold data (hot data is calculated daily, and cold data is calculated monthly).

[0051] Specifically, the system categorizes data to be processed into hot and cold datasets based on business attributes (such as the number of days a bill is overdue). Hot data (e.g., overdue within 90 days) undergoes daily incremental calculations, while cold data (e.g., overdue for more than 90 days) undergoes monthly incremental calculations. Cold data can be stored in a low-cost storage layer and retrieved via pagination using a federated cursor during queries. This strategy reduces overall computing power by over 60% while maintaining data integrity. The threshold for determining hot and cold data stratification can be configured by business personnel, and the system periodically (e.g., monthly) reassesses the hot and cold data classification.

[0052] In steps S2-6, the list of overdue bills under the current payment item is queried by pagination using the cursor.

[0053] In steps S2-7, after each page is processed, a cursor persistence strategy is executed: the cursor position of each level is persisted to the metadata table in real time to support breakpoint resumption.

[0054] Specifically, the current position (lastId value) of each cursor level is persisted to a metadata table in the format {shardIndex, level, entityParentId, lastId}. If a task is interrupted due to an exception, the system reads the most recently persisted cursor position from the metadata table and resumes traversal from the point of interruption, without having to start from the beginning. The cursor position is updated once after each page is processed.

[0055] To mitigate the risk of cursor read errors caused by master-slave latency during the multi-level progressive data traversal process, a master-slave routing strategy can be implemented: the outermost merchant traversal and payment item traversal use the slave database for queries (read-heavy, write-light, suitable for sharing read pressure), while the innermost bill traversal and incremental calculations use the master database (requiring the latest data for optimistic locking comparisons and writes). This strategy ensures data consistency while fully utilizing the read scalability of the slave database.

[0056] In other words, the outermost traversal goes through the slave database, and the innermost traversal goes through the master database, thereby eliminating the risk of missed reads caused by master-slave latency.

[0057] Alternatively, a local hotspot configuration caching strategy can be implemented. Each worker node independently caches the configuration of hotspot payment items (interest rates, exemption rules, etc.), and performs cache consistency checks using the configuration version number. Each time a payment item is queried, the local cache version number is compared with the database version number. If the version numbers match, the cached version is used; otherwise, it is refreshed. This strategy eliminates the need for a distributed cache cluster, maintaining the original decentralized nature while reducing database query pressure.

[0058] To help those skilled in the art better understand the hierarchical progressive data traversal stage, the following is combined with... Figure 2 This section will further detail the implementation of each step in the hierarchical progressive data traversal phase.

[0059] When a worker node receives a task trigger signal from the distributed scheduling center, it initiates the cost calculation process. The hierarchical progressive data traversal process officially begins at step 201.

[0060] Step 202: The worker node extracts scheduling parameters (sharding item (corresponding to the current sharding item number k mentioned earlier), total number of shards (shardingTotalCount, corresponding to the total number of shards N mentioned earlier), and batch identifiers (batch business date (batchBizDate) and globally unique serial number (batchSeqNo)) from the task trigger signal. Simultaneously, it parses optional task parameters (such as specified merchant ID (mchntCd), asset ID (assetCode), bill number (billNo), retroactive flag (retroactiveFlag), and other filtering conditions). These parameters are passed in, for example, in JSON format through the jobParameter field of the scheduling framework.

[0061] Step 203: Check the resume flag. The worker node queries the metadata table to see if there is an incomplete task record with the current sharding item. If it exists and the task parameters specify resume (resumeFlag=true), the most recent persistent position of each cursor level is loaded from the metadata table, and execution resumes from the point of interruption. If it does not exist or resume is not specified, each cursor level starts from the initial value (lastId=0).

[0062] Step 204: Determine if a merchant ID is specified. If a merchant ID is explicitly specified in the task parameters, proceed directly to the processing flow for that merchant. If no merchant ID is specified, proceed to the full-scale sharded query mode in Step 205.

[0063] Step 205: Cursor-based pagination query of the merchant list. The system uses the SQL-level sharding filter condition MOD(row_id, shardingTotalCount) = shardingItem in conjunction with the cursor pagination condition id > lastMchntId to query the list of merchants handled by the current shard. This query uses the database slave database. Each query has a fixed page size (e.g., 200 records), and the query results are sorted in ascending order by id.

[0064] Step 206: Determine if the merchant list is not empty. If the merchant list returned by the current query is empty, it means that all merchants in the current shard have been traversed, and the process jumps to the end. If the list is not empty, proceed to step 207.

[0065] Step 207: Traverse the current merchants (first loop). For each merchant on the current page, execute the subsequent second and third loops in sequence. If an exception occurs during the processing of a single merchant, the system will capture the exception, log it, and continue processing the next merchant, ensuring that a single point of failure does not affect the overall task progress.

[0066] Step 208: Secondary segmentation assessment within the merchant. For the current merchant, the system first queries the total number of overdue bills under its jurisdiction. If the number exceeds a preset threshold (e.g., 5000 bills), secondary nested segmentation is enabled: the bills are divided into several time sub-intervals based on their due dates, and each sub-interval is processed as an independent unit and proceeds to step 209; if the threshold is not exceeded, the process proceeds directly to step 209 without secondary segmentation.

[0067] Step 209: Cursor-based paginated query of payment items (second level). For the current merchant (or the current time sub-interval), the system queries the list of payment items for which late payment penalties need to be calculated, using the conditions `lateFeeEnabled = 1` (late payment penalty function is enabled) and `id > lastItemId`. This query uses the database slave database. Each query has a fixed page size (e.g., 100 records). The payment items include the core configuration parameters required for late payment penalty calculation: billing method (simple interest / compound interest / tiered / capped), daily interest rate, and the number of overdue days to start calculating, etc.

[0068] Step 210: Check if the payment item list is not empty. If there are no payment items with the penalty function enabled under the current merchant, return to the first loop to process the next merchant. Otherwise, proceed to step 211.

[0069] Step 211: Iterate through the payment items (second loop). For each payment item under the current merchant, read its late payment penalty configuration parameters and check the local cache version number. If the cache version number matches the database version number, use the cached configuration; otherwise, refresh the cache. Then, call the third-level processing logic.

[0070] Step 212: Hot and Cold Data Separation Determination. For the current payment item, the system determines whether the current batch should process hot or cold data based on the configured hot and cold data stratification threshold (e.g., overdue for 90 days). If it is the scheduling period for cold data (e.g., the beginning of the month), the system queries the cold data set; otherwise, it queries the hot data set. The hot and cold data sets use different overdue start dates.

[0071] Step 213: Cursor-based paginated query of overdue bills (third level). For the current payment item and the current dataset (hot / cold), the system calculates the overdue start date = today – lateFeeOverdueStartDay (number of overdue days). Then, it queries the list of overdue bills under this payment item using the condition period_end < overdue start date and id > lastBillId. This query uses the main database to ensure that the latest data is obtained for subsequent optimistic locking comparisons. Each query uses a fixed page size (e.g., 500 records).

[0072] Step 214: Check if the bill list is not empty. If there are no overdue bills under the current payment item, return to the second loop to process the next payment item. Otherwise, proceed to step 215.

[0073] Step 215: Iterate through overdue bills (third loop). For each overdue bill under the current payment item, sequentially call the incremental calculation and persistence process (see details). Figure 3 When a single invoice processing fails, the exception is captured, an error log is logged, the record is marked as "failed," and the next invoice is processed, ensuring fault tolerance and isolation.

[0074] Step 216: Incremental Calculation and Persistence. For the currently traversed overdue bills, the incremental calculation engine is invoked to complete incremental calculation and database persistence. Detailed calculation logic and persistence strategies are as follows... Figure 3 As stated above.

[0075] Step 217: Cursor persistence and statistics update.

[0076] After each page is processed, the current position (lastMchntId, lastItemId, lastBillId) of each cursor layer is updated to the metadata table. At the same time, the processing statistics for this batch are updated: number of records processed, number of successes, number of failures, and number of skips.

[0077] Step 218: Process ends.

[0078] Phase S3: Incremental calculation and persistence of single bill expenses.

[0079] exist Figure 3 The document illustrates the specific execution process of S3 in the single-bill expense incremental calculation and persistence phase: First, a two-dimensional idempotency check is performed (see details). Figure 4 Then, based on the backtracking marker and the existence of historical records, it is divided into three branches: backtracking correction path, initial calculation path, and subsequent incremental path. Each path completes incremental calculation, tail difference collection, and optimistic lock update (including exponential backoff retries). After successful writing, chained notarization is triggered synchronously (see details). Figure 5 Each worker node runs an incremental calculation engine, responsible for performing incremental calculations and database persistence operations on each overdue record encountered. The incremental calculation engine supports multiple billing methods (simple interest, compound interest, tiered interest rates, and capped rates), and supports three calculation modes: initial full accumulation, subsequent daily increments, and time backtracking correction.

[0080] The verification process includes the following detailed judgment logic: batch-level idempotency judgment based on batch business date (batchBizDate), execution tracing based on globally unique serial number (batchSeqNo), and a precise continuation mechanism based on record-level processing status markers (processing / success / failure / skipped) (the specific details will be further explained in conjunction with subsequent application examples).

[0081] Specifically, in step S3-1, for each overdue bill encountered, the incremental calculation engine first performs a two-dimensional idempotency check (batch business date + global serial number) and a record-level status check (processing / success / failure / skipped) to determine whether to skip the overdue bill, so as to prevent duplicate calculations and support accurate continuation.

[0082] In step S3-2, based on the retroactive flag in the task parameters, it is determined whether it is a retroactive correction mode, and the corresponding calculation path (retroactive correction path or normal calculation path) is selected.

[0083] In step S3-3, after the verification is passed, the incremental calculation engine automatically switches the calculation mode (initial calculation path or subsequent incremental path) according to whether there is a historical record. That is, in the initial calculation, the historical default fee is accumulated day by day from the overdue date; in subsequent calculations, only the incremental amount of one day is calculated each day.

[0084] In steps S3-4, the incremental calculation engine performs incremental calculations (supporting four billing methods: simple interest, compound interest, tiered interest, and capped interest). During the calculation process, the incremental calculation engine supports a time-backtracking correction mode (independent batch backtracking, without polluting the daily data stream).

[0085] In step S3-5, after the calculation is completed, the calculation results are rounded to the nearest integer and the tail difference is collected.

[0086] In steps S3-6, the calculation results are written to the database of the shared persistent storage using optimistic locking (in conjunction with exponential backoff retries) based on the original values ​​of the business fields. The persistent storage ensures data consistency and traceability during concurrent writes from multiple nodes through optimistic locking (in conjunction with exponential backoff retries), a two-dimensional idempotent control strategy, record-level processing status markers, and batch tracking identifiers.

[0087] In steps S3-7, after successful writing, the synchronization trigger phase S4 is initiated.

[0088] In step S3-8, the record processing status is updated to "success".

[0089] To help those skilled in the art better understand the incremental calculation and persistence phase of the single billing fee, the following is combined with... Figure 3Let's further explain the implementation details of each step in the aforementioned stage.

[0090] Step 301: Input Preparation. Receive core input: billing information (including fields such as bill amount, amount received, discount amount, and due date) and payment item configuration (including parameters such as billing method (lateFeeMethod), daily interest rate (lateFeeDailyRate), and overdue start day (lateFeeOverdueStartDay). Also receive batch identifier information (batchBizDate, batchSeqNo).

[0091] Step 302: Calculate the outstanding bill amount (base). Calculate the outstanding amount of the current bill using the formula "Outstanding Bill Amount = Bill Amount - Amount Received - Discount Amount". This amount is the base for incremental calculations.

[0092] Step 303: Determine if the amount due is greater than zero. If the amount due is less than or equal to 0, it means the bill has been fully paid or there is no amount due, so there is no need to calculate the increment and this bill can be skipped. If the amount due is greater than 0, continue to the next step.

[0093] Step 304: Two-dimensional idempotency verification

[0094] The system performs two-level idempotency checks: 1. First-level – Batch-level idempotency: Query the batchBizDate (batch business date) of the latest processed record for this bill. If the batchBizDate equals the current batch business date and the processing status is "successful," it means that the bill has been successfully processed within this business day and can be skipped directly. This mechanism uses the business date instead of the physical timestamp, so batch processing across midnight (e.g., starting at 23:59 and ending at 00:05 the next day) will not be mistakenly judged as duplicate processing.

[0095] 2. Second level—Record-level precise continuation: If the processing status of the bill in the current batch is "failed" or "interrupted", reprocessing is allowed, skipping records that have been successfully processed. Record-level status flags are stored in the batch processing details table in the format {batchSeqNo, billNo, status}, where status has the following values: processing (0), successful (1), failed (2), skipped (3).

[0096] Step 305: Determine the idempotency check result. If the batch-level check passes (processed successfully), skip this invoice. If the record-level check indicates that the batch has been processed successfully, skip this invoice. Otherwise, continue the subsequent calculation process and update the record status to "Processing".

[0097] Step 306: Determine if it is in backtracking correction mode. Check the backtracking flag (retroactiveFlag) in the task parameters. If it is in backtracking correction mode, proceed to the backtracking correction path (steps 306R-306Z); otherwise, proceed to the normal calculation path (starting from step 307).

[0098] [Backtracking and Correction Path]

[0099] Step 306R: Load Retrospective Baseline Data. Based on the target historical date (retroactiveDate) specified in the retrospective parameters, the system loads the original billing data and the rate configuration at that time from the data snapshot table or historical archive. Retrospective corrections use a separate retrospective batch number (batchSeqNo prefixed with "R-"), strictly isolated from daily incremental batches.

[0100] Step 306S: Calculate the retrospective incremental difference. Using the retrospective baseline data as input, calculate the incremental amount within the target date range day by day according to the billing method (simple interest / compound interest / tiered / capped) described in step S311. Compare the calculation result with the incremental amount already recorded on that date to obtain the difference (delta = new calculated value - recorded value).

[0101] Step 306T: Generate retrospective correction records and trigger evidence storage. Use the difference (delta) as the correction increment to generate retrospective correction records. These correction records are written to a separate retrospective correction details table via INSERT, linking the original batch number and the retrospective batch number to form a complete correction audit chain. This path does not modify the original historical record; it only appends correction records, ensuring audit traceability.

[0102] When a backtracking correction record is written, the following safeguard operations are performed synchronously: 1. Optimistic Locking Protection: If the retrospective correction involves adjusting the amount of existing penalty records (e.g., the difference delta needs to be added to the current receivable amount), the UPDATE is executed through the optimistic locking mechanism based on the original value of the business field as described in step S319, with the original receivable amount value before the update carried in the WHERE condition. In case of conflict, the process is retried according to the exponential backoff strategy. If the retrospective correction is only to add an independent correction record without modifying the original record, the INSERT operation prevents duplicate writes through the unique key constraint (UNIQUE KEY uk_retro_bill_batch: bill_no + retro_batch_seq_no) of the retrospective correction details table.

[0103] 2. Chained Evidence Storage: After the retroactive correction record is successfully written, the chained evidence storage writing in phase S4 is triggered synchronously. The evidence storage digest additionally carries a retroactive flag (retroactiveFlag=true) and the original batch number (originalBatchSeqNo), enabling the evidence storage chain to fully reflect the ins and outs of the retroactive correction. The evidence storage hash calculation formula adds two fields: retroactiveFlag and originalBatchSeqNo.

[0104] Step 306Z: Retrospective correction complete. After the retrospective correction record is written and the chained evidence storage is completed, the retrospective batch processing status is updated to "successful," and the retrospective processing of this bill is complete. If the evidence storage writing fails, the asynchronous compensation strategy described in step S319A is followed, without blocking the retrospective correction process.

[0105] [Conventional Calculation Path]

[0106] Step 307: Check existing penalty records. The system checks the penalty details table by bill number (billNo) and merchant number (mchntCd) to see if a penalty record for this bill already exists. The query results are used to determine whether this processing is an initial calculation or a subsequent incremental calculation.

[0107] Step 308: Determine if the record exists. If the record does not exist (initial processing), proceed to the initial calculation path (steps 309-313); if the record already exists (subsequent processing), proceed to the incremental calculation path (steps 314-320).

[0108] [Initial Calculation Path]

[0109] Step 309: Calculate the overdue start date. Overdue start date = Bill due date (dueDate) + Overdue start day (lateFeeOverdueStartDay). The default overdue start day is 0, meaning the calculation starts on the bill due date. If configured as N days, the calculation starts from the Nth day after the due date.

[0110] Step 310: Calculate the cumulative historical late payment penalties. Calculate the total number of overdue days from the overdue date to today (inclusive), then calculate the penalty for each day on a daily basis and add it to the cumulative historical amount. The reason for using a daily cycle instead of a direct formula is that under the compound interest model, the daily interest base (principal + accrued late payment penalties) changes with the number of days, forming a recursive relationship that cannot be accurately solved using a simple power formula (the daily penalty for late payment needs to be rounded to the nearest cent, and the accumulated rounding error will affect the final result).

[0111] Step 311: Determine the billing method. The billing mode is determined based on the `lateFeeMethod` field configured for the payment item: 1. Simple Interest Model: Daily late payment penalty = Bill Amount Due × Daily Interest Rate. Under the simple interest model, the daily interest calculation base is fixed at the bill amount due. The total late payment penalty for N days = Bill Amount Due × Daily Interest Rate × N.

[0112] 2. Compound Interest Model: Daily late payment penalty = (Bill Amount Payable + Current Accumulated Late Payment Penalty) × Daily Interest Rate. Current Accumulated Late Payment Penalty = Accumulated Late Payment Penalty Receivable - Late Payment Penalty Paid - Waived Late Payment Penalty.

[0113] 3. Tiered interest rate model: When the number of overdue days exceeds the preset threshold, the daily interest rate will increase in tiers (e.g., 0.05% daily interest rate for overdue days within 30 days, 0.08% daily interest rate for overdue days between 30 and 90 days, and 0.10% daily interest rate for overdue days).

[0114] 4. Capped Mode: The total penalty amount shall not exceed a certain percentage of the bill amount payable (e.g., not exceeding 30% of the principal), and any excess amount shall be automatically truncated.

[0115] Step 312: Rounding and Rounding Down. Round the accumulated historical penalty amount to the nearest cent (accurate to the integer cent). Record the rounding difference (precise value - rounded value) in the rounding down table, formatted as {batchSeqNo, billNo, roundDiff}. The rounding down table is summarized daily and compared with the theoretical total amount. Any discrepancies are automatically balanced through a fallback ledger to ensure compliance with financial audit standards.

[0116] Step 313: INSERT Initial Record. The initial penalty record is written to the database using an INSERT statement. The receivable amount (receivable_amt) of the new record is set to the cumulative amount, the paid amount (paid_amt), refund amount (refund_amt), and waived amount (exempt_amt) are all initialized to 0, the waiver status (exempt_st) is set to "not waived", the batch business date (batch_biz_date) is set to the current batch business date, the globally unique serial number (batch_seq_no) is set to the current batch serial number, and the processing status (process_status) is set to "success". Simultaneously, the incremental change summary is written to the chained evidence storage structure (triggering phase S4, see details). Figure 5 ).

[0117] INSERT Concurrency Conflict Capture and Handling: Under normal scheduling scenarios, the MOD sharding mechanism ensures that each bill is processed by only one worker node, and INSERT will not generate concurrency conflicts. However, in the following abnormal scenarios, two execution instances may process the same bill at the same time: (1) the time windows of the backtracking task and the daily incremental task overlap; (2) scheduling anomalies cause the same shard to be repeatedly distributed. To defend against such boundary situations, the system establishes a unique key constraint on the penalty details table: UNIQUE KEY uk_bill_mchnt (bill_no, mchnt_cd). The INSERT operation in step 313 adopts the "INSERT + unique key conflict capture" strategy: after executing INSERT, if the database returns a unique key conflict exception (such as the MySQL Duplicate entry error, error code 1062), the system captures the exception and does not throw an error, but automatically downgrades to the incremental calculation path - re-query the existing penalty records and transfer to the subsequent incremental calculation path in step 314 for execution. This degradation process is logged as an "INSERT conflict degradation" event, facilitating post-event troubleshooting of scheduling anomalies. Through this mechanism, even in extreme concurrency scenarios, the system can guarantee data integrity and prevent duplicate records.

[0118] [Subsequent Incremental Calculation Path]

[0119] Step 314: Calculate the current penalty payable. Current penalty payable = Existing penalty receivable (receivable_amt) - Paid penalty (paid_amt) - Exempted penalty (exempt_amt). If the result is negative (paid amount exceeds receivable amount), the payable amount is 0. This payable amount is a component of today's interest calculation base under the compound interest model.

[0120] Step 315: Calculate today's penalty. Calculate the daily increment based on the billing method: 1. Simple interest model: Today's penalty = Bill payable amount × Daily interest rate.

[0121] 2. Compound interest model: Today's penalty = (Bill amount due + Current penalty due) × Daily interest rate.

[0122] 3. Tiered interest rate model: The daily interest rate is determined based on the current overdue days range, and then calculated according to the corresponding formula.

[0123] 4. Capped Mode: After calculating the increment, check if the cumulative total exceeds the upper limit; if it does, truncate.

[0124] This only calculates the incremental amount for one day, rather than recalculating all historical amounts, so the computational overhead is extremely low.

[0125] Step 316: Determine the waiver status. Check the waiver status (exempt_st) of the current penalty record. If it is in the "waiver in effect" status (exempt_st = 1), it means that although the penalty for this bill is still being calculated (for the integrity of financial records), the user does not need to actually pay it.

[0126] Step 317a (Exemption Status): Today's late payment penalty is simultaneously added to both the receivable amount and the exempted amount fields. That is: New receivables = Original receivables + Today's late payment penalty, New exemption = Original exemption + Today's late payment penalty. In this way, the user's actual payable late payment penalty (receivables - exemption) remains unchanged, but both the total receivables and the total exempted amount have complete cumulative records.

[0127] Step 317b (Non-Exempt Status): Today's penalty is only added to the Accounts Receivable field. That is: New Accounts Receivable = Original Accounts Receivable + Today's Penalty, the exempted amount remains unchanged. The user's actual penalty payable continues to increase with each daily increment.

[0128] Step 318: Tail Difference Collection

[0129] Calculate the rounding difference of today's increment to the nearest cent and record it in the rounding difference collection table.

[0130] Step 319: Optimistic Lock Update (including Exponential Backoff Retry)

[0131] The system uses UPDATE statements to write the new receivable amount and exempted amount to the database. The key concurrency control mechanism lies in the WHERE clause of the UPDATE statement, which, in addition to matching the bill number and merchant number, also includes two "original value" fields: WHERE receivable_amt = original receivable amount AND exempt_st = original exempted status. This means that the update will only succeed if the current value in the database is exactly the same as the value read during this calculation.

[0132] If the number of rows affected by the UPDATE is greater than 0, the update is successful. If the number of rows affected by the UPDATE is 0, a concurrency conflict has occurred (optimistic locking failed), and the system uses an exponential backoff retry strategy. First retry: Wait 1 second, then reread the latest data and retry; Second retry: Wait 3 seconds, then reread the latest data and retry; Third retry: Wait 7 seconds, then reread the latest data and retry.

[0133] The backoff interval formula is: waitTime = baseDelay × (2^retryCount - 1) + randomJitter, where baseDelay is the base delay (1 second), and randomJitter is a random jitter of 0-500ms to avoid database traffic avalanche caused by multi-node synchronization retries. After the maximum number of retries is exceeded, the record is marked as "failed", and the conflict details are recorded in the exception log for subsequent manual or compensation task handling.

[0134] Transaction boundaries and evidence storage failure compensation strategies

[0135] The present invention defines the transaction boundary for processing a single bill in stage S3 as follows: Business Transaction (Strong Consistency Transaction): The optimistic locking UPDATE operation on the business database (step 319) and the update of the record processing status to "success" (step 321) are executed within the same local database transaction. That is, the UPDATE of the penalty record and the UPDATE of the status field in the batch processing details table are committed within the same transaction, ensuring atomic consistency between business data and processing status. If the transaction fails to commit, it is rolled back entirely, and the record status remains either "processing" or "failed," allowing for precise continuation during subsequent retries.

[0136] Evidence writing (eventual consistency, asynchronous compensation): Chained evidence writing (steps 320 / S4) is not executed within the aforementioned business transactions. Instead, it adopts an eventual consistency strategy of "business transaction commit first, evidence writing later." Specifically: After a business transaction is successfully committed, the system synchronously attempts to write the evidence record. If the evidence writing is successful, the record is processed. If the evidence writing fails (e.g., the evidence storage layer is temporarily unavailable), the system does not roll back the committed business transaction. Instead, it records the evidence writing request in the local compensation queue (compensation_queue), and the background compensation thread asynchronously retryes it according to an exponential backoff strategy (retry intervals are 5s, 15s, and 45s, with a maximum of 5 retries). After a successful compensation retry, the evidence record is normally written to the sub-chain. If the maximum number of retries is exceeded and the process still fails, an alarm is triggered and recorded in the anomaly log, requiring manual intervention from operations and maintenance personnel.

[0137] The design considerations are as follows: the evidence storage layer is independent of the business database. If evidence storage is included in business transactions, the availability of the evidence storage layer will directly affect the throughput of business computation, which does not conform to the design principle that "evidence storage is an auxiliary compliance means and should not block core business processes." Through an asynchronous compensation mechanism, the integrity of the evidence storage records is ultimately achieved while ensuring the continuity of business processing.

[0138] Transaction boundaries for the initial INSERT path (step 313): In the initial calculation path, the INSERT of the initial penalty record and the state writing of the batch processing details table are executed in the same transaction. Upon successful INSERT, the evidence storage write is triggered synchronously; the compensation strategy for evidence storage failure is the same as described above.

[0139] Step 320: Chained evidence writing (triggering phase S4, see details) Figure 5 )

[0140] After a successful update, the core summary of this incremental change is written into the chained evidence storage structure. The summary includes: bill number, amount receivable before the change, amount receivable after the change, amount exempted before the change, amount exempted after the change, batch transaction date, batch serial number, and work node number. The summary data is hashed using SHA-256 and linked with the hash value of the previous evidence storage record to form an immutable evidence storage chain.

[0141] Step 321: Update the record processing status

[0142] Update the status of this record in the batch processing details table to "success".

[0143] Step 322: This bill has been processed and the process is complete.

[0144] Regardless of whether the initial path, incremental path, or backtracking correction path was used, the incremental calculation and persistence operations for this invoice have been completed, and the process has returned. Figure 3 The loop continues processing the next bill.

[0145] In a preferred embodiment, Figure 4 The text further details the sub-processes of dual-dimensional idempotency protection and precise continuation of operation. These sub-processes are... Figure 3 The detailed expansion of step 304, "two-dimensional idempotency verification," is not an independent method step, but rather a guarantee mechanism embedded in stage S3.

[0146] like Figure 4 As shown, the idempotency guarantee mechanism of this invention adopts a two-dimensional design: 1. First Dimension: Batch Business Date (batchBizDate). This date is generated by the scheduling center when the task starts, and its value is the planned execution date (not the physical execution date). For example, a batch scheduled to execute at midnight on June 12th will still have a batchBizDate of June 12th even if it doesn't actually complete until 1:00 AM on June 13th. This design eliminates the idempotency blind spot when running batches across midnight.

[0147] 2. Second Dimension: Globally Unique Sequence Number (batchSeqNo). Each task execution (including retries and backtracking) is assigned a globally unique sequence number. Even if a task needs to be re-executed on the same day (e.g., rerunning after a partial failure in the initial execution), the new execution uses a different sequence number, allowing for the tracing of the complete record of each execution.

[0148] Record-level processing status flags: The batch processing details table maintains a status field (process_status) for each processed record, with values ​​of: processing (0), successful (1), failed (2), skipped (3). When the task is interrupted and then re-executed: Records with a status of "success" are skipped and not processed repeatedly. Records with a status of "failed": reprocess; Records with a status of "Processing" are determined to be remnants from the previous interruption, and are reset to "Failed" before being processed again. Records with a status of "Skip": Keep skipping them.

[0149] This mechanism allows for precise skipping of successfully processed records during retries, reprocessing only failed and interrupted records, thus enabling precise continuation of the process within a batch.

[0150] Phase S4: Incremental change chained evidence storage.

[0151] exist Figure 5 The diagram shows the structure of the incremental change chain evidence storage mechanism, which demonstrates the specific execution process of the incremental change chain evidence storage stage S4: performing a hash operation on the core field summary of the incremental change, linking it with the hash value of the previous evidence storage record to form a chain structure, and writing it into the evidence storage layer independent of the business database.

[0152] Specifically, in step S4-1, while each record is written to the business database, the core field summary of this incremental change is hashed (e.g., SHA-256), linked with the hash value of the previous evidence record, and then written into a chain-like evidence storage structure independent of the business database, thereby achieving the immutability of incremental modification records and meeting financial audit and compliance requirements.

[0153] The core fields for evidence storage of each incremental change record may include the following fields: bill identifier (billNo), amount receivable before and after the change (before_recv_amt and after_recv_amt), waiver amount before and after the change (before_waiver_amt and after_waiver_amt), batch identifier (batchBizDate + batchSeqNo), work node number (nodeId), timestamp (ts), and hash value of the previous evidence storage record (prevHash).

[0154] The example hash calculation formula for the stored hash value (currentHash) is as follows: currentHash = SHA256(billNo + oldAmt + newAmt + batchBizDate +batchSeqNo + nodeId + ts + prevHash).

[0155] After each evidence record is written, its hash value becomes the prevHash input for the next evidence record, thus forming a chain structure. The prevHash of the first evidence record is either all zeros or the system genesis hash.

[0156] The evidence storage data is stored independently of the business database (it can be written to an independent evidence storage database or file system) and archived in batches. This design ensures that even if the business database is tampered with, the evidence storage chain can provide tamper-proof change audit evidence, meeting the compliance requirements of highly regulated scenarios such as finance and public utilities.

[0157] Phase S5: Batch closing and reconciliation of outstanding balances.

[0158] After traversing all levels, the system summarizes the total difference records for this batch and compares them with the theoretical total amount. Any discrepancies are automatically balanced through a fallback ledger. The system records the execution statistics for this task, cleans up the cursor's persistent records, and writes the batch execution summary to the chained evidence storage layer, completing the entire process for this periodic cost increment processing.

[0159] Specifically, in step S5-1, after all levels have been traversed, the system summarizes the total tail difference collection records for this batch and compares them with the theoretical total amount. The difference is automatically balanced through the bottom-line ledger.

[0160] In step S5-2, the execution statistics of this cost calculation task are recorded, the cursor persistent records are cleaned up, and the batch execution summary is written to the chained evidence storage layer, thus completing the entire process of this periodic cost increment processing.

[0161] The five stages described above constitute the complete execution chain of the method of this invention. Stages S1-S2 address the problem of "how to efficiently and completely traverse massive hierarchical data"; stage S3 addresses the problem of "how to accurately and securely calculate and write the increment of each record"; and stages S4 and S5 address the problem of "how to ensure the compliance, auditability, and financial accuracy of the calculation results". The stages are strictly sequential and progressive: the output of S1 (the data subset after fragmentation) is the input of S2, the output of S2 (the traversed single invoice) is the input of S3, each successful write in S3 synchronously triggers S4, and S5 is executed only after all stages of S3 are completed.

[0162] It should be understood that among the aforementioned stages, the two-level data fragmentation and collaboration mechanism in stages S1 and S2 is the core improvement of this application. The other strategies / stages are auxiliary innovations proposed to solve various existing technical problems, and are not necessarily all included in the solution of this application. Those skilled in the art can make adaptive adjustments according to actual needs.

[0163] Having understood the main stages and steps of the hierarchical progressive periodic cost increment processing method based on fragmented collaboration in this application, the following section combines... Figure 6 To further illustrate an example structure of a hierarchical progressive periodic cost incremental processing system based on fragmented collaboration, this application is presented.

[0164] As shown, the overall system architecture of this application mainly consists of four parts: a distributed scheduling center, multiple worker nodes, a shared persistent storage layer, and a chained evidence storage layer. In terms of overall design, the distributed scheduling center allocates the data and scheduling parameters of the cost calculation task to multiple worker nodes using a deterministic mapping function sharding strategy. Subsequently, each worker node traverses the data in a multi-level progressive structure. During the traversal, a two-level data sharding collaboration mechanism, including a first-level sharding at the node level and a second-level nested sharding within the merchant level, can be adopted. Next, after the data undergoes processing using strategies such as hot / cold data stratified differentiated scheduling and master-slave database routing, the calculation results are written to the shared persistent storage layer by the incremental calculation engine of each worker node and simultaneously chained evidence storage is performed. Finally, batch cleanup is performed; that is, after all worker nodes have completed all traversal and calculation of their respective shards, the distributed scheduling center uniformly executes the tail difference summary and reconciliation, statistical information recording, and batch summary evidence storage.

[0165] Specifically, the distributed scheduling center is configured to initiate the cost calculation task at a preset trigger time (e.g., 1:00 AM daily), and after configuring the scheduling parameters, distribute the task trigger signal, including the cost calculation task and scheduling parameters, to each working node (stage S1). Furthermore, the distributed scheduling center is configured to perform batch closing and tail-end reconciliation after all levels have been traversed (stage S5).

[0166] After receiving the task trigger signal, the worker node traverses the data of the fee calculation task according to a multi-level progressive structure (stage S2). During the traversal, a two-level data sharding collaborative mechanism can be adopted, including a first-level sharding at the node level and a second-level nested sharding within the merchant (only executed when there are many merchants). The incremental calculation engine of the worker node also performs single-bill incremental calculation and persistence (stage S3), that is, for each overdue bill traversed, it performs operations such as idempotency verification, pattern judgment (first / incremental / backtracking), incremental calculation, tail difference aggregation, and optimistic locking update, and then writes the calculation result into the shared persistent memory (layer).

[0167] The shared persistent memory is configured to persistently store the calculation results output by the worker nodes. At the same time, optimistic locking mechanism (with exponential backoff retries), two-dimensional idempotent control strategy, record-level processing status markers and batch tracking identifiers are used to ensure data consistency and traceability when multiple worker nodes write concurrently.

[0168] The chain-like evidence storage is configured to synchronously write the summary of each incremental change (stage S4) together with the shared persistent storage. Specifically, the chain-like evidence storage hashes the core field summary of the change (values ​​before and after the change, batch identifier, node number, timestamp), links it with the hash value of the previous evidence record, and writes it into a chain-like evidence structure independent of the business database. This allows the evidence data to be stored independently of the business database, forming an immutable audit evidence chain.

[0169] Therefore, by using the technical solution disclosed in this application, at least the following advantages can be achieved: 1) A SQL-layer uncoordinated data sharding strategy based on deterministic mapping functions is proposed. The full amount of data to be processed is distributed to different computing nodes according to the modulo of the total number of shards based on the primary key identifier, so as to achieve decentralized parallel processing. Each node can complete its own data sharding without communication and coordination. Furthermore, a two-level nested sharding mechanism within a merchant is proposed. On the basis of node-level sharding, a secondary load balancing is performed on the large amount of data within a single merchant according to the time dimension to prevent a single shard from becoming a computing bottleneck due to the large amount of data of the top merchants. 2) A multi-level progressive cursor traversal architecture was designed, which delves deeper layer by layer according to the parent-child hierarchical relationship between business entities (such as "merchant - payment item - overdue bill"). Each layer adopts cursor pagination based on primary key ID (id > lastId) instead of traditional OFFSET pagination, which effectively avoids the performance degradation problem in deep pagination scenarios. At the same time, a master-slave routing strategy is introduced, with the outermost traversal going through the slave database and the innermost traversal going through the master database, eliminating the risk of cursor missed reads caused by master-slave latency. 3) A periodic fee incremental calculation model is proposed, which distinguishes between the initial calculation and subsequent incremental calculation scenarios, and automatically switches modes without manual intervention based on the existence of historical records: in the initial calculation, the historical default fee is accumulated day by day from the overdue date, and in subsequent days, only one day's incremental amount is calculated; further, the time retrospective incremental correction capability is proposed, which supports the retrospective recalculation of historical batches caused by policy changes or data error correction, without polluting the daily incremental data stream; 4) An optimistic locking concurrency control mechanism based on the comparison of the original values ​​of business fields is introduced, which does not require the introduction of an additional version number field. The original receivable amount value and exemption status value before the update are carried in the WHERE condition of the UPDATE statement to ensure the consistency of data modification in a concurrent environment, while avoiding the performance bottleneck caused by pessimistic locking; Furthermore, an exponential backoff retry strategy is introduced to prevent database traffic avalanche caused by multi-node synchronous retries. 5) A dual-dimensional idempotent guarantee mechanism based on batch business date and globally unique serial number was designed to replace the solution that simply relies on the midnight timestamp of the natural day. It fully covers production scenarios such as batch running across midnight, abnormal retries, and manual backtracking and rerunning. At the same time, record-level processing status marking is introduced to achieve accurate continuation of running within a batch instead of rerunning the entire batch. 6) A financial-grade batch rounding and reconciliation mechanism was proposed, which collects the precision differences generated by rounding each record into a bottom-line ledger and automatically reconciles the ledgers daily to meet the financial audit compliance requirements. 7) A lightweight chain-based evidence storage scheme for incremental computation logs is proposed, which only puts the core field summary of daily incremental changes on the chain, balancing compliance and storage costs, and ensuring that incremental modification records are tamper-proof.

[0170] 8) Synergistic effect of the combination of various technical points

[0171] It should be particularly noted that the aforementioned core improvements of this invention are not simply the sum of their individual functions, but rather, through organic combination, they generate an overall synergistic effect that transcends the sum of their parts. This is specifically reflected in the following aspects: a) Synergistic Effect 1: Cursor Pagination + Master-Slave Routing → Collaborative Guarantee to Eliminate Missed Reads. While standalone cursor pagination (id > lastId) solves the performance problem of OFFSET pagination, in scenarios with master-slave replication latency, if traversal queries go through the slave database, cursor missed reads may still occur because the slave database data lags behind the master database (i.e., a record has not yet been synchronized to the slave database when the cursor advances, and after synchronization, its ID is less than lastId and it is permanently skipped). The standalone "all through master database" strategy can avoid missed reads, but it loses the read expansion capability of the slave database. This invention combines cursor pagination with hierarchical master-slave routing—the outermost traversal goes through the slave database (outer data changes less frequently, so the master-slave latency impact is negligible), and the innermost traversal goes through the master database (inner data is directly used for optimistic locking comparison and writing, and the latest value must be read)—ensuring zero missed reads while fully utilizing the slave database to share the read pressure; both are indispensable.

[0172] b) Synergistic Effect Two: Dual-Dimensional Idempotency + Record-Level Status Marking → Precise Resumption of Processing Guarantee. Individual batch-level idempotency (based on batchBizDate to determine processing status) can only achieve coarse-grained control of "skip the entire batch" or "rerun the entire batch," failing to handle mixed states where some records within a batch succeed and others fail. While individual record-level status marking can identify the processing result of each record, without batch-level idempotency determination, it may be misjudged as a "new batch" due to changes in physical dates in cross-zero-point batch processing scenarios, leading to duplicate processing. This invention combines dual-dimensional idempotency (logical business date + global serial number) with record-level status marking (processing / success / failure / skipped), eliminating the idempotency blind spot across zero-point and achieving precise resume capability down to the individual record within a batch. The two work together to form a complete idempotency guarantee system.

[0173] c) Synergistic Effect 3: MOD Sharding + Cursor Persistence → Coordinated Guarantee for Resuming from Breakpoints Without Coordination. While standalone MOD sharding achieves non-overlapping data allocation, if a task needs to resume after an interruption, the lack of cursor persistence necessitates traversing the entire shard from the beginning. Standalone cursor persistence records the traversal progress, but if the sharding strategy is uncertain (e.g., dynamic sharding at the application layer), recovery may fail due to changes in shard mapping after an interruption. This invention combines deterministic MOD sharding (where shard ownership is always determined given row_id and N) with cursor persistence, ensuring that upon restarting after a task interruption, the same node queries the same subset of data using the same MOD conditions, and then accurately resumes to the interruption point based on the persisted cursor position, without any cross-node coordination.

[0174] d) Synergistic Effect Four: Optimistic Locking + Exponential Backoff + Two-Dimensional Idempotency → Synergistic Guarantee for Concurrency Security. Optimistic locking alone can detect concurrent conflicts, but synchronous retries after a conflict may trigger a traffic avalanche. Exponential backoff alone can mitigate retry storms, but without idempotency guarantees, duplicate writes may occur during retries. This invention combines optimistic locking based on the original value of business fields (conflict detection), exponential backoff retries (conflict mitigation), and two-dimensional idempotency (preventing duplication) to form a complete concurrency security chain of "detection → backoff → duplication prevention".

[0175] Based on the aforementioned advantages of this application, it can be widely applied in the following technical fields and product scenarios: 1) Property Management: This invention enables automated batch calculation of overdue penalties (late fees) for property management fees, allowing property management companies to accumulate daily penalties for a large number of overdue bills across their jurisdiction. The incremental calculation model is applicable to various recurring charges such as property management fees, parking management fees, garbage disposal fees, and public maintenance funds.

[0176] 2) Public Utility Sector: Calculation of overdue surcharges for public utility fees such as water, electricity, gas, and heating. These sectors typically have characteristics such as a large user base, fixed billing cycles, and overdue fees accruing daily, which highly match the technical features of this invention.

[0177] 3) Financial lending sector: Batch calculation of interest on microloans, consumer installment payments, and credit card overdue payments. This is particularly relevant for financial scenarios that require support for both simple and compound interest calculation methods, large-scale parallel processing, and guaranteed data consistency.

[0178] 4) SaaS subscription service field: Scenarios such as overdue payment penalty calculation for enterprise-level SaaS products, accumulation of membership expiration fees, decay of membership rights due to overdue payments, and daily accumulation of account violation points.

[0179] 5) Government-enterprise ledger and supervision: Periodic numerical tracking scenarios related to government governance, such as deduction of points for grid workers' overdue rectification of hidden dangers, late payment fees for social security, cumulative days of land idling, and fines for daily increases in pollution discharge.

[0180] 6) Warehousing and supply chain sector: Supply chain periodic costs or credit scoring scenarios such as overdue storage fees for goods, deduction of credit score for suppliers with overdue performance, and daily accumulation of interest on payment terms.

[0181] 7) General-purpose periodic numerical incremental processing platform: Any distributed batch processing system that needs to perform incremental numerical calculations and accumulations on a large number of structured master-slave hierarchical data items at fixed time periods, including but not limited to rent increment calculation, contract liquidation penalty accumulation, insurance late payment penalty calculation, and daily incremental deduction of cloud resource excess computing power.

[0182] Although the techniques have been described using language specific to structural features and / or methodological actions, it should be understood that the techniques are not necessarily limited to the described features or actions. Rather, these features and actions are described as illustrative examples of how these techniques can be implemented.

[0183] The operations of the example processes are shown in separate boxes and are summarized with reference to these boxes. These processes are shown as a flow of logical boxes, each of which may represent one or more operations that can be implemented using hardware, software, or a combination thereof. In the context of software, these operations represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, cause one or more processors to perform a given operation. Generally, computer-executable instructions include routines, programs, objects, modules, components, data structures, etc., that perform a particular function or implement a particular abstract data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the operations may be executed in any order, combined in any order, subdivided into multiple sub-operations, and / or executed in parallel to implement the described process. The described process may be executed by resources associated with one or more computing devices, such as one or more internal or external CPUs or GPUs, and / or one or more pieces of hardware logic, such as FPGAs, DSPs, or other types of accelerators.

[0184] All of the methods and processes described above can be embodied in software code modules executed by one or more general-purpose computers or processors, and can be fully automated via these software code modules. These code modules can be stored on any type of computer-executable storage medium or other computer storage device. This code can also be packaged into corresponding computer program products. Some or all of these methods can alternatively be embodied in dedicated computer hardware.

[0185] Any routine description, element, or box in the flowcharts described herein and / or in the accompanying drawings should be understood as potentially representing a module, segment, or portion of code comprising one or more executable instructions for implementing a specific logical function or element in that routine. Alternative implementations are included within the scope of the examples described herein, wherein elements or functions may be removed or performed inconsistently with the order shown or discussed, including substantially synchronous or reverse order execution, depending on the functionality involved, as will be understood by those skilled in the art.

[0186] While different embodiments have been described above, it should be understood that they are merely examples and not limitations. Those skilled in the art will appreciate that various modifications in form and detail may be made without departing from the spirit and scope of the invention as defined herein. Therefore, the breadth and scope of the invention disclosed herein should not be limited to the exemplary embodiments disclosed above.

Claims

1. A hierarchical, progressive, periodic cost increment processing method based on fragmented collaboration, comprising: Start the cost calculation task and configure the scheduling parameters, and distribute the task trigger signal containing the cost calculation task and the scheduling parameters to each working node; The data of the cost calculation task is traversed in a multi-level progressive structure, wherein a two-level sharding mechanism is enabled during the data traversal process to achieve data sharding collaboration. Perform incremental calculation and persistence of single-invoice expenses, and simultaneously perform incremental change chaining; and After all levels have been traversed, batch closing and reconciliation are performed.

2. The hierarchical progressive periodic cost increment processing method as described in claim 1, characterized in that, The configuration scheduling parameters include: Generate batch identification information corresponding to the cost calculation task, including: batch business date batchBizDate and globally unique batch serial number batchSeqNo; Set the total number of shards N and the shard number k for the cost calculation task.

3. The hierarchical progressive periodic cost increment processing method as described in claim 2, characterized in that, The two-level sharding mechanism includes: Node-level first-level sharding: WHERE MOD(merchant.row_id, N) = k; When the number of pending records under a single merchant exceeds a preset threshold, enable secondary sharding within the merchant: WHEREbill.due_date BETWEEN rangeStart AND rangeEnd; Here, merchant.row_id is the primary key ID of the merchant table, bill.due_date is the bill due date, rangeStart is the start time range, and rangeEnd is the end time range.

4. The hierarchical progressive periodic cost increment processing method as described in claim 2, characterized in that, The multi-level progressive structure traversal includes: The first level iterates through the merchant collection; The second layer iterates through the collection of payment items under the current merchant; The third layer iterates through the collection of overdue bills under the current payment item.

5. The hierarchical progressive periodic cost increment processing method as described in claim 1, characterized in that, During the data traversal process, at least one of the following strategies is executed: hot and cold data hierarchical scheduling strategy, cursor persistence strategy, master-slave routing strategy, and local hotspot configuration caching strategy.

6. The hierarchical progressive periodic cost increment processing method as described in claim 1, characterized in that, The incremental calculation and persistence of single bill expenses are ensured by using an optimistic locking mechanism in conjunction with exponential backoff retries, a two-dimensional idempotent control strategy, record-level processing status markers, and batch tracking identifiers to guarantee data consistency and traceability when multiple working nodes write concurrently.

7. The hierarchical progressive periodic cost increment processing method as described in claim 1, characterized in that, The incremental change chain storage includes: performing a hash operation on the core field summary of the incremental change, linking it with the hash value of the previous storage record to form a chain structure, and writing it into a storage layer independent of the business database.

8. The hierarchical progressive periodic cost increment processing method as described in claim 1, characterized in that, The batch closing and balance settlement includes: after all levels have been traversed, summarizing the full balance records of this batch, comparing them with the theoretical total amount, and automatically settling the difference through the fallback ledger.

9. A hierarchical, progressive, periodic incremental cost processing system based on fragmented collaboration, comprising: A distributed scheduling center is configured to initiate a cost calculation task, configure scheduling parameters, and distribute a task trigger signal containing the cost calculation task and the scheduling parameters to each working node. Multiple worker nodes are configured to traverse the data of the cost calculation task in a multi-level progressive structure, wherein during the data traversal process, a two-level sharding mechanism is enabled to achieve data sharding collaboration and to perform single bill cost incremental calculation and persistence. A shared persistent memory is configured to persistently store the computation results output by the worker node; and A chain-linked evidence storage system is configured to perform incremental change chain-linked evidence storage. The distributed scheduling center is also configured to perform batch closing and tail reconciliation after all levels have been traversed.

Images