Hotspot account processing method and system
By combining the sliding window counting model and the TCC flexible transaction framework with CDC technology, the performance crash and data consistency issues of the payment system in high-concurrency scenarios are solved, achieving stability and efficient processing of the high-concurrency payment system.
Patent Information
- Application Number
- CN202511708522.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-20
- Publication Date
- 2026-02-17
AI Technical Summary
Existing payment systems cannot dynamically adapt to sudden traffic surges in high-concurrency scenarios, leading to system performance crashes and loss of data consistency control. Static sharding and strong consistency transaction solutions cannot effectively cope with the pressure from hot accounts.
A sliding window counting model is used to identify hot accounts. Combined with the TCC flexible transaction framework and CDC technology, asynchronous processing and second-level sharding scaling are achieved through distributed aggregation and intelligent sharding migration, reducing database pressure and ensuring data consistency.
It significantly improves the throughput and stability of high-concurrency payment systems, reduces transaction failure rates and hardware costs, and increases user-side payment success rates and system capacity.
Smart Images
Figure CN121544252A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of hotspot account processing method and system, in particular, to a hotspot account processing method and system. BACKGROUND
[0002] At present, the mainstream scheme of payment system processing high concurrency account operation generally adopts static database and table splitting and strong consistency transaction mechanism. Under this technical mode, the system pre-splits the account data according to the user ID hash value to a fixed number of database nodes, and guarantees the atomicity of cross-node operation through two-phase commit protocol and the like. This combination of technologies can maintain basic operation in a conventional traffic scenario, but does not dynamically adapt to the pressure of hotspot accounts caused by sudden traffic. When a single account bears far more than the design threshold of concurrent operation (such as balance deduction in a flash sale activity) in a short time, the system still mechanically executes the static sharding rule and the strong transaction process.
[0003] The core defect of static sharding and strong transaction scheme lies in that its rigid architecture design causes the system to be unable to dynamically adapt to sudden traffic. When a certain account encounters a high concurrency operation (such as balance deduction in a red envelope rain) in a short time, the request is still mechanically routed to the fixed database node, causing catastrophic row lock competition and service delay soaring; at the same time, the multi-round cross-node communication of strong consistency transaction makes the transaction timeout rate high, which is easy to cause large-scale payment failure. This structural mismatch between static rules and dynamic traffic eventually leads to performance collapse and data consistency double out of control of the system in a high concurrency scenario. SUMMARY
[0004] One of the purposes of the present application is to propose a hotspot account processing method, which can greatly reduce the database pressure, replace the strong consistency transaction model, and realize second-level sharding expansion and contraction.
[0005] The technical solution of the present application is as follows: a hotspot account processing method, comprising the following steps: S100: an account initiates a payment request; S200: the API gateway performs authentication, flow limiting and pre-operation on the request; S300: hot account identification is performed through a sliding window counting model; S400: branch processing is performed according to the identification result and routing table information; S500: if the user is identified as a non-hotspot user, the request is routed to the account master database for payment processing; if the user is identified as a hotspot user, asynchronous processing is performed through a TCC flexible transaction framework and a request aggregation mechanism; S600: while performing the payment operation, the system completes the persistence, synchronization and migration between shards of data through CDC and an intelligent sharding migrator. S700: Complete the payment and ensure eventual data consistency.
[0006] Furthermore, the identification logic of the sliding window counting model in step S300 is as follows: ; in, For the duration of the statistical window; Hotspot threshold; Indicates the first The number of requests per moment.
[0007] Furthermore, the hotspot request processing and asynchronous steps in step S500 include: S510: Request Entry and Aggregation: Aggregates multiple requests for the same popular account within a preset time window into a batch request; S520: Aggregate Commit: When a preset time or quantity threshold is reached, batch requests are committed to the transaction processing core. S530: Transaction Execution and Routing Table: Based on the TCC flexible transaction framework, batch requests are sent to the hotspot shard or account master database where the account is located for final processing according to the routing table information of the account.
[0008] Furthermore, the hotspot sharding in step S530 is an independent database sharding set up for extremely hot accounts to isolate high-concurrency traffic; The system synchronizes data changes between shards in real time through a Change Data Capture (CDC) mechanism; When a TCC flexible transaction or data synchronization process fails, the system initiates an automatic compensation operation based on the CDC log and transaction records to ensure eventual data consistency. By monitoring system traffic and CDC logs through an intelligent shard migration tool, hot account data is dynamically migrated between the account master database and hot shards.
[0009] Furthermore, the request aggregation algorithm in step S510 is as follows: Aggregate net value ; in, This is the increment for a single operation; n is the number of requests within the window.
[0010] Furthermore, the window time for requesting entry in step S510 is 10 milliseconds.
[0011] Furthermore, the CDC data synchronization process in step S600 includes: S611: Data capture: The source shard records data changes to the MySQLBinlog in real time; S612: Log parsing: The CDC parser continuously listens to and parses the binary logs of the source shard, generating incremental data streams; S613: Data Synchronization: Apply the parsed incremental data to the target shard in real time as a stable data stream; S614: Traffic Switching: Once it is confirmed that the target shard and the source shard data are consistent, pause the source shard writing and switch the business request to the target shard, while updating the data routing table information; S615: Synchronization complete: The target shard begins to independently handle all subsequent read and write requests.
[0012] Another objective of this invention is to provide a hotspot account handling system for implementing a hotspot account handling method, comprising: API Gateway Module: Used to receive payment requests and perform pre-processing; Hotspot identification module: used to identify hot accounts in real time based on a sliding window counting model; Request processing module: Used for distributed aggregation and asynchronous processing of requests from frequently accessed accounts; Transaction processing module: Used to manage distributed transactions based on the TCC flexible transaction framework to ensure data consistency; Data sharding module: includes the account master database and hotspot shards, used for data storage; Data synchronization module: Used to achieve data synchronization and migration between shards through CDC synchronization unit and intelligent shard migration.
[0013] The beneficial effects of this invention are as follows:
[0014] This invention innovatively integrates distributed aggregation, flexible transactions, and intelligent sharding migration to construct an adaptive high-concurrency processing system, solving three core problems in high-concurrency payment scenarios. It significantly reduces database pressure through real-time aggregated write operations; replaces the strong consistency transaction model with lightweight compensating transactions (which only record the operation flow); and achieves second-level sharding scaling based on CDC (without request awareness during migration).
[0015] Implement a three-tiered collaborative control system in the payment system, which combines high-frequency operations, asynchronous transactions, and elastic resource scheduling.
[0016] The distributed aggregation engine targets short-term, high-frequency operations on hot accounts, merging multiple requests into a single batch write operation within a time window, significantly reducing database write pressure. The flexible transaction framework employs an "asynchronous execution + compensation guarantee" mechanism. Core operations are committed locally, generating sequential snapshots to avoid row lock resource contention. In case of failure, a reverse SQL rollback is triggered based on the snapshot, ensuring eventual consistency while significantly improving throughput. The intelligent sharding migration system automatically identifies hot accounts based on real-time traffic monitoring and achieves second-level data migration through CDC technology, isolating hot accounts to independent high-performance shards and completely eliminating single-point bottlenecks. Attached Figure Description
[0017] Fig. 1 This is a flowchart illustrating the overall process of handling hot accounts based on distributed aggregation and flexible transactions in this invention. Fig. 2 This is a flowchart of the CDC data synchronization process. Detailed Implementation
[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0019] Example 1
[0020] like Figs. 1-2 As shown, a hot account handling method, through the synergy of distributed aggregation, flexible transactions, and intelligent sharding migration, brings a triple breakthrough improvement to high-concurrency payment systems: Leapfrog optimization of performance metrics: the throughput of hot accounts has been increased from 500 TPS in the traditional solution to 5,000+ TPS, and a single account can support tens of thousands of concurrent operations per second; the user-side deduction response speed has reached the "seamless" level; the transaction inconsistency rate has been reduced from 1% to <0.1%, ensuring zero errors in funds.
[0021] The system's stability is significantly enhanced: dynamic sharding migration reduces the time for capacity expansion to take effect from hours to ≤3 seconds, and availability reaches 99.99% during the campaign; the flexible transaction framework reduces the transaction failure rate from 30% to <0.1%, and the fault recovery time is ≤1 second; resource utilization is improved by 40%, and hardware costs are reduced by 50% for the same business volume.
[0022] Win-win situation for user experience and business value: On the user side: the payment success rate reaches 99.9% in high-concurrency scenarios such as red envelope rain and flash sales, and the number of customer complaints decreases by 80%; On the enterprise side: the system's promotional activity carrying capacity is increased by 10 times, and annual maintenance costs are reduced by more than 70%.
[0023] The specific steps are as follows: S100: The account initiates a payment request; S200: The API gateway performs pre-processing operations such as request authentication and rate limiting; S300: Hotspot account identification using a sliding window counting model; S400: Performs branching processing based on the identification results and routing table information; S500: If identified as a non-hotspot user, the request will be routed to the account master database to complete the payment processing; If a user is identified as a hotspot, asynchronous processing will be performed using the TCC flexible transaction framework and request aggregation mechanism. The hotspot request processing and asynchronous steps in step S500 include: Specifically, S510: Request Inbound and Aggregation: Aggregates multiple requests to the same hot account within a preset time window into a batch request; S520: Aggregate Commit: When a preset time or quantity threshold is reached, batch requests are committed to the transaction processing core. S530: Transaction Execution and Routing Table: Based on the TCC flexible transaction framework, batch requests are sent to the hotspot shard or account master database where the account is located for final processing according to the routing table information of the account.
[0024] When a TCC flexible transaction fails in the Confirm phase, the system performs reverse logic on the already effective local operations, causing the data to return to a state where it is "business-equivalent to not having occurred".
[0025] Independent database shards are set up for accounts experiencing extreme high traffic, isolating their traffic from the main database and other shards to prevent them from dragging down the entire system.
[0026] The hotspot sharding in step S530 is an independent database sharding set up for extremely hot accounts to isolate high-concurrency traffic; The system synchronizes data changes between shards in real time through a Change Data Capture (CDC) mechanism; When a TCC flexible transaction or data synchronization process fails, the system initiates an automatic compensation operation based on the CDC log and transaction records to ensure eventual data consistency.
[0027] By monitoring system traffic and CDC logs through an intelligent shard migration tool, hot account data is dynamically migrated between the account master database and hot shards.
[0028] The request aggregation algorithm in step S510 is as follows: By merging operations within a time window into batch tasks, and aggregating net asset value per unit time, "100 row locks + 100 writes" is reduced to "1 row lock + 1 write." In scenarios with high-frequency accounts, this achieves a reduction of over 99% in write operations, lock contention approaches zero, and the cost of failure rollback remains constant, truly achieving "performance improvement and zero data distortion." ; in, This represents the increment for a single operation (positive numbers indicate entries, negative numbers indicate deductions). n is the number of requests within the window (e.g., 100 times).
[0029] Furthermore, the window time for requesting entry in step S510 is 10 milliseconds.
[0030] S600: While performing payment operations, the system uses CDC and intelligent shard migration to complete data persistence, synchronization and migration between shards; Specifically, the CDC data synchronization process in step S600 includes: S611: Data capture: The source shard records data changes to MySQLBinlog (binary log) in real time. S612: Log parsing: The CDC parser continuously listens to and parses the binary logs of the source shard, generating incremental data streams; S613: Data Synchronization: Apply the parsed incremental data to the target shard in real time as a stable data stream; S614: Traffic Switching: Once it is confirmed that the target shard and the source shard data are consistent, pause the source shard writing and switch the business request to the target shard, while updating the data routing table information; S615: Synchronization complete: The target shard begins to independently handle all subsequent read and write requests.
[0031] S700: Complete the payment and ensure eventual data consistency.
[0032] The recognition logic of the sliding window counting model in step S300 is as follows: ; in, Sets the duration of the statistics window (default 10 seconds). The hotspot threshold (e.g., 10,000 TPS). Indicates the first The number of requests per moment.
[0033] The above formula dynamically identifies hot accounts by calculating the moving average of request volume in real time. The sliding window counting model upgrades hotspot identification from a "coarse impulse response" to "precise real-time perception" through "second-level smoothing + continuous detection." Compared with conventional methods, it can identify hot accounts with low latency and high accuracy, reduce excessive expansion, and provide a more stable, faster, and more accurate decision-making basis for high-concurrency payment systems. This embodiment, besides its application in payment systems, also has broad technological expansion and application potential. The three-layer collaborative architecture of "distributed aggregation + flexible transactions + intelligent sharding" in this embodiment focuses on "window aggregation - asynchronous transactions - dynamic sharding" for any high-concurrency, single-line hot resources, thus extending beyond payment balance scenarios. For example, in e-commerce platforms, it is suitable for high-concurrency resource operation scenarios such as inventory deduction for flash sales and bulk coupon redemption during promotional periods. By replacing "hot accounts" with "hot SKUs," distributed aggregation technology merges hundreds of inventory deductions into a single batch update, avoiding overselling issues. In the financial sector, it can support businesses such as bank red envelope distribution and fund transfers during peak securities trading periods, with intelligent sharding migration ensuring account operation latency of less than 50 milliseconds under sudden traffic surges. In the gaming industry, it can be applied to scenarios such as virtual item flash sales and item synchronization in densely populated player areas, with the flexible transaction framework ensuring eventual consistency in item distribution.
[0034] Example 2
[0035] A hotspot account handling system, used to implement the hotspot account handling method in Embodiment 1, includes: API Gateway Module: Used to receive payment requests and perform pre-processing.
[0036] Hotspot identification module: Used to identify hot accounts in real time based on a sliding window counting model. Hot accounts are those that receive more concurrent access requests than the system's set threshold within a unit of time, causing database row lock contention or service bottlenecks due to their high-frequency operations. This embodiment specifically refers to balance / points accounts in the payment system.
[0037] Request processing module: Used for distributed aggregation and asynchronous processing of requests from popular accounts.
[0038] The aforementioned Distributed Aggregation is a technique that combines multiple operation requests from the same account within a time window into a single batch operation, thereby reducing the load by decreasing the number of database writes.
[0039] Transaction processing module: Used to manage distributed transactions based on the TCC flexible transaction framework, dividing transactions into a Try phase for resource reservation, a Confirm phase for execution confirmation, and a Cancel phase for asynchronous compensation. This asynchronous compensation mechanism ensures eventual data consistency and allows transactions to exist in intermediate states, thereby significantly reducing database lock contention and improving system performance in high-concurrency scenarios.
[0040] Data sharding module: includes logically or physically isolated account master databases and hotspot shards, used for persistent data storage.
[0041] Data synchronization module: Used to achieve data synchronization and migration between shards through CDC synchronization unit and intelligent shard migration.
[0042] The aforementioned CDC is a technology that captures and transmits incremental database changes in real time. By monitoring the database transaction log, it synchronizes data insertion, update, and deletion operations to the target system in the order they occur, achieving low-latency data consistency between the source and target databases.
[0043] The intelligent sharding migrater dynamically decides on data migration strategies based on system load or hotspot identification results, and instructs the CDC synchronization unit to execute a seamless synchronization process of data from the source shard to the target shard, thereby realizing dynamic sharding capabilities. It is based on the technology of automatically migrating account data to independent storage nodes based on real-time hotspot detection. The migration process achieves seamless data synchronization through CDC.
[0044] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for handling hot accounts, characterized in that, Includes the following steps: S100: The account initiates a payment request; S200: The API gateway performs pre-processing operations such as request authentication and rate limiting; S300: Hotspot account identification using a sliding window counting model; S400: Performs branching processing based on the identification results and routing table information; S500: If identified as a non-hotspot user, the request will be routed to the account master database to complete the payment processing; If a user is identified as a hotspot, asynchronous processing will be performed using the TCC flexible transaction framework and request aggregation mechanism. S600: While performing payment operations, the system uses CDC and intelligent shard migration to complete data persistence, synchronization and migration between shards; S700: Complete the payment and ensure eventual data consistency.
2. The method for handling hot accounts according to claim 1, characterized in that, The recognition logic of the sliding window counting model in step S300 is as follows: ; in, For the duration of the statistics window; Hotspot threshold; Indicates the first The number of requests per moment.
3. The method for handling hot accounts according to claim 2, characterized in that, The hotspot request processing and asynchronous steps in step S500 include: S510: Request Entry and Aggregation: Aggregates multiple requests for the same popular account within a preset time window into a batch request; S520: Aggregate Commit: When a preset time or quantity threshold is reached, batch requests are committed to the transaction processing core. S530: Transaction Execution and Routing Table: Based on the TCC flexible transaction framework, batch requests are sent to the hotspot shard or account master database where the account is located for final processing according to the routing table information of the account.
4. The method for handling hot accounts according to claim 3, characterized in that, The hotspot sharding in step S530 is an independent database sharding set up for extremely hot accounts to isolate high-concurrency traffic; The system synchronizes data changes between shards in real time by changing the data capture mechanism; When a TCC flexible transaction or data synchronization process fails, the system initiates an automatic compensation operation based on the CDC log and transaction records to ensure eventual data consistency. By monitoring system traffic and CDC logs through an intelligent shard migration tool, hot account data is dynamically migrated between the account master database and hot shards.
5. The hotspot account processing method according to claim 4, characterized in that, The request aggregation algorithm in step S510 is as follows: Aggregate net value ; in, This is the increment for a single operation; n is the number of requests within the window.
6. The hotspot account processing method according to claim 5, characterized in that, The window time for requesting entry in step S510 is 10 milliseconds.
7. A method for handling hot accounts according to claim 6, characterized in that, The CDC data synchronization process in step S600 includes: S611: Data capture: The source shard records data changes to the MySQLBinlog in real time; S612: Log parsing: The CDC parser continuously listens to and parses the binary logs of the source shard, generating incremental data streams; S613: Data Synchronization: Apply the parsed incremental data to the target shard in real time as a stable data stream; S614: Traffic Switching: Once it is confirmed that the target shard and the source shard data are consistent, pause the source shard writing and switch the business request to the target shard, while updating the data routing table information; S615: Synchronization complete: The target shard begins to independently handle all subsequent read and write requests.
8. A hotspot account processing system, characterized in that, Implementing the hotspot account processing method according to any one of claims 1 to 7, comprising: API Gateway Module: Used to receive payment requests and perform pre-processing; Hotspot identification module: used to identify hot accounts in real time based on a sliding window counting model; Request processing module: Used for distributed aggregation and asynchronous processing of requests from frequently accessed accounts; Transaction processing module: Used to manage distributed transactions based on the TCC flexible transaction framework to ensure data consistency; Data sharding module: includes the account master database and hotspot shards, used for data storage; Data synchronization module: Used to achieve data synchronization and migration between shards through CDC synchronization unit and intelligent shard migration.