A method and system for dynamic invoice data fragmentation and hot table synchronization based on seller tax number hash routing
By using a dynamic sharding method based on the seller's tax number hash routing and a hot table synchronization mechanism, the performance bottleneck and state consistency issues of invoice data in the financial and tax SaaS platform were resolved, enabling efficient data querying and expansion, and ensuring data consistency and stability.
Patent Information
- Application Number
- CN202611169846.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-04
- Publication Date
- 2026-11-17
AI Technical Summary
The centralized storage of invoice data in existing financial and tax SaaS platforms leads to performance bottlenecks, uneven load during querying and scaling, difficulty in ensuring state consistency, and easy occurrence of state rollback or missing fields during message processing.
A dynamic sharding method based on seller tax number hash routing is adopted. The seller tax number is used as a stable business routing key to centralize the invoice data of the same enterprise to a certain sharding resource. A hot table is used to store recent data and a sharding table is used to store historical data. Combined with a single-threaded queue and a compensation task mechanism, data consistency and sequential updates are ensured.
It enables efficient querying and expansion of invoice data, reduces cross-shard access, ensures eventual data consistency, reduces concurrency out-of-order and memory pressure, and avoids the risk of data interruption.
Smart Images

Figure CN123115271A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer data storage and data synchronization technology, and more specifically, to a method and system for dynamic fragmentation and hot table synchronization of invoice data based on seller tax number hash routing. Background Technology
[0002] In existing financial and tax SaaS platforms, basic invoice information, details, and high-frequency status transition records are typically stored centrally in relational databases. When the number of connected enterprises surges or during the month-end invoicing peak, the data volume in a single table can quickly reach a performance bottleneck. This data expansion not only leads to increased depth of the underlying B+ tree index and a dramatic increase in query scanning overhead, but also easily triggers severe concurrent write lock contention. Over time, the database backup and recovery window is extended indefinitely, directly causing severe delays in online services with high real-time requirements, such as front-end invoice reprinting and status polling.
[0003] Existing static sharding typically allocates data based on auto-incrementing identifiers, random identifiers, or time ranges. When sharding by auto-incrementing or random identifiers, invoices from the same company may be scattered across multiple shards, requiring cross-shard aggregation for company-level queries. When sharding by time ranges, historical data from the same company is scattered across different time periods, similarly increasing the overhead of routing decisions and query merging. While some consistent hashing schemes can reduce the migration scope during scaling, they do not consider the company's invoicing scale, tenant relationships, and available shard capacity for resource selection, which may still result in uneven load distribution across some shards.
[0004] Furthermore, invoice data exhibits significant temporal locality. Recent invoices are frequently used for result confirmation, file downloads, and status display, while historical invoices are primarily used for low-frequency retrieval, statistics, or auditing. If both recent and historical data are read from large-scale sharded tables, frequently used queries still require dynamic routing and large table indexes, making it difficult to fully leverage the concentrated access characteristics of recent data.
[0005] In a message-driven invoice processing architecture, invoices under the same seller's tax number may sequentially undergo state changes such as initialization, successful invoicing, signature completion, SMS notification, and email notification. If different state messages are executed concurrently by different threads, the later-generated state may be written first, while the earlier-generated state subsequently overwrites the new state, resulting in state rollback or missing fields. Relational databases and search indexes typically use asynchronous synchronization; network jitter, index service anomalies, or processing interruptions may cause inconsistencies between the main database and the search index. Summary of the Invention
[0006] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a method and system for dynamic fragmentation and hot table synchronization of invoice data based on seller tax number hash routing, in order to solve the problems mentioned in the background art.
[0007] To achieve the above objectives, the present invention provides the following technical solution:
[0008] Receive the status change message corresponding to the invoice data, and extract the seller's tax number from the status change message;
[0009] Based on the seller's tax number, the status change message is assigned to the sequential processing queue corresponding to that seller's tax number, so that status change messages with the same seller's tax number are executed by the same processing thread in the order of receipt.
[0010] Query the routing mapping information corresponding to the seller's tax number to obtain the target database instance and target shard table. If the routing mapping information does not exist, establish routing mapping information for the seller's tax number based on available shard resources.
[0011] Based on the invoice status data contained in the status change message, insert or update operations are performed on the unsharded hot table, and insert or update operations are performed on the sharded table based on the target database instance and the target sharded table.
[0012] After the sharded table is written, the complete invoice data is read and asynchronously synchronized to the search index. When the hot table write, sharded table write, or search index synchronization fails, a compensation task containing the invoice identifier, seller's tax number, routing mapping information, operation type, and failure reason is recorded, and the latest invoice data is reread according to the compensation task to perform the corresponding data operation.
[0013] Preferably, the querying of routing mapping information corresponding to the seller's tax number includes: firstly querying routing mapping information from the distributed cache using the seller's tax number as the query key; when the distributed cache is not found, querying routing mapping information from the routing mapping storage and writing the query result into the distributed cache; directly accessing the routing mapping storage when the distributed cache is unavailable; and actively updating the corresponding routing mapping information in the distributed cache after the routing migration is completed.
[0014] Preferably, establishing routing mapping information for the seller's tax ID includes: determining the capacity level based on the estimated invoice data size of the seller entity, specifically including: obtaining the average monthly invoice issuance quantity of the seller entity within a historical preset period (e.g., the past 12 months), and calculating the estimated invoice data size by combining it with the expected growth rate configured in the business front end; or, for newly registered seller entities without historical data, directly receiving the initial estimated invoice issuance quantity parameters manually configured by the external business system or enterprise user front end to determine the estimated invoice data size; selecting shard resources that meet the capacity level and have available capacity from the capacity usage information; obtaining a distributed lock corresponding to the selected shard resource; reducing the available capacity of the selected shard resource under the protection of the distributed lock, writing the routing mapping between the seller's tax ID and the selected shard resource, and caching the routing mapping.
[0015] Preferably, the invoice data includes the corresponding tenant identifier. When other sales tax numbers with routing mapping already exist under the same tenant, the available capacity of the shard resources corresponding to the other sales tax numbers is checked first. When the available capacity meets the requirements, the sales tax numbers to be allocated are allocated to the same shard resource. When a new database instance or a new shard table is added, the new resources are registered in the capacity usage information and the new resources are used for subsequent sales tax numbers that have not yet established routes. The routing mapping of existing sales tax numbers remains unchanged.
[0016] Furthermore, the system also includes a fragment capacity monitoring mechanism: periodically checking the remaining available capacity of each fragment resource; when it is detected that the remaining available capacity of the target fragment table corresponding to an existing seller's tax number is lower than a preset safety threshold (such as the total capacity), the system will detect when the remaining available capacity is lower than a preset safety threshold (such as the total capacity). When the daily average data increment of the seller's tax number exceeds the preset alarm value, the system automatically triggers the routing migration process, marks the seller's tax number as a migration preparation state, and reallocates new shard resources with sufficient capacity under the protection of distributed lock, thereby realizing the dynamic and smooth migration of invoice data of existing high-frequency invoicing enterprises.
[0017] Preferably, the hot table uses a default data source and does not perform sharding routing, and is used to store invoice data within a preset recent range; the sharded table stores complete historical invoice data; the hot table operation and the sharded table operation are triggered by the same state change message and are executed sequentially in the same processing thread. When either the hot table operation or the sharded table operation fails, a compensation task for the corresponding storage target is generated respectively.
[0018] Preferably, the process of allocating the queue according to the sales tax number includes: performing hash processing on the sales tax number, determining the target processing thread according to the hash result and the number of processing threads, and delivering the state change message to the bounded blocking queue corresponding to the target processing thread; each target processing thread retrieves a task from the corresponding bounded blocking queue in a single-threaded manner and executes it to maintain the data operation order of the same sales tax number, and blocks message delivery when the queue is full to form back pressure.
[0019] Preferably, the asynchronous synchronization to the search index includes: reading basic invoice information, detailed information, and extended information from the sharding table according to the routing mapping, and assembling them into a search index document; performing an overriding addition or update using the unique invoice identifier as the search index document identifier, and performing a repeatable deletion operation in the deletion scenario; writing a compensation task when synchronization fails, and the background task rereading the latest invoice data and performing synchronization according to the creation order of the compensation tasks, deleting the compensation task after success, and updating the failure reason and retry count when failure occurs.
[0020] Preferably, it also includes route migration and shutdown protection; the route migration includes adjusting the capacity usage information of the source shard resource and the target shard resource under lock protection, updating the route mapping and cache corresponding to the seller's tax number, so that subsequent invoice data is written to the target shard resource, and historical data is gradually migrated by offline tasks; the shutdown protection includes stopping the reception of new tasks, waiting for the confirmation of the completion of received messages, closing the processing thread, and waiting for the data operation in transit to be completed before exiting.
[0021] This invention also provides a dynamic invoice data sharding and hot table synchronization system based on seller tax number hash routing, comprising:
[0022] The system includes a message receiving unit, a thread sequence processing unit, a routing management unit, a hot table synchronization unit, a sharded storage unit, a search index synchronization unit, and a compensation processing unit.
[0023] The message receiving unit is used to receive invoice status change messages and extract the seller's tax number;
[0024] The thread sequence processing unit is used to assign messages with the same sales tax number to the same sequence processing queue.
[0025] The routing management unit is used to query or establish routing mappings between the seller's tax number and the database instance and sharded table;
[0026] The hot meter synchronization unit is used to perform invoice data operations on unsharded hot meters; the sharded storage unit is used to perform invoice data operations on target sharded tables according to routing mapping.
[0027] The search index synchronization unit is used to asynchronously synchronize the complete invoice data in the sharded table to the search index.
[0028] The compensation processing unit is used to record failed tasks and re-execute the corresponding data operations.
[0029] Preferably, the routing management unit includes a cache query component, a routing mapping component, a capacity management component, and a distributed lock component; the thread sequence processing unit includes multiple independent single-threaded execution components and bounded blocking queues corresponding to each single-threaded execution component; and the compensation processing unit is configured to distinguish between hot table compensation and search index compensation based on the target type in the compensation task, and to perform add / update compensation or delete compensation based on the operation type.
[0030] The technical effects and advantages of this invention are as follows:
[0031] This invention uses the seller's tax number as a stable business routing key to ensure that invoice data of the same seller entity is centralized to a specific shard resource, thereby significantly reducing cross-shard access when querying at the enterprise level. At the same time, by decoupling the business key from the physical shard through routing mapping and capacity usage information, the newly added shard resource can directly carry subsequent business after registration, and there is no need to redistribute all historical data when expanding capacity. Building upon this foundation, the system employs a hot table to store recent high-frequency data and a sharded table to store complete historical data. This allows frequently used invoice queries to directly access the hot table, which is kept under control, while preserving long-term traceability capabilities. The execution mechanism, which binds tax ID hashes to a single-threaded queue, effectively ensures the sequential updating of invoice statuses for the same seller entity, preventing out-of-order concurrency, and provides a foundation for parallel processing of multiple global tax IDs. To enhance eventual consistency, we introduce a compensation task mechanism that includes a failure context, coupled with stable document identifiers and idempotent operations, to ensure that data between the main database, hot table, and search index eventually converges to consistency. Furthermore, a bounded blocking queue is used to create backpressure, and in-transit tasks are waited for to complete during shutdowns, thereby reducing memory pressure caused by message backlog and avoiding the risk of data interruption due to process exits. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of the overall system architecture of the present invention.
[0033] Figure 2 This is a schematic diagram of the dynamic segmentation and heat meter synchronization method of the present invention.
[0034] Figure 3 This is a schematic diagram of the routing allocation and dynamic expansion process of the present invention.
[0035] Figure 4 This is a schematic diagram of the search index synchronization and failure compensation process of the present invention. Detailed Implementation
[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0037] Example 1
[0038] like Figure 1 As shown, the system includes an invoice business service layer, a message queue, a message receiving unit, a thread sequence processing unit, a routing management unit, a hot table synchronization unit, a sharded storage unit, a search index synchronization unit, and a compensation processing unit. The invoice business service layer may include invoicing services, signature services, document services, notification services, and status services. Each business service generates a status change message when invoice data is generated or its status changes, and sets the seller's tax number as the message key, ensuring that the message queue maintains the sequential relationship of messages with the same seller's tax number within the same partition.
[0039] Status change messages can be serialized using structured binary or structured text format. The message content must include at least a message identifier, invoice transaction identifier, seller's tax ID, tenant identifier, status type, transaction occurrence time, operation type, and status data. The message identifier is used for message deduplication, the invoice transaction identifier is used to locate database records and search index documents, and the transaction occurrence time can be used to detect obviously late messages.
[0040] The message receiving unit receives invoice status change messages and extracts the seller's tax number from the message key or message body. After successful verification, it determines the target processing thread. The thread order processing unit assigns messages with the same seller's tax number to the same sequential processing queue. It consists of multiple independent single-threaded execution components, each configured with a bounded blocking queue. Because the hash result of the same seller's tax number is stable, messages with the same seller's tax number are assigned to the same single-threaded execution component. Different seller's tax numbers may enter different execution components, thus achieving overall parallelism while maintaining local order.
[0041] The routing management unit is used to query or establish routing mappings between seller tax numbers and database instances and sharded tables. It includes a cache query component, a routing mapping component, a capacity management component, and a distributed lock component, providing routing query, routing allocation, routing migration, and cache update functions. The hot table synchronization unit performs invoice data operations on unsharded hot tables. It connects to the default database instance and primarily handles the homepage list, recent invoice queries, and quick status confirmation. The sharded storage unit performs invoice data operations on target sharded tables based on the routing mapping. It consists of one or more database instances and multiple physical sharded tables within each database instance, handling complete historical data storage. The search index synchronization unit asynchronously synchronizes complete invoice data from sharded tables to the search index, primarily for fuzzy search by buyer name, status filtering, time range queries, and statistical aggregation. The compensation processing unit records failed tasks and re-executes the corresponding data operations, saving a recoverable context when database or search index operations fail. It is configured to differentiate between hot table compensation and search index compensation based on the target type in the compensation task, and to perform add / update compensation or delete compensation based on the operation type.
[0042] Example 2
[0043] like Figure 2 As shown, a single invoice status change process may include the following steps.
[0044] In the specific process of state change handling, the message consumer retrieves the state change message from the message queue and completes deserialization, message type verification, and necessary field verification. For messages with an empty seller tax number, an empty invoice unique identifier, or a state type that does not belong to the allowed set, the reason for the exception is recorded and transferred to the exception message storage; for duplicate messages that have already been processed and whose business content has not changed, success can be returned directly based on the message identifier.
[0045] After successful verification, the system will determine the target processing thread based on the seller's tax number and submit the task to the bounded blocking queue corresponding to that thread. The processing thread retrieves the task from the queue and queries the routing map corresponding to the seller's tax number. The routing map may include the database instance identifier, shard table identifier, routing status, business table prefix, and routing type. The routing type can distinguish between dynamic sharding routes, default data source routes, manually specified routes, and read-only routes.
[0046] Subsequently, insert or partial field updates are performed on the hot table based on the status type. The initialization status is used to create hot table records; the invoicing status is used to update the invoice code, number, invoicing time, and invoicing result; the signature status is used to update the signature result and electronic document address; and the notification status is used to update SMS or email notification results. Partial field updates prevent irrelevant fields from being overwritten by older messages.
[0047] After hot table processing, the data access layer switches database connections based on routing mapping and converts logical table names to corresponding physical shard table names. The same seller tax number uses the same shard table identifier in the basic invoice information table, invoice details table, and invoice extended information table, facilitating related reads within the same database instance. For newly added data, insert or update operations with unique key constraints are performed; for status changes, condition updates are performed based on the invoice's unique identifier.
[0048] Once the sharded table data is readable, the system will further build a search index document and perform asynchronous synchronization. If any target operation fails, a compensation task is generated according to the failed target, ensuring that the failure of one target does not prevent other target operations that could already be completed. To avoid long-term discrepancies between hot tables and sharded tables, the compensation task rereads the latest business data or uses validated status data during retries, rather than simply reusing potentially outdated fields.
[0049] Example 3
[0050] During route queries, the system constructs a cache query key based on the standardized seller's tax number. The cache record includes at least the database instance identifier, shard table identifier, and route status. If the cache is hit and the route status allows for writing, the route is returned directly; if the cache is missed, the route mapping storage is queried, and the query result is written to the cache. To avoid dirty reads after changes in route relationships, cache records are proactively overwritten after route migration and manual route adjustments.
[0051] like Figure 3 As shown, for new seller tax IDs without assigned routes, the system can classify capacity levels based on parameters such as the company's estimated annual invoice volume and recent growth rate. For example, those with an estimated annual invoice volume greater than 1 million are set to a high capacity level (allocated to a separate large-capacity physical shard); those between 100,000 and 1 million are set to a medium capacity level; and those below 100,000 are set to a low capacity level. During allocation, the system prioritizes target shard resources with remaining available quantities greater than 1.5 times the estimated annual invoice volume for that tax ID. Capacity usage information records the total capacity, allocated quantity, remaining available quantity, and availability status for each database instance and shard table. The system selects resources with more remaining capacity or lower current load from candidate shard resources that meet the capacity level and availability status requirements.
[0052] In multi-tax-number tenant scenarios, the system first queries the most recently used valid routes for the same tenant. If the corresponding shard resources still have sufficient capacity, the new seller tax number is preferentially assigned to the same resource to reduce the probability of cross-database execution of tenant-level summary queries; if the capacity is insufficient, the general capacity selection process is entered. Route allocation must be completed under the protection of distributed locks, and it is verified again whether the seller tax number has already been routed by other requests. Only if no route has been established will capacity deduction and mapping write be performed.
[0053] During expansion, the operations team creates a new shard table corresponding to the business table set in the target database instance and registers it as an allocable resource in the capacity usage information. Routing selection only uses the newly added resources for sales tax numbers for which routes have not yet been established; existing routes are not recalculated due to changes in the number of shards, thus avoiding large-scale migrations caused by static modulo expansion.
[0054] When individual seller entities need to actively migrate, the routing status is first set to migration preparation state to prevent new parallel migrations; historical data is copied and the data volume or checksum is verified; within a short switchover window, locks are acquired, capacity information and routing mappings are updated, and the cache is refreshed; then, new data is written to the target shard. If historical replication adopts a copy-then-add incremental approach, incremental data added during replication can be processed before the switchover, thereby shortening the write pause time.
[0055] Example 4
[0056] The hot table can be set as a fixed logical table to store invoice records within a preset recent range. The recent range can be determined by invoice issuance time, last status update time, or business-side configuration. For example, the system can be configured to only store invoice data issued within the last three months, or to retain a maximum of 50,000 most recently generated invoice status records for each seller's tax ID. Hot table queries do not require dynamic sharding routing and are suitable for users to view recent invoice records, poll invoice status, and download recent electronic files after entering the system. Sharded tables store complete invoice data and are not deleted due to changes in the data lifecycle of the hot table.
[0057] Within the same processing thread, the same message is processed by first performing hot table operations, then shard table operations, ensuring that recent queries on the user's side see the current state change first. To avoid inconsistencies caused by exceptions during the hot table-first, shard table-later process, each target operation independently captures exceptions and writes them into a clearly defined compensation task. The target type in the compensation task indicates whether the failure occurred in the hot table, shard table, or search index. The background task only retryes failed targets and does not repeatedly modify other successful targets.
[0058] Both hot tables and sharded tables can be configured with unique constraints for invoice unique identifiers. When initialization messages arrive repeatedly, an add / update semantics are used; for status change messages, whether to update is determined based on the business occurrence time, status version, or allowed status transition relationships. When a late message significantly earlier than the current record version is detected, only audit information can be registered without overwriting the current status, thereby further preventing status rollback.
[0059] For hot table records that have exceeded the recent range, a separate cleanup task can delete them after confirming that the shard table contains complete records and that there are no incomplete hot table compensation tasks. The cleanup process does not change the shard table or routing mapping, and therefore will not affect historical data queries. This cleanup mechanism is an optional implementation for long-term control of hot table data volume.
[0060] Example 5
[0061] The number of processing threads can be configured based on the number of CPUs, database connections, and average message processing time. The system performs stable hashing on the exporter's tax ID (e.g., using the MurmurHash3 algorithm to generate a 32-bit hash value), and then performs a modulo operation on this hash value with the current total number of single-threaded execution units, selecting the corresponding processing thread based on the modulo result. The number of processing threads remains constant throughout a single runtime cycle to avoid switching between different queues for the same exporter's tax ID due to changes in the thread count during runtime. If the number of threads needs to be adjusted, receiving new messages can be stopped, and the system can wait for the existing queues to be cleared before re-initializing.
[0062] The target processing thread is determined based on the hash result and the number of processing threads, specifically using the following modulo operation formula:
[0063]
[0064] Among them, variables The index value of the target processing thread, variable The variable is the seller's tax number in the current status change message. This represents the total number of single-threaded execution units in the current runtime cycle. By taking the absolute value, it is ensured that... When the algorithm produces a negative result, the final calculated target thread index It must always be a valid non-negative integer, thus ensuring accurate positioning in the delivery queue.
[0065] Each processing thread corresponds to a bounded blocking queue. When the queue capacity is small, the reduced message processing speed will cause consumers to wait during the delivery phase, thus limiting the speed at which messages enter the application memory from the message queue, creating back pressure. The queue capacity can be set according to the allowed memory usage and instantaneous traffic. Within the processing thread, the same task is no longer split into parallel executors to maintain the deterministic order of hot table, shard table, and index task generation.
[0066] The message queue itself can use the seller's tax ID as the message key, ensuring that the same seller's tax ID enters the same message partition as much as possible. In-application hash distribution further guarantees thread affinity within the same consumer instance. For partition reallocation caused by scaling up or down the consumer instance, the consumer ensures that the corresponding message's database operation has been completed or the compensation task has been reliably registered before committing, thereby avoiding message loss during instance switching.
[0067] Example 6
[0068] like Figure 4 As shown, when search index synchronization begins, the system reads basic invoice information, detailed information, extended information, and buyer information that needs to be included in the search from the sharded table based on the routing mapping information. The read results are then assembled into a search index document after field normalization, null value handling, and sensitive field filtering. The index can be divided according to physical sharded tables or business time, and a query entry point is provided through a unified index alias.
[0069] The invoice index document uses a unique invoice identifier as the document identifier. Adding or updating status uses an overwrite approach, ensuring that repeated operations do not generate multiple documents; deletion is considered complete even if the document does not exist. For buyer association scenarios, the seller's tax number and standardized buyer name can be combined to form the buyer document identifier, ensuring that only the latest version of the same buyer information under the same seller entity is retained.
[0070] When synchronization fails, a compensation task is written. Background scheduled tasks organize retry order based on creation time and invoice unique identifier. If multiple incomplete tasks exist for the same invoice, they can be merged into a single overwrite update after reading the latest shard data, or executed item by item according to business version order. Before executing the compensation task, the current route is queried again to avoid accessing the original shard after route migration; if necessary, the original route saved in the task is used to locate data before migration.
[0071] For hot table compensation in the database, the background task performs inserts or partial updates based on the status type, and verifies the current record in the shard table before the operation. For search index compensation, the background task reads the latest complete data from the shard table to avoid using old documents cached when failures occur. Tasks that fail after reaching a preset number of retries are marked for manual handling, and the most recent failure reason, the time of the first failure, and the time of the most recent retry are retained to facilitate fault localization.
[0072] Example 7
[0073] Upon receiving a shutdown signal, the system first switches its running state to stop receiving new tasks, and message consumers either complete the processing of already retrieved batches or stop retrieving more. Then, it sends ordered shutdown requests to each processing thread and waits for tasks in the blocking queue to complete. Only after hot table operations, shard table operations, and synchronization tasks have been completed, or after failed targets have been reliably written to compensation storage, does the system commit the message consumption point and exit.
[0074] If the waiting time reaches the preset limit, the system will persist unfinished tasks as recovery tasks or not commit the corresponding message positions, allowing the system to consume them again upon the next startup. Because database and search index operations use unique identifiers and idempotent semantics, re-consumption will not result in duplicate data.
[0075] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for invoice data dynamic sharding and hot table synchronization based on vendor tax number hash routing, characterized in that, include: Receive the status change message corresponding to the invoice data, and extract the seller's tax number from the status change message; Based on the seller's tax number, the status change message is assigned to the sequential processing queue corresponding to that seller's tax number, so that status change messages with the same seller's tax number are executed by the same processing thread in the order of receipt. Query the routing mapping information corresponding to the seller's tax number to obtain the target database instance and target shard table. If the routing mapping information does not exist, establish routing mapping information for the seller's tax number based on available shard resources. Based on the invoice status data contained in the status change message, insert or update operations are performed on the unsharded hot table, and insert or update operations are performed on the sharded table based on the target database instance and the target sharded table. After the sharded table is written, the complete invoice data is read and asynchronously synchronized to the search index. When the hot table write, sharded table write, or search index synchronization fails, a compensation task containing the invoice identifier, seller's tax number, routing mapping information, operation type, and failure reason is recorded, and the latest invoice data is reread according to the compensation task to perform the corresponding data operation.
2. The invoice data dynamic sharding and hot table synchronization method based on the vendor tax number hash routing according to claim 1, characterized in that: The query for routing mapping information corresponding to the seller's tax number includes: firstly querying routing mapping information from the distributed cache using the seller's tax number as the query key; if the distributed cache is not found, querying routing mapping information from the routing mapping storage and writing the query result into the distributed cache; directly accessing the routing mapping storage when the distributed cache is unavailable; and actively updating the corresponding routing mapping information in the distributed cache after the routing migration is completed.
3. The method of claim 1, wherein the method is characterized by: The step of establishing routing mapping information for the seller's tax number includes: determining the capacity level based on the estimated invoice data size of the seller entity; selecting a shard resource that meets the capacity level and has available capacity from the capacity usage information; acquiring a distributed lock corresponding to the selected shard resource; reducing the available capacity of the selected shard resource under the protection of the distributed lock; writing the routing mapping between the seller's tax number and the selected shard resource; and caching the routing mapping.
4. The method of claim 3, wherein the method is characterized by: The invoice data includes the corresponding tenant identifier; when other sales tax numbers with routing mapping already exist under the same tenant, the available capacity of the shard resources corresponding to the other sales tax numbers is checked first, and the sales tax numbers to be allocated are allocated to the same shard resource when the available capacity meets the requirements; when a new database instance or a new shard table is added, the new resources are registered in the capacity usage information, and the new resources are used for subsequent sales tax numbers that have not yet established routes, while the routing mapping of existing sales tax numbers remains unchanged.
5. The method of claim 1, wherein: The hot table uses a default data source and does not perform sharding routing; it is used to store invoice data within a preset recent range. The sharded table stores complete historical invoice data. The hot table operation and the sharded table operation are triggered by the same state change message and are executed sequentially in the same processing thread. When either the hot table operation or the sharded table operation fails, a compensation task for the corresponding storage target is generated.
6. The method of claim 1, wherein: The process of allocating the queue according to the sales tax number includes: performing hash processing on the sales tax number, determining the target processing thread based on the hash result and the number of processing threads, and delivering the state change message to the bounded blocking queue corresponding to the target processing thread; each target processing thread retrieves a task from the corresponding bounded blocking queue in a single-threaded manner and executes it to maintain the data operation order of the same sales tax number, and blocks message delivery when the queue is full to form back pressure.
7. The method of claim 1, wherein the method is characterized by: The asynchronous synchronization to the search index includes: reading basic invoice information, detailed information, and extended information from the sharded table according to the routing mapping, and assembling them into a search index document; performing overriding additions or updates using the unique invoice identifier as the search index document identifier, and performing repeatable deletion operations in deletion scenarios; writing a compensation task when synchronization fails, and having the background task reread the latest invoice data and perform synchronization according to the creation order of the compensation tasks, deleting the compensation task upon success, and updating the failure reason and retry count upon failure.
8. The method of claim 1, wherein the method is characterized by: It also includes route migration and shutdown protection; the route migration includes adjusting the capacity usage information of the source shard resources and the target shard resources under lock protection, updating the route mapping and cache corresponding to the seller's tax number, so that subsequent invoice data is written to the target shard resources, and historical data is gradually migrated from offline tasks; the shutdown protection includes stopping the reception of new tasks, waiting for the confirmation of the completion of received messages, closing the processing thread, and waiting for the data in transit to be processed to be completed before exiting.
9. A system for dynamic sharding of invoice data based on vendor tax number hash routing and hot table synchronization, using the method of any of claims 1-8, wherein, It includes a message receiving unit, a thread sequence processing unit, a routing management unit, a hot table synchronization unit, a sharded storage unit, a search index synchronization unit, and a compensation processing unit; The message receiving unit is used to receive invoice status change messages and extract the seller's tax number; The thread sequence processing unit is used to assign messages with the same sales tax number to the same sequence processing queue. The routing management unit is used to query or establish routing mappings between the seller's tax number and the database instance and sharded table; The hot meter synchronization unit is used to perform invoice data operations on unsharded hot meters; the sharded storage unit is used to perform invoice data operations on target sharded tables according to routing mapping. The search index synchronization unit is used to asynchronously synchronize the complete invoice data in the sharded table to the search index. The compensation processing unit is used to record failed tasks and re-execute the corresponding data operations.
10. The system for dynamic sharding and hot table synchronization of invoice data based on pin tax number hash routing according to claim 9, wherein: The routing management unit includes a cache query component, a routing mapping component, a capacity management component, and a distributed lock component. The thread sequence processing unit includes multiple independent single-threaded execution components and bounded blocking queues corresponding to each single-threaded execution component. The compensation processing unit is configured to distinguish between hot table compensation and search index compensation based on the target type in the compensation task, and to perform add / update compensation or delete compensation based on the operation type.
