A Smooth Migration Method for Distributed Databases in High-Concurrency Supply Chain Scenarios

Abstract

This invention relates to the field of distributed database migration technology in high-concurrency scenarios, specifically to a smooth distributed database migration method for high-concurrency supply chain scenarios. To address the problems of existing migration solutions requiring downtime, data inconsistency, performance degradation, and difficulty in rollback, this invention establishes a bidirectional asynchronous data synchronization channel between the source and target databases and employs the Snowflake algorithm to avoid ID conflicts during reverse synchronization. Dynamic rate limiting at the interface level is implemented on the business write interface through an API gateway, bringing data synchronization latency close to zero. When the latency reaches zero, based on a distributed configuration center and multiple data source components, the entire service cluster can perform real-time hot switching of the database without restarting or interrupting ongoing requests. If an anomaly is detected after switching, a one-click rollback is possible. This achieves zero business downtime, zero intrusion, request-level lossless cutover, and anytime rollback, significantly improving the smoothness and reliability of database migration in high-concurrency scenarios.

Description

Technical Field

[0001] This invention relates to the field of distributed database migration technology in high-concurrency scenarios, and more specifically, to a method for smooth migration of distributed databases in high-concurrency supply chain scenarios. Background Technology

[0002] In enterprise supply chain systems, as business scales up, monolithic databases (such as document databases and product databases) will face severe performance bottlenecks: limited by the performance of a single database instance, system request throughput and response latency cannot be optimized; large tables lead to table lock contention and a sharp increase in query CPU time. At this point, it is necessary to split the data in the monolithic database according to business rules and migrate it to multiple database instances to distribute the load.

[0003] The overall database migration process mainly includes two steps: writing data to the target database according to rules; and changing the database connection address used by the business service. Existing technologies mainly offer the following two solutions: Option 1: One-time full migration (downtime migration).

[0004] The solution involves completely halting business services to prevent new data generation. After the shutdown, the data to be migrated is synchronized to the target database using DataX according to predefined rules. Once synchronization is complete, the database address configuration for the business services is changed to the target database address, and all services are restarted to complete the migration. This approach requires the business system to be shut down, during which time related services are completely unavailable, severely impacting user experience. The old database immediately stops updating after the switch, making rollback impossible. Furthermore, third-party services cannot maintain synchronization with the business system during the shutdown, posing a risk of data loss.

[0005] Option 2: Incremental dual-write migration.

[0006] The business service is modified to enable dual-write capability (Spring services typically use MyBatis components for dual-write). Simultaneously, data is written to the target database using a combination of Canal monitoring incremental data and DataX synchronizing full data (or using a cloud service provider's DTS service), enabling near real-time synchronization between the old and new databases. Once the historical data is caught up, dual-write is enabled to synchronously write newly generated data to both the old and new databases. Afterward, the business system switches all read requests to the new database and observes its operation. If no anomalies are found, writes to the old database are disabled, completing the migration. While this solution can be implemented without downtime, it is highly complex: during data synchronization, the business service writes to the new database while the incremental listening component based on binlog is also writing, leading to data synchronization conflicts. Improper handling will result in data inconsistency after migration. Dual writing turns the original one data write into two, incurring additional performance overhead and causing service performance degradation, which affects user experience. Dual writing introduces additional multi-database transaction processing, changing the original business code logic. To avoid transaction issues, complex Spring transaction process adaptation is required. In addition, the final database cutover control is usually performed at the data persistence layer of the business service, which can only be done at the SQL statement granularity. During the cutover, it will disrupt the ongoing request transactions, causing existing business logic to malfunction.

[0007] Therefore, existing technologies cannot simultaneously meet the smooth migration requirements of "uninterrupted business operations, consistent data, lossless performance, and support for rollback". Summary of the Invention

[0008] The purpose of this invention is to provide a method for smooth migration of distributed databases in high-concurrency supply chain scenarios to solve the above-mentioned technical problems.

[0009] To achieve the above objectives, the embodiments of this application provide the following technical solutions: This application provides a method for smooth migration of distributed databases in high-concurrency supply chain scenarios. The method includes: synchronizing all data and real-time incremental data from the source database to the target database, while establishing a reverse synchronization link from the target database to the source database to form a bidirectional synchronization channel; implementing dynamic rate limiting on key write interfaces of business services through an API gateway to reduce the rate of new data generation, thereby reducing the synchronization latency between the source database and the target database; when the data synchronization latency is lower than a preset latency threshold, issuing a configuration change command through a distributed configuration center to enable all business service instances to switch to the target database in real time, while subsequent database operations are performed in the target database; after the business services are running stably in the target database, the gateway rate limiting is lifted, and the migration is complete.

[0010] Optionally, when constructing the reverse synchronization link, the snowflake algorithm is used to modify the ID generation strategy of the business service, so that the newly generated data of the business service has a globally unique ID, thereby avoiding ID conflicts with the forward synchronization data.

[0011] Optionally, the dynamic rate limiting is implemented by integrating the Resilience4j rate limiter through Spring Cloud Gateway, which configures an upper limit on the call frequency of key interfaces that generate database write operations in the business service, and the rate limiting configuration supports fine-grained control at the interface level.

[0012] Optionally, to enable all business service instances to switch to the target database in real time, including: The dynamic-datasource component is used to manage and maintain multiple database connections, enabling business services to maintain active connections with both the source and target databases simultaneously. The Nacos distributed configuration center maintains the configuration items for currently enabled data sources, and all business service instances listen for change events of this data source configuration item. When the data source configuration option is switched to the target database, the Nacos distributed configuration center will push the configuration change to all business service instances. The dynamic-datasource component will route subsequent new requests to the target database, while requests that are being processed will continue to complete transactions on the source database.

[0013] Optionally, after the migration is complete, the following may also be included: If a business service malfunctions in the target database, a configuration change is issued through the distributed configuration center, causing all business service instances to immediately switch back to the source database, removing gateway rate limiting, and completing the rollback. Because the bidirectional synchronization link is continuously running, new data generated by the target database during the rollback period has been synchronized back to the source database.

[0014] Optionally, the forward synchronization uses the data transmission service of a cloud service provider to split and synchronize the data in the single source database to multiple target database instances according to business rules. The synchronization rules are configured to synchronize DML and DDL operations. Reverse synchronization uses a data transfer service, configured to synchronize only DML operations and ignore ID conflicts.

[0015] Optionally, the dynamic rate limiting of the API gateway is implemented using the Sentinel component or the Nginx upstream weight adjustment method; The distributed configuration center uses Apollo or ZooKeeper instead of Nacos; The multi-data source management adopts a self-developed routing proxy that modifies the Spring framework's AbstractRoutingDataSource to replace dynamic-datasource.

[0016] The beneficial effects of this invention are as follows: This invention employs a bidirectional asynchronous data synchronization mechanism independent of business services. Business services do not require any code modifications related to data synchronization, saving significant development time. During data synchronization, the quality of business services will not degrade, overcoming the performance degradation problem caused by the increased number of database requests within business services in dual-write solutions.

[0017] Secondly, this invention implements dynamic rate limiting at the interface level for business write interfaces through an API gateway. This allows for flexible selection of the business scope to be rate-limited, limiting only write interfaces without affecting query interfaces. This avoids problems such as abnormal third-party callback service requests, internal scheduled task timeouts, and message queue data backlogs caused by control delays. Compared to dual-write solutions, which can only control the database read / write rate of all businesses at the data operation level, this invention's fine-grained control significantly improves service stability during the migration process.

[0018] Secondly, this invention employs a multi-data source management component combined with a distributed configuration center to enable real-time database switching across the entire cluster without restarting or interrupting ongoing requests. The switching granularity is at the request level; ongoing transactions continue to complete in the source database unaffected. Compared to dual-write solutions that perform cutovers at the SQL statement granularity in the data persistence layer, potentially disrupting ongoing transactions, this invention's hot-switching mechanism is easier to implement and more reliable.

[0019] Secondly, this invention establishes a bidirectional synchronization channel between the source and target databases. If an anomaly is detected after switching to the target database, the business service can be immediately switched back to the source database through the configuration center. Furthermore, due to continuous reverse synchronization, new data generated in the target database during the switch is synchronized back to the source database, ensuring data integrity. In contrast, the shutdown migration solution stops updating the old database after switching and cannot be rolled back; the rollback logic of the dual-write solution is complex and prone to data inconsistency.

[0020] By dynamically limiting the synchronization latency to near zero before hot switching, the switching process is completed in milliseconds without affecting ongoing transactions. The overall migration process ensures that business services continue to operate normally, and the user experience is unaffected, avoiding customer complaints and direct economic losses caused by downtime migration.

[0021] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating the principle of state change during a smooth migration process of a distributed database for high-concurrency supply chain scenarios, as described in this embodiment of the invention. Figure 2 This is a schematic diagram illustrating the principle of a bidirectional synchronization channel during the smooth migration of a distributed database in a high-concurrency supply chain scenario, as described in this embodiment of the invention. Figure 3 This is a schematic diagram illustrating the business gateway rate limiting principle during the smooth migration of a distributed database in a high-concurrency supply chain scenario, as described in this embodiment of the invention. Figure 4 This is a schematic diagram illustrating the principle of hot switching of business service databases during the smooth migration of distributed databases in a high-concurrency supply chain scenario, as described in this embodiment of the invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0025] Example:

[0026] like Figure 1The diagram shown illustrates the principle of state change during the smooth migration of a distributed database for high-concurrency supply chain scenarios described in this embodiment. The core idea is to completely decouple data synchronization from business services, ensure data consistency and rollback through asynchronous bidirectional data synchronization, control data synchronization latency through gateway dynamic rate limiting to achieve non-stop migration, and achieve real-time lossless cutover of business services to the database through a multi-data source hot-switching mechanism.

[0027] Specifically, a method for smooth migration of distributed databases in high-concurrency supply chain scenarios is provided, including: Step S100: Synchronize the full data and real-time incremental data of the source database to the target database, and simultaneously establish a reverse synchronization link from the target database to the source database to form a bidirectional synchronization channel. The specific implementation method can be as follows: like Figure 2 As shown, forward synchronization (source database → target database): The DTS (Data Transmission Service) provided by the cloud service provider is used to split and synchronize data from a single source database to multiple target database instances according to business rules. The DTS synchronization rules are configured to synchronize DML (Data Manipulation Language, including INSERT, UPDATE, DELETE) and DDL (Data Definition Language, including table structure changes) operations, ensuring that both full historical data and real-time incremental data from the source database are synchronized to the target database.

[0028] Reverse synchronization (target database → source database): To support rollback capabilities after migration, a reverse synchronization link from the target database to the source database needs to be established. The core problem facing reverse synchronization is that newly generated data in the target database (written by business services) may conflict with the primary key IDs of data written during forward synchronization. This invention uses the Snowflake ID algorithm to modify the ID generation strategy of business services, ensuring that newly generated data in the distributed database environment has a globally unique ID, thus avoiding conflicts with the IDs of data synchronized forward. The reverse DTS synchronization rules are configured to synchronize only DML operations and ignore ID conflicts (data uniqueness is guaranteed by the Snowflake algorithm), merging new data generated by multiple target database instances and synchronizing it back to the source database.

[0029] Through the aforementioned two-way synchronization mechanism, the data in the source database and the target database remain in near real-time two-way synchronization throughout the entire migration window, providing data assurance for rollback at any time.

[0030] Data synchronization inherently involves latency. Performing database cutover when latency is significant will result in incomplete data in the target database, causing business anomalies. This module uses dynamic rate limiting to reduce synchronization latency to near zero, creating conditions for a safe cutover.

[0031] Step S200: Implement dynamic rate limiting on critical write interfaces of business services through the API gateway to reduce the rate of new data generation, thereby reducing the synchronization latency between the source database and the target database. The specific implementation method is as follows: like Figure 3 As shown, Spring Cloud Gateway is used as the unified API gateway for business services, and the Resilience4j rate limiter component is integrated to implement dynamic rate limiting for key interfaces in business services that generate database write operations.

[0032] The rate limiting strategy is as follows: The Resilience4j rate limiter allows for flexible configuration of the maximum call frequency for a specified interface. Reducing the frequency of API calls decreases the number of requests received by business services, thereby reducing the amount of new business data. With the reduction in the amount of new data, the backlog of DTS data synchronization decreases, and the synchronization latency decreases accordingly until it approaches zero. Rate limiting configuration supports fine-grained control at the interface level, allowing for targeted rate limiting of write interfaces to avoid impacting query interfaces.

[0033] Compared to the dual-write solution, which can only control the database operation read and write rates generated by all business operations, this solution's interface-granular rate limiting allows for flexible selection of the business scope to be limited, avoiding problems such as abnormal third-party callback service requests, timeouts of internal scheduled tasks, and data backlog in message queues caused by control delays.

[0034] Step S300: When the data synchronization delay is lower than the preset delay threshold, a configuration change command is issued through the distributed configuration center to enable all business service instances to switch to the target database in real time. Subsequent database operations are performed in the target database. After the business services are running stably in the target database, the gateway rate limiting is lifted, and the migration is complete. The specific implementation method can be: like Figure 4 As shown, the open-source component dynamic-datasource is used to manage and maintain multiple database connections, enabling business services to maintain active connections with both the source and target databases simultaneously. All data sources maintained by dynamic-datasource are always available, eliminating the need to re-establish connections during switching and achieving millisecond-level switching.

[0035] Nacos, a distributed configuration service, is used to manage configurations from multiple data sources in a unified manner. Nacos maintains a configuration item for "currently enabled data source," and all business service instances listen for changes to this configuration item.

[0036] The hot-switching process is as follows: The operations and maintenance personnel can modify the "Currently Enabled Data Source" configuration item in the Nacos console to switch it from the source database to the target database; Nacos will push the configuration changes to all business service instances that have subscribed to the configuration. After each business service instance receives the configuration change notification, the dynamic-datasource component immediately routes subsequent new requests to the target database; Requests that are currently being processed continue to complete transactions on the source database without being affected; The database switch for the entire cluster is completed automatically after a single configuration operation, without the need to restart each service instance individually.

[0037] If the target database malfunctions, the operations and maintenance personnel can change the "Currently Enabled Data Source" configuration item back to the source database in the Nacos console. All business service instances will immediately switch back to the source database. Due to continuous bidirectional synchronization, the data generated by the target database during the switchover is synchronized back to the source database, ensuring data integrity.

[0038] Secondly, in addition to DTS, bidirectional data synchronization can also be achieved by combining DataX (a full data synchronization tool) with Canal (an incremental data monitoring tool based on MySQL binlog); The Sentinel component can be used to replace Resilience4j for interface rate limiting; or traffic limiting can be achieved through network load balancing components (such as Nginx upstream weight adjustment). You can develop your own multi-data source routing proxy by modifying the Spring framework's data source abstraction interface (AbstractRoutingDataSource), replacing the dynamic-datasource open-source component.

[0039] Other distributed configuration centers such as Apollo and ZooKeeper can be used to replace Nacos.

[0040] This invention employs a bidirectional asynchronous data synchronization mechanism independent of business services. Business services do not require any code modifications related to data synchronization, saving significant development time. During data synchronization, the quality of business services will not degrade, overcoming the performance degradation problem caused by the increased number of database requests within business services in dual-write solutions.

[0041] Secondly, this embodiment implements dynamic rate limiting at the interface level for business write interfaces through an API gateway. This allows for flexible selection of the business scope to be rate-limited, limiting only write interfaces without affecting query interfaces. This avoids problems such as abnormal third-party callback service requests, internal scheduled task timeouts, and message queue data backlogs caused by control delays. Compared to dual-write solutions, which can only control the database read / write rate of all businesses at the data operation level, the fine-grained control of this invention significantly improves service stability during the migration process.

[0042] Secondly, this embodiment employs a multi-data source management component combined with a distributed configuration center to enable real-time database switching across the entire cluster without restarting or interrupting ongoing requests. The switching granularity is at the request level; ongoing transactions continue to complete in the source database unaffected. Compared to dual-write schemes that perform cutovers at the SQL statement granularity in the data persistence layer, potentially disrupting ongoing transactions, the hot-switching mechanism of this invention is easier to implement and more reliable.

[0043] Secondly, this embodiment establishes a bidirectional synchronization channel between the source database and the target database. If an anomaly is detected after switching to the target database, the business service can be immediately switched back to the source database through the configuration center. Furthermore, because reverse synchronization continues, new data generated in the target database during the switch is synchronized back to the source database, ensuring data integrity. In contrast, the shutdown migration solution stops updating the old database after switching and cannot be rolled back; the rollback logic of the dual-write solution is complex and prone to data inconsistency.

[0044] By dynamically limiting the synchronization latency to near zero before hot switching, the switching process is completed in milliseconds without affecting ongoing transactions. The overall migration process ensures that business services continue to operate normally, and the user experience is unaffected, avoiding customer complaints and direct economic losses caused by downtime migration.

[0045] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for smooth migration of distributed databases in high-concurrency supply chain scenarios, characterized in that, The method includes: Synchronize the full data and real-time incremental data of the source database to the target database, and at the same time establish a reverse synchronization link from the target database to the source database to form a two-way synchronization channel. By implementing dynamic rate limiting on critical write interfaces of business services through the API gateway, the rate of new data generation is reduced, thereby reducing the synchronization latency between the source database and the target database. When the data synchronization delay is lower than the preset delay threshold, a configuration change command is issued through the distributed configuration center so that all business service instances switch to the target database in real time, and subsequent database operations are performed in the target database. Once the business services are running stably in the target database, the gateway rate limit is lifted, and the migration is complete.

2. The method for smooth migration of distributed databases in high-concurrency supply chain scenarios according to claim 1, characterized in that, When constructing the reverse synchronization link, the snowflake algorithm is used to modify the ID generation strategy of the business service, so that the newly generated data of the business service has a globally unique ID, thereby avoiding ID conflicts with the forward synchronization data.

3. The method for smooth migration of distributed databases in high-concurrency supply chain scenarios according to claim 1, characterized in that, The dynamic rate limiting is implemented by integrating the Resilience4j rate limiter through Spring Cloud Gateway. It configures the call frequency limit for key interfaces that generate database write operations in business services, and the rate limiting configuration supports fine-grained control at the interface level.

4. The method for smooth migration of distributed databases in high-concurrency supply chain scenarios according to claim 1, characterized in that, To enable all business service instances to switch to the target database in real time, including: The dynamic-datasource component is used to manage and maintain multiple database connections, enabling business services to maintain active connections with both the source and target databases simultaneously. The Nacos distributed configuration center maintains the configuration items for currently enabled data sources, and all business service instances listen for change events of this data source configuration item. When the data source configuration option is switched to the target database, the Nacos distributed configuration center will push the configuration change to all business service instances. The dynamic-datasource component will route subsequent new requests to the target database, while requests that are being processed will continue to complete transactions on the source database.

5. The method for smooth migration of distributed databases in high-concurrency supply chain scenarios according to claim 1, characterized in that, After the migration is complete, it also includes: If a business service malfunctions in the target database, a configuration change is issued through the distributed configuration center, causing all business service instances to immediately switch back to the source database, removing gateway rate limiting, and completing the rollback. Because the bidirectional synchronization link is continuously running, new data generated by the target database during the rollback period has been synchronized back to the source database.

6. The method for smooth migration of distributed databases in high-concurrency supply chain scenarios according to claim 2, characterized in that, The forward synchronization uses the data transmission service of the cloud service provider to split and synchronize the data in the single source database to multiple target database instances according to business rules. The synchronization rules are configured to synchronize DML and DDL operations. Reverse synchronization uses a data transfer service, configured to synchronize only DML operations and ignore ID conflicts.

7. The method for smooth migration of distributed databases in high-concurrency supply chain scenarios according to claim 1, characterized in that, The dynamic rate limiting of the API gateway is implemented using the Sentinel component or Nginx upstream weight adjustment method; The distributed configuration center uses Apollo or ZooKeeper instead of Nacos; The multi-data source management adopts a self-developed routing proxy that modifies the Spring framework's AbstractRoutingDataSource to replace dynamic-datasource.

Images