Cross-database batch data synchronization method and system, electronic equipment and storage medium

Abstract

The invention discloses a cross-database batch data synchronization method and system, electronic equipment and a storage medium, and the method comprises the steps: creating a database connection pool, and recognizing to-be-synchronized incremental data according to an update record of a source database; determining a data synchronization mode according to the synchronization configuration information in combination with the current system load and the data volume of the incremental data; distributing and initializing corresponding thread resources according to the data synchronization mode; based on the database connection pool and the thread resources, executing batch data synchronization operation from the source database to the target database; wherein the batch data synchronization operation comprises the following steps: writing incremental data into a target database in a batch operation mode according to a dynamically determined batch size; and after the batch data synchronization operation is completed, recycling the thread resources and the database connection pool. According to the invention, the problems of poor large-batch synchronization performance, high delay and easy inconsistency are solved, and the data synchronization real-time performance, stability and resource utilization efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, and in particular to a method, system, electronic device, and storage medium for cross-database batch data synchronization. Background Technology

[0002] In the field of enterprise data management, large-scale data synchronization across databases has become a crucial step in ensuring data consistency and timeliness. As enterprise data volumes continue to grow, the need for data synchronization between different databases is becoming increasingly urgent. However, existing technologies generally have many limitations when handling large-scale data synchronization, leading to frequent problems such as data delays and inconsistencies, making it difficult to meet enterprises' requirements for data real-time performance and integrity.

[0003] Currently, the industry mainly uses technologies such as Change Data Capture (CDC), data import and export, message queues, ETL tools, and customized scripts to achieve data synchronization. Change Data Capture technology requires configuring database logs or triggers, increasing operational complexity. Real-time monitoring may also cause performance degradation to the source database, and not all databases support this mechanism, resulting in weak traceability of historical data changes. Data import and export typically uses batch operation modes, making it difficult to achieve real-time data updates. Data inconsistencies or loss are prone to occur during transmission, and some steps rely on manual intervention, leading to high operational risks. Large-scale data transmission can also significantly impact performance. Message queue mechanisms may experience message loss or duplicate processing in abnormal scenarios, requiring additional investment in architecture design and operational resources. High loads can easily cause processing delays, and the asynchronous nature makes problem tracking and debugging difficult. Commercial ETL tools are costly, inefficient when processing large datasets, have limited support for complex transformation logic, and require users to undergo professional training. Customized script development demands highly skilled technical personnel, has poor reusability, lacks standardized processes that can introduce defects, and becomes significantly more difficult to maintain after changes in key personnel.

[0004] The inherent flaws in the aforementioned technical solutions pose significant risks to the large-scale data synchronization process. Synchronization interruptions can lead to data loss or delayed updates, causing inconsistencies between multiple databases and consequently affecting the accuracy of decision analysis. Excessive resource consumption can slow down platform response times, and in severe cases, even cause service outages, impacting user experience. In disaster recovery scenarios, a lack of timely data synchronization will prolong recovery cycles and increase operational risks. Ultimately, these issues may damage corporate reputation and trigger compliance risks.

[0005] Therefore, there is an urgent need to provide a technical solution to address the above problems. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method, system, electronic device, and storage medium for cross-database batch data synchronization.

[0007] Firstly, the present invention provides a method for cross-database batch data synchronization, the technical solution of which is as follows: Create a database connection pool and identify the incremental data to be synchronized based on the update records of the source database; Based on the synchronization configuration information, and in combination with the current system load and the amount of incremental data, the data synchronization mode is determined; According to the data synchronization mode, allocate and initialize the corresponding thread resources; Based on the database connection pool and the thread resources, a batch data synchronization operation is performed from the source database to the target database; wherein, the batch data synchronization operation includes: writing the incremental data into the target database in a batch operation manner according to dynamically determined batch sizes; Once the batch data synchronization operation is complete, the thread resources and the database connection pool are reclaimed.

[0008] The beneficial effects of the cross-database batch data synchronization method of the present invention are as follows: The method of this invention identifies incremental data by creating a database connection pool, dynamically determines the synchronization mode and batch size by combining system load and data volume, allocates thread resources to perform batch writing and reclaims resources, solves the problems of poor performance, high latency and inconsistency in large-batch synchronization, and improves the real-time performance, stability and resource utilization efficiency of data synchronization.

[0009] Based on the above scheme, the cross-database batch data synchronization method of the present invention can be further improved as follows.

[0010] In one alternative approach, the step of identifying incremental data to be synchronized based on update records in the source database includes: Based on the last synchronization time recorded in the source database, query the update records of the source database, where the update records contain a time identifier of the data change; From the update record, extract the data changes whose time is after the last synchronization time point, and determine the data changes as the incremental data.

[0011] The advantages of adopting the above optional methods are: by further querying the update records by recording the last synchronization time point, the incremental data can be accurately identified, avoiding duplicate synchronization and omissions, reducing the resource consumption of the source database, and improving the accuracy and reliability of data synchronization.

[0012] In one optional approach, the synchronization configuration information includes: a data volume threshold and a load threshold; the step of determining the data synchronization mode based on the synchronization configuration information, combined with the current system load and the data volume of the incremental data, includes: Determine whether the amount of incremental data is greater than the data amount threshold to obtain a first determination result, and determine whether the current system load is less than the load threshold to obtain a second determination result; When both the first and second judgment results are true, the data synchronization mode is determined to be a multi-threaded synchronization mode; otherwise, the data synchronization mode is determined to be a single-threaded synchronization mode.

[0013] The advantages of adopting the above optional approach are: further introducing a dual judgment mechanism of data volume threshold and load threshold to realize adaptive switching of synchronization mode, enabling multi-threading when the data volume is large and the system load is low, and using single-threading otherwise, thus balancing synchronization efficiency and system stability.

[0014] In one alternative approach, the step of allocating and initializing the corresponding thread resources according to the data synchronization mode includes: When the data synchronization mode is the multi-threaded synchronization mode, the target number of threads required is calculated based on the amount of incremental data and the threshold for the amount of data processed by a single thread, and a thread pool is created and initialized; wherein, the thread pool contains the target number of threads; When the data synchronization mode is the single-threaded synchronization mode, a single thread is created and initialized.

[0015] The advantages of adopting the above optional methods are: further differentiated allocation of thread resources based on the synchronization mode, creation of a thread pool based on the number of threads calculated according to the data volume in the multi-threaded mode, and creation of only a single thread in the single-threaded mode, thereby realizing elastic configuration and on-demand use of computing resources.

[0016] In one alternative approach, the step of writing the incremental data into the target database in batches according to dynamically determined batch sizes includes: Based on the current system load, the batch size for writing to the target database is dynamically determined, and a database connection is obtained from the database connection pool. When the data synchronization mode is a multi-threaded synchronization mode, multiple threads in the thread pool are used to divide the incremental data into data batches that meet the batch size, and multiple data batches are written to the target database in a batch operation manner through the database connection. When the data synchronization mode is a single-threaded synchronization mode, the incremental data is divided into multiple data batches that conform to the batch size using the single thread, and each data batch is written to the target database in a batch operation manner through the database connection.

[0017] The advantages of adopting the above optional method are: further dynamically determining the batch size based on the current system load, obtaining a connection through the database connection pool, and writing to the target database in batches, thereby achieving adaptive adjustment of write pressure, avoiding overload of the target database, and improving batch write performance.

[0018] In one alternative approach, the step of reclaiming the thread resources and the database connection pool after the batch data synchronization operation is completed includes: After the batch data synchronization operation is completed and all data batches have been written to the target database, the thread pool or the individual thread is shut down. Release all database connections obtained through the database connection pool during the batch data synchronization operation back to the database connection pool.

[0019] The advantages of using the above optional methods are: further closing the thread pool or individual thread after synchronization is completed, and releasing the acquired database connection back to the connection pool, ensuring timely resource reclamation, preventing connection leakage, maintaining the healthy state of the connection pool, and reducing system resource consumption.

[0020] In one alternative approach, the step of dynamically determining the batch size for writing to the target database based on the current system load includes: Based on the preset correspondence between load and batch size, the batch size corresponding to the current system load is determined.

[0021] The advantages of adopting the above optional method are: by further pre-setting the correspondence between load and batch size, the batch size that matches the current system load can be quickly determined, simplifying the dynamic decision-making logic, making the batching strategy transparent and configurable, and facilitating operation and maintenance management and optimization.

[0022] Secondly, this invention provides a cross-database batch data synchronization system, the technical solution of which is as follows: The identification module is used to create a database connection pool and identify incremental data to be synchronized based on the update records of the source database. The determination module is used to determine the data synchronization mode based on the synchronization configuration information, combined with the current system load and the amount of incremental data; The configuration module is used to allocate and initialize the corresponding thread resources according to the data synchronization mode; The synchronization module is used to perform batch data synchronization operations from the source database to the target database based on the database connection pool and the thread resources; wherein, the batch data synchronization operation includes: writing the incremental data into the target database in a batch operation manner according to dynamically determined batch sizes; The recycling module is used to reclaim the thread resources and the database connection pool after the batch data synchronization operation is completed.

[0023] The beneficial effects of the cross-database batch data synchronization system of the present invention are as follows: The system of this invention identifies incremental data by creating a database connection pool, dynamically determines the synchronization mode and batch size by combining system load and data volume, allocates thread resources to perform batch writing and reclaims resources, solves the problems of poor performance, high latency and inconsistency in large-batch synchronization, and improves the real-time performance, stability and resource utilization efficiency of data synchronization.

[0024] Thirdly, the technical solution of an electronic device according to the present invention is as follows: It includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the cross-database batch data synchronization method of the present invention.

[0025] Fourthly, the technical solution of a computer-readable storage medium provided by the present invention is as follows: The computer-readable storage medium stores instructions that, when read, cause the computer-readable storage medium to perform the steps of the cross-database batch data synchronization method of the present invention.

[0026] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0027] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an embodiment of a cross-database batch data synchronization method according to the present invention. Figure 2 This is a schematic diagram of the overall process; Figure 3 This is a schematic diagram of an embodiment of a cross-database batch data synchronization system according to the present invention; Figure 4This is a schematic diagram of an embodiment of an electronic device according to the present invention. Detailed Implementation

[0028] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0029] Figure 1 This diagram illustrates a flowchart of an embodiment of a cross-database batch data synchronization method provided by the present invention. This method can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the cross-database batch data synchronization method by having its processor call computer-readable instructions stored in its memory. Figure 1 As shown, it includes the following steps: S1. Create a database connection pool and identify the incremental data to be synchronized based on the update records of the source database.

[0030] A database connection pool refers to a technical component used to manage and reuse database connections. It pre-creates and maintains a certain number of database connections. When interaction with the database is needed, an idle connection is obtained from the pool and returned after use, instead of repeatedly creating and destroying new connections. For example, in a policy query service, a database connection pool with a maximum of 25 connections is configured. When multiple business requests simultaneously query policy details, an available connection is allocated from this pool to perform the operation, and the connection is returned to the pool after use. The source database refers to the database that serves as the starting point for data synchronization operations, storing the original or business data to be synchronized. For example, a relational database supporting core insurance business, storing data tables such as "Policy Information Table," "Customer Information Table," and "Claim Application Form Table." Update logs refer to a mechanism or storage structure used by the source database to track data changes, recording when the data was modified. For example, through the database's transaction log or a dedicated "data change audit table," it might be recorded that on January 1, 2025, at 10:30:00, a policy with policy number "BK202501010001" in the "Policy Information Table" had its "Payment Status" field updated from "Pending Payment" to "Paid." Incremental data to be synchronized refers to the data content that has changed in the source database since the last synchronization operation but has not yet been synchronized to the target database. For example, since the last synchronization, the source database may have added 300 new policies and experienced 150 claims status updates; these 450 data changes constitute the incremental data to be synchronized this time.

[0031] S2. Based on the synchronization configuration information and in combination with the current system load and the amount of incremental data, determine the data synchronization mode.

[0032] Synchronization configuration information refers to a pre-defined set of parameters or rules used to guide and control the data synchronization process. For example, a configuration table might store parameters such as a data volume threshold of 8000 records to trigger multi-threaded mode and a CPU utilization threshold of 65% to indicate system idleness. These parameters together constitute the synchronization configuration information. Current system load refers to a quantitative indicator of the resource usage of the server performing the data synchronization task at a given moment. For example, if, when preparing to perform data synchronization, the server's CPU utilization is monitored at 50% and memory utilization at 60%, this 50% CPU utilization is the current system load indicator used for evaluation. The data volume of incremental data refers to the measure of the number of data records contained in the incremental data to be synchronized. For example, if it is identified that the incremental data to be synchronized contains 1800 customer information update records, then the data volume of this incremental data is 1800 records. Data synchronization mode refers to the computing resource scheduling strategy selected based on system status and data characteristics to execute the current synchronization task, mainly distinguishing between concurrent and serial processing; for example, based on configuration and current status, it is determined to adopt "single-thread synchronization mode" to serially process the current synchronization task with a small amount of data.

[0033] S3. Allocate and initialize the corresponding thread resources according to the data synchronization mode.

[0034] Among them, thread resources refer to system threads allocated for performing computational tasks in data synchronization operations. These can be a single thread or a thread pool containing multiple threads. For example, according to a defined pattern, a thread pool containing only one worker thread is initialized, and this one thread is the thread resource available for this synchronization task.

[0035] S4. Based on the database connection pool and the thread resources, perform a batch data synchronization operation from the source database to the target database; wherein, the batch data synchronization operation includes: writing the incremental data into the target database in a batch operation manner according to dynamically determined batch sizes.

[0036] The target database refers to the database that serves as the destination for data synchronization operations, used to receive and store data synchronized from the source database. For example, a data warehouse used for big data risk analysis needs to periodically receive policy and customer data synchronized from the core business database. Batch data synchronization operation refers to the complete process of transferring identified incremental data from the source database to the target database. For example, writing 1800 incremental customer data records into the risk analysis data warehouse in batches through an established connection and a single allocated thread constitutes a batch data synchronization operation. Batch size refers to the number of data records submitted to the target database each time in a batch data synchronization operation. For example, if the 1800 data records are dynamically determined based on the current load and submitted in batches of 300, then these 300 records constitute the batch size for this operation. Batch operation refers to a function provided by a database programming interface that allows multiple data operation statements to be combined into a single unit and sent to the database for execution at once, thereby reducing network round trips and transaction overhead. For example, using the batch processing function of the programming interface, 300 "INSERT" statements can be added to a batch processing unit, and then these 300 insert requests can be sent to the target database for execution at once.

[0037] S5. After the batch data synchronization operation is completed, the thread resources and the database connection pool are reclaimed.

[0038] The process of reclaiming thread resources and database connection pools refers to the resource cleanup actions performed after the data synchronization operation is completed. This includes terminating the threads used for computation and releasing the occupied database connections back to the database connection pool. For example, after synchronization is completed, the previously initialized thread pool is closed, and the two database connections obtained from the database connection pool in this task are marked as idle and returned to the database connection pool.

[0039] The technical solution of this embodiment identifies incremental data by creating a database connection pool, dynamically determines the synchronization mode and batch size by combining system load and data volume, allocates thread resources to perform batch writing and reclaims resources, solves the problems of poor performance, high latency and inconsistency of large-batch synchronization, and improves the real-time performance, stability and resource utilization efficiency of data synchronization.

[0040] In one alternative approach, the step of identifying incremental data to be synchronized based on update records in the source database includes: Based on the last synchronization time recorded in the source database, query the update records of the source database, which contain the time identifier of the data change.

[0041] The last synchronization time point refers to the completion time of the last successful data synchronization operation recorded. For example, if the last successful synchronization of customer data recorded in the metadata management table is "2024-12-31 22:00:00", this time point is the last synchronization time point. The data change timestamp refers to the timestamp or sequence number field in the update record used to identify when a piece of data was changed. For example, in the "Data Change Audit Table", each change record contains a "Change Time" field with a value of "2025-01-01 10:30:00", this timestamp is the data change timestamp.

[0042] From the update record, extract the data changes whose time is after the last synchronization time point, and determine the data changes as the incremental data.

[0043] In this context, data change details refer to the specific information in the update record that describes what changes have occurred to the data, in addition to the time stamp. This usually includes the change type and the changed data value. For example, an update record might state that for the record in the "Policy Information Table" with policy number "BK202501010001", the value of the "Payment Status" field has changed from "Pending Payment" to "Paid". This describes the specific details of the change.

[0044] Among the above optional methods, further incremental data can be accurately identified by querying the update record at the last synchronization time point, avoiding duplicate synchronization and omissions, reducing resource consumption on the source database, and improving the accuracy and reliability of data synchronization.

[0045] In one optional approach, the synchronization configuration information includes: a data volume threshold and a load threshold; S2 specifically includes: Determine whether the amount of incremental data is greater than the data amount threshold to obtain a first determination result, and determine whether the current system load is less than the load threshold to obtain a second determination result.

[0046] The data volume threshold refers to a preset threshold in the synchronization configuration information for the number of data entries required to trigger different synchronization modes. For example, if the data volume threshold is set to 8000 entries, this means that when the incremental data volume exceeds 8000 entries, it is a condition for considering enabling multi-threaded mode. The load threshold refers to a preset threshold in the synchronization configuration information for the resource utilization rate used to characterize the system's workload. For example, if the load threshold is set to 65% CPU utilization, this means that when the current CPU utilization is below 65%, it is another condition for considering enabling multi-threaded mode.

[0047] When both the first and second judgment results are true, the data synchronization mode is determined to be a multi-threaded synchronization mode; otherwise, the data synchronization mode is determined to be a single-threaded synchronization mode.

[0048] Multi-threaded synchronization mode refers to a data synchronization mode in which multiple threads from a thread pool are created or used to concurrently execute data partitioning, transmission, or writing tasks. For example, when the data volume exceeds 8000 records and the CPU utilization is below 65%, the defined "multi-threaded synchronization mode" means that multiple threads will be used to process different batches of data simultaneously. Single-threaded synchronization mode refers to a data synchronization mode in which only a single thread is created or used to serially execute all data synchronization steps. For example, when the data volume is 1800 records and does not exceed the threshold, the defined "single-threaded synchronization mode" means that only one thread will be used to process all incremental data sequentially.

[0049] Among the above optional methods, a dual judgment mechanism of data volume threshold and load threshold is further introduced to realize adaptive switching of synchronization mode. When the data volume is large and the system load is low, multi-threading is enabled, otherwise single-threading is used to balance synchronization efficiency and system stability.

[0050] In one alternative approach, S3 specifically includes: When the data synchronization mode is the multi-threaded synchronization mode, the target number of threads is calculated based on the amount of incremental data and the threshold for the amount of data processed by a single thread, and a thread pool is created and initialized; wherein, the thread pool contains the target number of threads.

[0051] The single-threaded data processing threshold refers to a parameter preset in the synchronization configuration information used to estimate the number of threads required in multi-threaded mode, representing the expected amount of data that a single thread can process. For example, if the configuration sets the single-threaded data processing threshold to 600 records, it means that when estimating the required number of threads, it is assumed that one thread can efficiently handle a synchronization task of approximately 600 records. The target number refers to the approximate number of threads to be created in multi-threaded synchronization mode, calculated based on the total incremental data volume and the single-threaded data processing threshold. For example, for 8500 incremental records, with a single thread processing threshold of 600 records, approximately 14.2 threads are needed; after rounding up, the target number of threads is determined to be 15. A thread pool refers to a computer resource management component initialized in multi-threaded synchronization mode, containing multiple reusable worker threads. For example, based on the target number of 15, a thread pool of fixed size 15 is created, managing 15 worker threads in a ready state.

[0052] When the data synchronization mode is the single-threaded synchronization mode, a single thread is created and initialized.

[0053] In this context, a single thread refers to a computing unit initialized in single-threaded synchronization mode that is uniquely used to execute the current data synchronization task; for example, an independent thread is created that will be responsible for the entire synchronization process from data partitioning to writing.

[0054] Among the above optional methods, thread resources are further allocated differently according to the synchronization mode. In the multi-threaded mode, the number of threads is calculated based on the amount of data to create a thread pool, while in the single-threaded mode, only a single thread is created, so as to realize the elastic configuration and on-demand use of computing resources.

[0055] In one alternative approach, the step of writing the incremental data into the target database in batches according to dynamically determined batch sizes includes: Based on the current system load, the batch size of writing to the target database is dynamically determined, and database connections are obtained from the database connection pool.

[0056] A database connection refers to an active session channel obtained through a database connection pool to communicate with a target database and to perform data query or write operations. For example, a connection to a target database used for risk analysis is obtained from the database connection pool, and a structured query language statement is executed through this connection to write synchronous data into the "policy analysis table" of that database.

[0057] When the data synchronization mode is multi-threaded synchronization mode, multiple threads in the thread pool are used to divide the incremental data into data batches that meet the batch size, and multiple data batches are written to the target database in a batch operation manner through the database connection.

[0058] When the data synchronization mode is a single-threaded synchronization mode, the incremental data is divided into multiple data batches that conform to the batch size using the single thread, and each data batch is written to the target database in a batch operation manner through the database connection.

[0059] In the above optional methods, the batch size is further dynamically determined based on the current system load. After obtaining a connection through the database connection pool, the data is written to the target database in batches, thereby achieving adaptive adjustment of write pressure, avoiding overload of the target database, and improving batch write performance.

[0060] In one alternative approach, S5 specifically includes: After the batch data synchronization operation is completed and all data batches have been written to the target database, the thread pool or the individual thread is shut down.

[0061] Release all database connections obtained through the database connection pool during the batch data synchronization operation back to the database connection pool.

[0062] In the above optional methods, the thread pool or a single thread is further closed after synchronization is completed, and the acquired database connection is released back to the database connection pool to ensure timely resource reclamation, prevent connection leakage, maintain the healthy state of the database connection pool, and reduce system resource consumption.

[0063] In one alternative approach, the step of dynamically determining the batch size for writing to the target database based on the current system load includes: Based on the preset correspondence between load and batch size, the batch size corresponding to the current system load is determined.

[0064] The load-batch size correspondence refers to a preset configuration rule that describes the mapping relationship between different system load ranges and recommended batch sizes. For example, the preset rule is: when the CPU utilization is below 40%, the batch size is 500 rows; when the CPU utilization is between 40% and 70%, the batch size is 300 rows; and when the CPU utilization is above 70%, the batch size is 100 rows. Based on the current CPU utilization of 50%, the batch size is determined to be 300 rows according to this correspondence.

[0065] Among the above optional methods, the batch size that matches the current system load can be quickly determined by pre-setting the correspondence between load and batch size, simplifying the dynamic decision-making logic, making the batching strategy transparent and configurable, and facilitating operation and maintenance management and optimization.

[0066] like Figure 2 As shown, after the cross-database batch data synchronization process begins and undergoes routine business logic processing, it enters the core data synchronization processing stage: 1) Create a database connection pool. After creating the database connection pool, identify the incremental data to be synchronized based on the update records of the source database. The identification process is based on the last synchronization time recorded in the source database, querying update records containing data change time identifiers, extracting data change content whose change time is after the last synchronization time, and determining the extracted data change content as the incremental data to be synchronized this time.

[0067] 2) Entering the data synchronization mode determination stage. This stage determines whether to use multi-threading based on the preset synchronization configuration information, the current system load, and the amount of incremental data identified. The synchronization configuration information includes a data volume threshold and a load threshold. The determination logic is as follows: determine whether the amount of incremental data is greater than the data volume threshold, and whether the current system load is less than the load threshold; if the amount of incremental data is greater than the data volume threshold and the current system load is less than the load threshold, the data synchronization mode is determined to be multi-threaded synchronization mode; otherwise, the data synchronization mode is determined to be single-threaded synchronization mode.

[0068] 3) Based on the determined data synchronization mode, allocate and initialize the corresponding thread resources. When the data synchronization mode is multi-threaded, calculate the target number of threads required based on the incremental data volume and the threshold for processing data by a single thread, and create and initialize a thread pool containing the target number of threads. When the data synchronization mode is single-threaded, create and initialize a single thread.

[0069] 4) After thread resource initialization, based on the database connection pool and thread resources, a batch data synchronization operation is performed from the source database to the target database. The core of the batch data synchronization operation is to write incremental data to the target database in batches according to dynamically determined batch sizes. Specifically, the batch size to be written to the target database is dynamically determined based on the current system load, and a database connection is obtained from the database connection pool. If the data synchronization mode is multi-threaded, multiple threads in the thread pool concurrently divide the incremental data into batches that meet the batch size, and write these batches to the target database in batches through the database connection. If the data synchronization mode is single-threaded, a single thread divides the incremental data into multiple batches that meet the batch size, and writes each batch to the target database in batches through the database connection. The step of dynamically determining the batch size can be accomplished based on a preset load-batch size correspondence.

[0070] 5) During batch data synchronization operations, the monitoring log manager is responsible for collecting, storing, and analyzing application runtime logs, performance metrics, and potential errors, enabling real-time tracking of synchronization progress and timely detection of problems. This monitoring mechanism ensures the observability and maintainability of the system.

[0071] 6) Once the batch data synchronization operation is complete and it is confirmed that all data batches have been written to the target database, the process enters the resource reclamation phase. Reclamation includes shutting down previously initialized thread pools or individual threads, and releasing all database connections obtained through the database connection pool during this batch data synchronization operation back to the database connection pool. Finally, this data processing request ends.

[0072] It should be noted that the entire process described above is supported collaboratively by the task scheduler, resource manager, and monitoring log manager. The task scheduler sets and manages the execution plan of synchronization tasks according to the actual business scenario. The resource manager includes a data access component and a multi-threading component. The data access component effectively manages and reuses database connections through a database connection pool, while the multi-threading component manages and reuses threads through a thread pool, jointly improving resource utilization and system concurrency performance. The entire solution reduces unnecessary data transmission by combining incremental synchronization strategies, reduces database interaction overhead by utilizing batch operations and optimized batch sizes, and allows for performance tuning of the source and target databases before and after synchronization. Ultimately, it achieves efficient, real-time, and stable synchronization of large amounts of data across databases, improving overall resource utilization efficiency and system reliability.

[0073] To better illustrate the technical solution of this embodiment, the following complete example is used for explanation, specifically: S10: The task scheduler triggers a policy data synchronization request from the core business database to the analysis data warehouse according to the preset business plan; S20. Create a database connection pool. Based on the last synchronization time recorded in the core business database, "2024-12-31 23:59:59", query the update records of the database. The update records contain a data change time identifier field. Extract the data change content after "2024-12-31 23:59:59" from the update records. Determine the extracted data change content as the incremental data to be synchronized. The incremental data identified this time represents all new insurance policies and policies with status changes since the last synchronization. The data volume is 6500 records. S30. Read the synchronization configuration information. The synchronization configuration information includes a data volume threshold of 5000 records and a load threshold of 65% CPU utilization. The current system load is obtained as 35% CPU utilization. It is determined that the incremental data volume of 6500 records is greater than the data volume threshold of 5000 records, so the first judgment result is yes. It is determined that the current system load of 35% is less than the load threshold of 65%, so the second judgment result is yes. Since both the first and second judgment results are yes, the data synchronization mode is determined to be multi-threaded synchronization mode. S40. Allocate and initialize the corresponding thread resources according to the determined multi-threaded synchronization mode. Based on the incremental data volume of 6500 data entries and the threshold of 800 data entries processed by a single thread, calculate the target number of required threads as 9, and create and initialize a thread pool containing 9 worker threads. S50. Based on the current system load and CPU utilization of 35%, dynamically determine the batch size to be written to the target database, query the preset relationship between load and batch size, determine that the batch size corresponding to the current load is 500 rows, and obtain an available database connection from the database connection pool. S60. Using 9 threads in the thread pool, 6500 incremental data entries are concurrently divided into multiple data batches, each containing 500 data entries, resulting in a total of 13 data batches. Through the obtained database connection, the batch processing function of the database programming interface is used to write each data batch into the target table of the analysis data warehouse in a batch operation mode. S70. During the execution of batch data synchronization operations, the monitoring log manager continuously collects and records the synchronization start time, the amount of data processed, the thread status and performance indicators, and stores this information as a running log for real-time tracking and troubleshooting. S80. After all 13 data batches have been confirmed to be written to the target database, the batch data synchronization operation is completed. The previously initialized thread pool containing 9 threads is closed, and the database connection obtained through the database connection pool in this operation is released back to the database connection pool. S90. This data synchronization request has been completed.

[0074] Figure 3 A schematic diagram of an embodiment of a cross-database batch data synchronization system 200 provided by the present invention is shown. Figure 3 As shown, the cross-database batch data synchronization system 200 includes: The identification module 201 is used to create a database connection pool and identify incremental data to be synchronized based on the update records of the source database. The determining module 202 is used to determine the data synchronization mode based on the synchronization configuration information and in combination with the current system load and the amount of incremental data; Configuration module 203 is used to allocate and initialize corresponding thread resources according to the data synchronization mode; Synchronization module 204 is used to perform batch data synchronization operations from the source database to the target database based on the database connection pool and the thread resources; wherein, the batch data synchronization operation includes: writing the incremental data into the target database in a batch operation manner according to dynamically determined batch sizes; The recycling module 205 is used to reclaim the thread resources and the database connection pool after the batch data synchronization operation is completed.

[0075] In one alternative embodiment, the identification module 201 is specifically used for: Based on the last synchronization time recorded in the source database, query the update records of the source database, where the update records contain a time identifier of the data change; From the update record, extract the data changes whose time is after the last synchronization time point, and determine the data changes as the incremental data.

[0076] In one optional approach, the synchronization configuration information includes: a data volume threshold and a load threshold; the determining module 202 is specifically used for: Determine whether the amount of incremental data is greater than the data amount threshold to obtain a first determination result, and determine whether the current system load is less than the load threshold to obtain a second determination result; When both the first and second judgment results are true, the data synchronization mode is determined to be a multi-threaded synchronization mode; otherwise, the data synchronization mode is determined to be a single-threaded synchronization mode.

[0077] In an alternative embodiment, the configuration module 203 is specifically used for: When the data synchronization mode is the multi-threaded synchronization mode, the target number of threads required is calculated based on the amount of incremental data and the threshold for the amount of data processed by a single thread, and a thread pool is created and initialized; wherein, the thread pool contains the target number of threads; When the data synchronization mode is the single-threaded synchronization mode, a single thread is created and initialized.

[0078] In an alternative embodiment, the synchronization module 204 is specifically used for: Based on the current system load, the batch size for writing to the target database is dynamically determined, and a database connection is obtained from the database connection pool. When the data synchronization mode is a multi-threaded synchronization mode, multiple threads in the thread pool are used to divide the incremental data into data batches that meet the batch size, and multiple data batches are written to the target database in a batch operation manner through the database connection. When the data synchronization mode is a single-threaded synchronization mode, the incremental data is divided into multiple data batches that conform to the batch size using the single thread, and each data batch is written to the target database in a batch operation manner through the database connection.

[0079] In an alternative embodiment, the recycling module 205 is specifically used for: After the batch data synchronization operation is completed and all data batches have been written to the target database, the thread pool or the individual thread is shut down. Release all database connections obtained through the database connection pool during the batch data synchronization operation back to the database connection pool.

[0080] In an alternative embodiment, the synchronization module 204 is specifically used for: Based on the preset correspondence between load and batch size, the batch size corresponding to the current system load is determined.

[0081] It should be noted that the beneficial effects of the cross-database batch data synchronization system 200 provided in the above embodiments are the same as those of the cross-database batch data synchronization method described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0082] The cross-database batch data synchronization system 200 of the present invention can be a computer program (including program code) running on a computer device. For example, the cross-database batch data synchronization system 200 of the present invention is an application software that can be used to execute the corresponding steps in the cross-database batch data synchronization method of the present invention.

[0083] In some embodiments, the cross-database batch data synchronization system 200 of the present invention can be implemented in a combination of hardware and software. As an example, the cross-database batch data synchronization system 200 of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the cross-database batch data synchronization method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0084] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0085] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned cross-database batch data synchronization methods. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the cross-database batch data synchronization method shown in any embodiment of the present invention by calling the computer program.

[0086] In one alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0087] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0088] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0089] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0090] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0091] Among them, electronic devices can also be terminal devices. A terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0092] It should be noted that, Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0093] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described cross-database batch data synchronization methods.

[0094] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0095] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned cross-database batch data synchronization method.

[0096] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0098] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0099] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0100] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0101] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0102] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0103] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for batch data synchronization across databases, characterized in that, include: Create a database connection pool and identify the incremental data to be synchronized based on the update records of the source database; Based on the synchronization configuration information, and in combination with the current system load and the amount of incremental data, the data synchronization mode is determined; According to the data synchronization mode, allocate and initialize the corresponding thread resources; Based on the database connection pool and the thread resources, a batch data synchronization operation is performed from the source database to the target database; wherein, the batch data synchronization operation includes: writing the incremental data into the target database in a batch operation manner according to dynamically determined batch sizes; Once the batch data synchronization operation is complete, the thread resources and the database connection pool are reclaimed.

2. The cross-database batch data synchronization method according to claim 1, characterized in that, The step of identifying incremental data to be synchronized based on the update records of the source database includes: Based on the last synchronization time recorded in the source database, query the update records of the source database, where the update records contain a time identifier of the data change; From the update record, extract the data changes whose time is after the last synchronization time point, and determine the data changes as the incremental data.

3. The cross-database batch data synchronization method according to claim 1, characterized in that, The synchronization configuration information includes: a data volume threshold and a load threshold; the step of determining the data synchronization mode based on the synchronization configuration information, combined with the current system load and the data volume of the incremental data, includes: Determine whether the amount of incremental data is greater than the data amount threshold to obtain a first determination result, and determine whether the current system load is less than the load threshold to obtain a second determination result; When both the first and second judgment results are true, the data synchronization mode is determined to be a multi-threaded synchronization mode; otherwise, the data synchronization mode is determined to be a single-threaded synchronization mode.

4. The cross-database batch data synchronization method according to claim 3, characterized in that, The step of allocating and initializing the corresponding thread resources according to the data synchronization mode includes: When the data synchronization mode is the multi-threaded synchronization mode, the target number of threads required is calculated based on the amount of incremental data and the threshold for the amount of data processed by a single thread, and a thread pool is created and initialized; wherein, the thread pool contains the target number of threads; When the data synchronization mode is the single-threaded synchronization mode, a single thread is created and initialized.

5. The cross-database batch data synchronization method according to claim 4, characterized in that, The step of writing the incremental data into the target database in batches according to dynamically determined batch sizes includes: Based on the current system load, the batch size for writing to the target database is dynamically determined, and a database connection is obtained from the database connection pool. When the data synchronization mode is a multi-threaded synchronization mode, multiple threads in the thread pool are used to divide the incremental data into data batches that meet the batch size, and multiple data batches are written to the target database in a batch operation manner through the database connection. When the data synchronization mode is a single-threaded synchronization mode, the incremental data is divided into multiple data batches that conform to the batch size using the single thread, and each data batch is written to the target database in a batch operation manner through the database connection.

6. The cross-database batch data synchronization method according to claim 4 or 5, characterized in that, The step of reclaiming the thread resources and the database connection pool after the batch data synchronization operation is completed includes: After the batch data synchronization operation is completed and all data batches have been written to the target database, the thread pool or the individual thread is shut down. Release all database connections obtained through the database connection pool during the batch data synchronization operation back to the database connection pool.

7. The cross-database batch data synchronization method according to claim 5, characterized in that, The step of dynamically determining the batch size for writing to the target database based on the current system load includes: Based on the preset correspondence between load and batch size, the batch size corresponding to the current system load is determined.

8. A cross-database batch data synchronization system, characterized in that, include: The identification module is used to create a database connection pool and identify incremental data to be synchronized based on the update records of the source database. The determination module is used to determine the data synchronization mode based on the synchronization configuration information, combined with the current system load and the amount of incremental data; The configuration module is used to allocate and initialize the corresponding thread resources according to the data synchronization mode; The synchronization module is used to perform batch data synchronization operations from the source database to the target database based on the database connection pool and the thread resources; wherein, the batch data synchronization operation includes: writing the incremental data into the target database in a batch operation manner according to dynamically determined batch sizes; The recycling module is used to reclaim the thread resources and the database connection pool after the batch data synchronization operation is completed.

9. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the cross-database batch data synchronization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, implements the cross-database batch data synchronization method as described in any one of claims 1 to 7.

Images