Database synchronization method, device and equipment and computer readable storage medium

Abstract

The invention relates to the technical field of databases, in particular to a database synchronization method, device and equipment and a computer readable storage medium, and the method comprises the following steps: when a data synchronization request is received, obtaining a to-be-synchronized data table and attribute parameters in a database, determining to-be-synchronized data, reducing invalid transmission and processing amount, and reducing system resource consumption; whether the attribute parameters meet the synchronization conditions that the storage capacity, the real-time change frequency and the change data volume are all larger than respective corresponding thresholds is judged, target data are screened from the source, and the subsequent synchronization accuracy is guaranteed; if the parameters meet the conditions, a corresponding minute-level and hundred million-level data synchronization mode is obtained, and accurate matching of the data and the processing mode is achieved; and the data table is synchronized to the target database according to the mode, so that the target database and the source database are ensured to be completely consistent in hundred million-level data, and the timeliness of minute-level hundred million-level data synchronization is improved.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a database synchronization method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] As enterprises accelerate their digital transformation, Oracle databases, as the core of core business data storage, have made cross-database data synchronization the foundation for supporting key scenarios such as real-time business analysis and off-site disaster recovery.

[0003] Current mainstream Oracle data synchronization solutions, such as log mining tools or native data pump technology, have significant shortcomings in scenarios involving incremental synchronization of hundreds of millions of data points: completing the synchronization of hundreds of millions of data points generally takes 15-30 minutes, far exceeding the business's requirement for near real-time synchronization within 10 minutes, directly causing business data updates to lag and failing to support real-time business decisions.

[0004] This issue has a significant impact on time-sensitive industries such as finance and e-commerce. Real-time risk control in finance may miss the opportunity for risk warning due to data lag, and delayed synchronization of order status in e-commerce can easily lead to user disputes, hindering the efficient operation of enterprise business. There is an urgent need to solve the timeliness problem of synchronizing hundreds of millions of data points on a minute-by-minute basis.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a database synchronization method, apparatus, device, and storage medium, which aims to improve the timeliness of synchronizing hundreds of millions of data points on a minute-by-minute basis.

[0007] To achieve the above objectives, this application provides a database synchronization method, which includes:

[0008] When a data synchronization request is received, the data table to be synchronized and its attribute parameters are obtained from the source database. The attribute parameters include the storage capacity of the data table to be synchronized, the real-time change frequency of the data table to be synchronized, and the amount of changed data of the data table to be synchronized per unit time.

[0009] Determine whether the attribute parameters meet the preset synchronization conditions, where the preset synchronization conditions are that the storage capacity, real-time change frequency, and change data volume are all greater than their respective thresholds;

[0010] When the attribute parameter is detected to meet the preset synchronization condition, the preset synchronization method is obtained, wherein the preset synchronization method includes at least one of the multi-threaded synchronization method and the single-threaded synchronization method;

[0011] The data table to be synchronized is synchronized to the target database based on the preset synchronization method.

[0012] Furthermore, to achieve the above objectives, this application also provides a database synchronization device, which includes:

[0013] The data acquisition module is used to acquire the data table to be synchronized and its attribute parameters in the source database. The attribute parameters include the storage capacity of the data table to be synchronized, the real-time change frequency of the data table to be synchronized, and the amount of changed data of the data table to be synchronized per unit time.

[0014] The condition judgment module is used to determine whether the attribute parameters meet the preset synchronization conditions. The preset synchronization conditions are that the storage capacity, real-time change frequency, and change data volume are all greater than their respective thresholds.

[0015] The synchronization mode acquisition module is used to acquire the preset synchronization mode when the attribute parameter is detected to meet the preset synchronization conditions. The preset synchronization mode includes at least one of the multi-threaded synchronization mode and the single-threaded synchronization mode.

[0016] The data synchronization module synchronizes the data tables to be synchronized to the target database based on a preset synchronization method.

[0017] Each functional module of the database synchronization device of this application implements the steps of the database synchronization method of this application as described above during operation.

[0018] In addition, to achieve the above objectives, this application also provides a database synchronization device, which includes a memory, a processor, and a database synchronization program stored in the memory and executable on the processor. When the database synchronization program is executed by the processor, it implements the steps of the database synchronization method described above.

[0019] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, and stores a database synchronization program thereon. When the database synchronization program is executed by a processor, it implements the steps of the database synchronization method described above.

[0020] This application provides a database synchronization method. Upon receiving a data synchronization request, this method obtains the data table to be synchronized and its attribute parameters from the source database. By determining the data to be synchronized, it reduces invalid data transmission and processing, thus lowering system resource consumption. It then determines whether the attribute parameters meet preset synchronization conditions, where the preset conditions are that storage capacity, real-time change frequency, and changed data volume are all greater than their respective thresholds, thus filtering out minute-level and billion-level data from the source to ensure the accuracy of subsequent data synchronization. When the preset synchronization conditions are detected, a preset synchronization method is obtained, which includes at least one of multi-threaded and single-threaded synchronization methods to achieve precise matching between data and processing methods. Based on the preset synchronization method, the data table to be synchronized is synchronized to the target database, resulting in successful synchronization. This ensures that the target database and the source database have complete consistency in the billions of data points, thereby improving the timeliness of minute-level and billion-level data synchronization. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the first embodiment of the database synchronization method of this application;

[0022] Figure 2 This is a schematic diagram corresponding to the multi-threaded synchronization involved in this application;

[0023] Figure 3 This is a schematic diagram corresponding to the Kafka push involved in this application;

[0024] Figure 4 This is a schematic diagram of the database synchronization device involved in the embodiments of this application;

[0025] Figure 5 This is a schematic diagram of the database synchronization device involved in the embodiments of this application.

[0026] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0027] This application provides a database synchronization method, referring to... Figure 1 As shown, Figure 1 This is a flowchart illustrating the first embodiment of the database synchronization method of this application.

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0029] As enterprises accelerate their digital transformation, Oracle databases, as the core business storage carrier, require cross-database data synchronization as a crucial foundation for real-time business analysis and off-site disaster recovery. Currently, mainstream industry solutions fall into two categories: First, traditional log mining tools (such as Oracle GoldenGate) and native Data Pump. The former parses redo / undo logs, captures incremental changes, and pushes them to the target database, while the latter relies on an "export-transfer-import" process for data migration. Second, the CDC+Kafka model, commonly used in recent years, involves CDC (Change Data Capture) capturing changes and then transferring them to the target database via Kafka (Apache Kafka).

[0030] However, both solutions have significant shortcomings in high-volume, high-time-sensitivity scenarios: First, real-time performance is inadequate, with synchronization of tens of millions to hundreds of millions of changing data taking over 10 minutes—traditional solutions take 15-30 minutes for log parsing and file export, and Kafka, limited by partition throughput and consumption processing, also exceeds timeliness requirements, leading to fraud omissions and overselling risks in financial real-time risk control and e-commerce inventory management due to data lag. Second, stability is poor and data loss is common, with weak fault response capabilities—CDC struggles to locate lost data in network / node failures, and Kafka lacks a rapid degradation mechanism for consumption failures; traditional solutions rely on full retransmission after interruptions, wasting 2-3 times the resources and potentially overwriting business fields in the target database. Furthermore, both solutions lack proactive degradation and timely alerts, requiring manual intervention and extending the impact time. Third, accuracy verification is lacking; only data volume is compared without verifying field consistency and operational integrity, making it prone to hidden data problems due to format deviations, jeopardizing the accuracy of financial transaction reconciliation and e-commerce orders.

[0031] Therefore, there is an urgent need to build an efficient and stable solution that can adapt to the synchronization of tens of millions to hundreds of millions of data points per minute, and can solve problems such as insufficient real-time performance, difficulty in handling data loss, and lack of degradation alarms. This application provides a database synchronization method, device, equipment, and storage medium.

[0032] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, database system, etc., or a device capable of performing the above functions, such as a database synchronization device. The following description uses a database synchronization device as an example to illustrate this embodiment and the subsequent embodiments.

[0033] The database synchronization method of this application includes the following implementation steps S10 to S40.

[0034] Step S10: Upon receiving a data synchronization request, obtain the data table to be synchronized and its attribute parameters from the source database. The attribute parameters include the storage capacity of the data table to be synchronized, the real-time change frequency of the data table to be synchronized, and the amount of changed data in the data table to be synchronized per unit time.

[0035] In this embodiment, as an example, the application scenario targeted is the cross-database synchronization of fund / bond valuation table data of financial institutions.

[0036] In related technologies, financial institutions need to synchronize asset valuation data ranging from tens of millions to hundreds of millions of yuan from the trading database to the valuation analysis database after the market closes each day for the next day's net asset value (NAV) disclosure and risk assessment, requiring synchronization to be completed within 10 minutes. However, the traditional CDC+Kafka solution takes more than 20 minutes to synchronize, easily missing the NAV calculation window, and losing detailed valuation data in case of node failure. There is no automatic degradation mechanism, requiring manual full retransmission, which not only delays NAV disclosure but may also lead to investor confidence risks due to valuation data discrepancies. To address this scenario, the efficient and stable solution in this embodiment can achieve minute-level synchronization, accurately locate missing valuation entries through data sharding verification, automatically switch to the source and backup databases for full updates in case of failure, and trigger alarm notifications to maintenance personnel, ensuring the real-time performance and integrity of valuation data.

[0037] As an example, the application scenarios for data synchronization are not limited to the data synchronization of financial product information services mentioned above, but also include various other data synchronization scenarios, without specific limitations here.

[0038] In this embodiment, the data synchronization method is applied to the data synchronization device.

[0039] In this embodiment, a data synchronization request refers to a request issued by a user to synchronize data from a source database to a target database (hereinafter referred to as the "target database"). The data synchronization request contains data to be synchronized, and the data to be synchronized is obtained by parsing the data synchronization request.

[0040] Optionally, the attribute parameters of the data table to be synchronized are a set of information characterizing the core features of the data table that the current application needs to synchronize. Before starting data synchronization, these parameters must be obtained to clarify the synchronization adaptation rules and provide a basis for resource allocation and strategy selection. The attribute parameters can be determined based on the application's preset table registration configuration or the metadata statistics of the source database (hereinafter referred to as "source database"). The parameters are standardized definitions shared by the application, covering data table identification, data characteristics, and business adaptation requirements, providing parameter support for the efficiency and accuracy of synchronizing tens of millions to hundreds of millions of data points per minute.

[0041] Optionally, in data synchronization scenarios, especially in minute-level and billion-level data synchronization, the attribute parameters, including the storage capacity of the data table to be synchronized, the real-time change frequency of the data table to be synchronized, and the amount of changed data of the data table to be synchronized per unit time, are the core basis for synchronization strategy selection, resource allocation, and risk avoidance. Capacity determines the synchronization scale, change frequency determines the capture intensity, and the amount of changed data determines the transmission / write efficiency.

[0042] Optionally, before receiving a data synchronization request, a data synchronization task needs to be generated. The generation of the data synchronization task is determined based on business requirements and system capabilities. After the data synchronization task is generated, the task metadata (including task ID, sharding information, and data range) needs to be registered with a distributed task center (such as a coordination component based on ZooKeeper or Redis) and marked as "pending". This allows each instance of the application to be aware of the existence of the task through the center to build a globally visible task pool, providing a unified data source for multiple instances to compete for the task.

[0043] Step S20: Determine whether the attribute parameters meet the preset synchronization conditions, wherein the preset synchronization conditions are that the storage capacity, real-time change frequency, and change data volume are all greater than their respective thresholds;

[0044] In this embodiment, the system determines whether to preset synchronization conditions based on attribute parameters, so as to divide the data to be synchronized into minute-level and billion-level data synchronization data and non-minute-level and billion-level data synchronization data. This avoids performing redundant operations on data that does not meet the conditions and reduces the system's synchronization resource consumption. The preset synchronization conditions are a set of quantitative constraints predefined in the data synchronization system to determine whether the data to be synchronized meets specific synchronization requirements. The preset synchronization conditions are that the storage capacity, real-time change frequency, and change data volume are all greater than their respective thresholds.

[0045] Optionally, in core business scenarios such as Oracle cross-database synchronization, the preset synchronization conditions specifically refer to minute-level and billion-level data synchronization conditions. Minute-level and billion-level data synchronization conditions refer to quantitative constraints and technical adaptation requirements covering the entire chain of data capture, transmission, processing, and fault tolerance. They can be defined according to specific business scenarios. For example, minute-level and billion-level data synchronization conditions mean completing the cross-database / cross-system synchronization of billion-level data within ≤10 minutes.

[0046] Step S30: When the attribute parameter is detected to meet the preset synchronization condition, obtain the preset synchronization method, wherein the preset synchronization method includes at least one of the multi-threaded synchronization method and the single-threaded synchronization method;

[0047] In this embodiment, when the attribute parameters are detected to meet the preset synchronization conditions, the preset synchronization method is obtained as the synchronization method corresponding to the minute-level and billion-level data synchronization conditions. The preset synchronization method includes at least one of the multi-threaded synchronization method and the single-threaded synchronization method, so as to achieve accurate matching between data and processing method, avoid data anomalies caused by processing mismatch, and achieve timeliness and stability of minute-level and billion-level data synchronization.

[0048] Optionally, data under different conditions have different resource requirements, and corresponding processing can realize on-demand resource allocation, avoiding the waste of processing methods that consume hundreds of millions of data on low-demand data.

[0049] Optionally, in scenarios requiring the synchronization of hundreds of millions of data points within minutes, the preset synchronization method should be designed with "high throughput, low latency, and high reliability" as its core objectives. Stable synchronization of hundreds of millions of data points within ≤10 minutes should be achieved through the collaboration of technical components and process optimization. Synchronization methods corresponding to the minute-level hundreds of millions of data point synchronization conditions include, but are not limited to, log-based incremental capture, distributed parallel transmission, full data sharding parallel synchronization, and fault-tolerant and breakpoint resume methods.

[0050] Step S40: Synchronize the data table to be synchronized to the target database based on the preset synchronization method.

[0051] In this embodiment, by synchronizing the data table to be synchronized to the target database based on a preset synchronization method, it is possible to directly ensure that the target database and the source database have complete consistency of hundreds of millions of data, eliminating the risk of data loss and errors.

[0052] Optionally, in a data synchronization scenario, obtaining a successful synchronization result is the core closed-loop point of the entire synchronization process. It not only completes data transmission, but also verifies the effectiveness of all preceding technical means through the confirmation of the successful result, ultimately achieving the core goal of stable, accurate, and timely synchronization of hundreds of millions of data points at the minute level.

[0053] In this embodiment, upon receiving a data synchronization request, the system obtains the data table to be synchronized and its attribute parameters from the source database. By determining the data to be synchronized, invalid data transmission and processing are reduced, thus lowering system resource consumption. The system then determines whether the attribute parameters meet preset synchronization conditions, where the preset conditions are that storage capacity, real-time change frequency, and changed data volume are all greater than their respective thresholds. This filters out minute-level and billion-level data from the source, ensuring the accuracy of subsequent data synchronization. When the preset synchronization conditions are met, a preset synchronization method is obtained. This preset synchronization method includes at least one of multi-threaded and single-threaded synchronization methods, achieving precise matching between data and processing methods. Based on the preset synchronization method, the data table to be synchronized is synchronized to the target database, resulting in successful synchronization. This ensures that the target database and the source database have complete consistency in the billions of data points, thereby improving the timeliness of minute-level and billion-level data synchronization.

[0054] Furthermore, based on the above, a second embodiment of the database synchronization method of this application is proposed. In some feasible embodiments, step S20 above, which determines whether the attribute parameters meet preset synchronization conditions, wherein the preset synchronization conditions are that the storage capacity, real-time change frequency, and change data volume are all greater than their respective thresholds, also includes the following implementation steps B201 to B203.

[0055] Step B201: Compare the storage capacity with the preset capacity threshold, compare the real-time change frequency with the preset frequency threshold, and compare the changed data volume with the preset data volume threshold.

[0056] Step B202: If the storage capacity is greater than the preset capacity threshold, the real-time change frequency is greater than the preset frequency threshold, and the amount of changed data is greater than the preset data amount threshold, then it is determined that the preset synchronization conditions are met.

[0057] Step B203: If the storage capacity is less than or equal to the preset capacity threshold, and / or the real-time change frequency is less than or equal to the preset frequency threshold, and / or the changed data volume is less than or equal to the preset data volume threshold, then the attribute parameters are determined not to meet the preset synchronization conditions.

[0058] In this embodiment, before performing the comparison, quantified attribute parameters need to be obtained from the source database metadata through the data interface. The parameter definitions, units, and acquisition methods need to be clearly defined. Specifically, storage capacity refers to the physical storage size occupied by the data table to be synchronized in the source database (such as an Oracle database); real-time change frequency refers to the average number of data changes (including additions, modifications, and deletions) that occur in the data table to be synchronized within each preset time period; changed data volume refers to the total number of change records in the data table to be synchronized within the preset time period (the sum of the number of additions, modifications, and deletions).

[0059] Optionally, for example, the preset capacity threshold is 100W, where W represents 10,000 data records, the preset frequency threshold is 0 to determine whether the change frequency is fixed or not, and the preset data volume threshold is 10W of daily changed data.

[0060] Optionally, by comparing the storage capacity with a preset capacity threshold, the real-time change frequency with a preset frequency threshold, and the changed data volume with a preset data volume threshold, technical support is provided for accurately matching the business data synchronization requirements and synchronization methods in the future, ultimately achieving a real-time, stable, efficient, and low-cost synchronization effect.

[0061] Optionally, in a scenario of synchronizing hundreds of millions of data points per minute, if the storage capacity is greater than a preset capacity threshold, the real-time change frequency is greater than a preset frequency threshold, and the amount of changed data is greater than a preset data amount threshold, then the preset synchronization conditions are met. In other words, the storage capacity, real-time change frequency, and amount of changed data per unit time of the data table to be synchronized are all greater than the corresponding preset thresholds for synchronizing hundreds of millions of data points per minute, providing a reliable basis for starting the subsequent high-performance synchronization process.

[0062] Optionally, in minute-level or billion-level data synchronization scenarios, if the storage capacity is less than or equal to a preset capacity threshold, and / or the real-time change frequency is less than or equal to a preset frequency threshold, and / or the changed data volume is less than or equal to a preset data volume threshold, then the attribute parameters are determined not to meet the preset synchronization conditions. In this case, a non-minute-level or billion-level data synchronization method is called to achieve precise adaptation to a lightweight synchronization solution, thereby avoiding the resource waste and risks of high-performance methods and ensuring that synchronization requirements are met.

[0063] Optionally, lightweight synchronization solutions include, but are not limited to, a synchronization scheme that combines timed full table comparison and capture, local file transfer, and single-batch writing to the target database.

[0064] In this embodiment, by comparing the storage capacity with a preset capacity threshold, the real-time change frequency with a preset frequency threshold, and the changed data volume with a preset data volume threshold, accurate identification of data characteristics is achieved, avoiding fuzzy adaptation. If the storage capacity is greater than the preset capacity threshold, the real-time change frequency is greater than the preset frequency threshold, and the changed data volume is greater than the preset data volume threshold, it is determined that the preset synchronization conditions are met, providing technical support for triggering high-performance synchronization and ensuring the stable landing of hundreds of millions of data points per minute. If at least one of the storage capacity, real-time change frequency, and changed data volume is not greater than the corresponding preset threshold, it is determined that the preset synchronization conditions are not met, providing technical support for triggering lightweight synchronization and avoiding resource waste and business mismatch.

[0065] Furthermore, based on the above, in some feasible embodiments, the preset synchronization method includes a multi-threaded synchronization method. The step S30 above, which involves obtaining the preset synchronization method when the attribute parameter is detected to meet the preset synchronization conditions, further includes the following implementation steps B301-B304.

[0066] Step B301: When the attribute parameter is detected to meet the preset synchronization conditions, the business attribute characteristics of the attribute parameter are obtained. The business attribute characteristics include at least one of industry adaptation characteristics and timeliness control characteristics.

[0067] In this embodiment, when it is determined that the attribute parameters of the data table to be synchronized meet the preset synchronization conditions for synchronizing data at the minute level and the level of hundreds of millions of data, the attribute parameters of the data table to be synchronized are parsed to obtain the corresponding business attribute features. This provides technical support for decomposing the database to be synchronized into multiple synchronization tasks and improving the efficiency of synchronizing data at the minute level and the level of hundreds of millions of data.

[0068] Optionally, in scenarios involving minute-level or billion-level data synchronization, the business dimension of the data table to be synchronized refers to the set of business attributes that determine the adaptability of the data synchronization strategy, such as industry type, data sensitivity level, and business real-time requirements. By obtaining business attribute characteristics based on these dimensions, the synchronization strategy can be accurately matched with the business scenario, avoiding the disconnect between the technical solution and actual business needs.

[0069] Optionally, the business attribute features include at least one of industry adaptation features and timeliness control features. The business attribute features include, but are not limited to, business data type tags, industry compliance identifiers, and industry-specific scenario markers. The timeliness control features include, but are not limited to, synchronization timeliness level tags, latency tolerance thresholds, and real-time priority identifiers, to guide the adaptation of synchronization strategies.

[0070] Step B302: Divide the data table to be synchronized according to the business attribute characteristics to obtain multiple independent synchronization subtasks;

[0071] Optionally, the database of data tables to be synchronized can be divided according to business attribute characteristics to obtain multiple independent synchronization subtasks, which provides technical support for improving the timeliness of synchronization by implementing multi-threaded and multi-deployment synchronization methods.

[0072] Optionally, taking a "real-time transaction log" in the financial industry as an example, this log has a capacity of 105GB (112 million records) and contains two types of data: "highly sensitive + extremely high real-time (transfer transactions, latency ≤ 5 minutes)" and "medium sensitive + medium real-time (refund transactions, latency ≤ 10 minutes)". The business partitioning module first filters the feature fields, splitting the transfer transactions into 16 shards based on "transaction ID hash value % 16" (each shard approximately 7 million records, ensuring no business dependency in a single shard), and splitting the refund transactions into 8 shards based on "transaction ID hash value % 8" (each shard approximately 1.4 million records). Subsequently, the thread scheduling module allocates 16 high-priority threads (priority = 1, single-thread capture efficiency 12,000 records / second, total efficiency 192,000 records / second) to the transfer transaction shards, and to the refund transaction shards. Eight medium-priority threads were allocated (priority = 2, single-thread capture efficiency of 8,000 records / second, total efficiency of 64,000 records / second) to avoid low-priority threads consuming core resources. The deployment management module also sharded transfer transactions onto two synchronization nodes (each carrying eight shards) to avoid CPU overload caused by 16 threads on a single node (16-core CPU single node utilization of 75%). Refund transactions were sharded onto one node (8 threads utilization of 50%). Ultimately, the time taken for transfer transaction synchronization was reduced from 18 minutes with traditional single-threaded synchronization to 4.5 minutes, and the time taken for refund transactions was reduced from 18 minutes to 8 minutes. The total time taken for full table synchronization was 8 minutes (≤10-minute window), and the timeout rate was reduced from 100% to 0. The core of this approach is that after partitioning, multiple threads operate without waiting, and multiple deployments are deployed without overload, achieving efficient parallelism.

[0073] Step B303: Compare the number of synchronous subtasks with the preset single-threaded processing threshold;

[0074] Step B304: If the number of synchronized subtasks is greater than the preset single-thread processing threshold, then the preset synchronization method is determined to be a multi-threaded synchronization method. The number of threads in the multi-threaded synchronization method is determined based on the number of synchronized subtasks and the processing capacity of each single thread corresponding to the multi-threaded synchronization method.

[0075] In this embodiment, the number of synchronized subtasks is compared with a preset single-thread processing threshold. When it is determined that the number of synchronized subtasks is greater than the preset single-thread processing threshold, the preset synchronization method is determined to be a multi-threaded synchronization method. This achieves precise allocation of thread resources by dynamically matching the number of subtasks with the thread processing capacity. This avoids synchronization timeouts caused by insufficient single-thread processing capacity and prevents the idle waste of multi-thread resources when there are too few subtasks, ultimately ensuring the synchronization efficiency of hundreds of millions of data points per minute.

[0076] Optionally, the number of threads in the multi-threaded synchronization method is determined based on the number of synchronization subtasks and the processing capacity of each single thread corresponding to the multi-threaded synchronization method. Here, the synchronization subtask refers to the independent data fragments formed after the data table to be synchronized is divided according to the business dimension. Each subtask contains a fixed amount of data and needs to complete the entire process of capture, transmission, and writing within the synchronization window. The preset single-threaded processing threshold is the maximum subtask capacity determined based on the single-threaded hardware performance and the synchronization window. For example, based on historical data calculations, a single thread can stably process up to 5 subtasks within 10 minutes (6.68 million records per subtask, with a total processing volume of approximately 33.4 million records), so the threshold is set to 5. This threshold will be dynamically updated according to hardware iterations (such as upgrading to a 2-core CPU (Central Processing Unit)) or synchronization window adjustments (such as shortening to 8 minutes), and is stored and calibrated by the system's threshold configuration module.

[0077] Optionally, if the number of synchronized subtasks is less than or equal to the preset single-threaded processing threshold, single-threaded synchronization can be enabled directly to avoid the additional overhead caused by multi-threaded scheduling.

[0078] In this embodiment, by acquiring business attribute characteristics, a basis for dividing the data table to be synchronized is provided. By dividing the data table to be synchronized, multiple independent synchronization subtasks are obtained, and the number of synchronization subtasks is compared with a preset single-thread processing threshold. When the number of synchronization subtasks is greater than the preset single-thread processing threshold, the thread resources are accurately allocated, and the multi-thread synchronization method is determined. This breaks through the limitations of the traditional fixed number of threads and realizes dynamic adaptation of the number of threads. This is a key technical feature to support the efficiency and stability of minute-level and billion-level data synchronization.

[0079] Furthermore, based on the above, in some feasible embodiments, before step S40, the following implementation steps B50 to B70 are also included.

[0080] Step B50: After receiving the instruction to perform synchronization for the synchronization subtask, initiate a request to the preset distributed resource locking unit to acquire the distributed lock corresponding to the synchronization subtask;

[0081] Step B60: Upon detecting that the distributed lock has been successfully acquired, execute the step of synchronizing the data table to be synchronized to the target database based on the preset synchronization method;

[0082] Step B70: When the information of failure to acquire the distributed lock is detected, determine the next data table to be synchronized in the source database, and perform the steps of acquiring the data table to be synchronized and its attribute parameters in the source database based on the next data table to be synchronized.

[0083] In this embodiment, after receiving the instruction to synchronize the execution of the synchronized subtask, a request is sent to the preset distributed resource locking unit to acquire the distributed lock corresponding to the synchronized subtask. The distributed lock is used to identify the processing status of the synchronized subtask and prevent multiple threads from repeatedly executing the same synchronized subtask.

[0084] Optionally, upon detecting that the distributed lock has been successfully acquired, the step of synchronizing the data table to be synchronized to the target database based on a preset synchronization method is executed to ensure the timeliness of synchronizing hundreds of millions of data.

[0085] Optionally, after the synchronization request of the synchronization subtask is triggered, the lock request module of the synchronization node will determine the lock based on the unique identifier of the synchronization subtask (such as "table ID_shard ID (Identifier)").

[0086] The FIN_TRADE_001_008 key is used to generate a lock key and send a lock request to a pre-defined distributed resource locking unit (such as Redisson lock or ZooKeeper distributed lock based on Redis). The lock service will check whether the lock key is already held by another node. If it is already held, it will return failure; otherwise, it will grant the lock. The lock validity period is set to 1.5 times the estimated time of the synchronization subtask. For example, if the subtask is estimated to complete in 5 minutes, the lock validity period is 7.5 minutes to avoid the lock being permanently occupied due to node failure.

[0087] Optionally, upon detecting a failure to acquire a distributed lock, the "Lock Failure Reason Analysis Module" determines the failure type and then executes a differentiated processing strategy accordingly: If the failure reason is "the synchronization subtask has been locked," the processing logic focuses on "conflict avoidance and timeliness assurance": First, the synchronization node records the "lock failure timestamp" and "current lock-holding node ID" of the synchronization subtask in its local cache to avoid repeatedly sending lock requests within a short period of time; Second, the retry strategy is dynamically adjusted based on the business priority of the synchronization subtask (obtained from the business attribute feature library, such as high-sensitivity + extremely high real-time sub-data having a priority of P0, and low-sensitivity + low real-time having a priority of P2). — For P0-level synchronization subtasks, immediately after the cooldown period, initiate the second and third lock requests (maximum of 3 retries, total time ≤ 30 seconds, controlled within a 10-minute synchronization window). If all 3 attempts fail, send a "synchronization subtask lock preemption request" to the "synchronization task scheduling center". The scheduling center will coordinate the node currently holding the lock to prioritize the synchronization of the P0-level synchronization subtask. For P2-level synchronization subtasks, do not retry immediately, but add them to the "off-peak synchronization queue". Initiate the request again after the current lock is released or during non-peak business periods to avoid competing for lock resources with P0 / P1-level synchronization subtasks, while not affecting the synchronization timeliness of core synchronization subtasks.

[0088] If the failure is due to "distributed lock service anomaly," the handling logic focuses on "fault degradation and data security": First, the synchronization node immediately switches to the "local backup lock mechanism"—writing the lock key of the synchronization subtask to the "backup lock file" on the local disk, and simultaneously sending an alarm to the operation and maintenance monitoring module (such as SMS or log alarms, indicating lock service anomaly); Second, based on the local backup lock, it determines whether the synchronization subtask can be synchronized: if the local backup lock is not held by the current node, it performs avoidance according to the logic of "synchronization subtask is locked"; if the local backup lock can be successfully held, the synchronization subtask is started, but during the synchronization process, an additional "data consistency verification mechanism" will be enabled to avoid potential conflicts caused by the inability of the local lock to be verified across nodes; Third, after the distributed lock service recovers, the synchronization node will synchronize the state of the local backup lock to the distributed lock service to ensure global consistency of the lock state and avoid misjudgment in subsequent synchronization tasks.

[0089] Optionally, by deeply binding "lock failure handling" with "minute-level timeliness" and "synchronization subtask business priority", it differs from the traditional fixed retry strategy that does not distinguish business scenarios. This not only ensures the real-time synchronization needs of high-priority synchronization subtasks, but also avoids invalid retries of low-priority synchronization subtasks. At the same time, the fault degradation mechanism ensures that the synchronization task is not interrupted in extreme cases, forming a complete closed loop of "conflict avoidance - priority adaptation - fault degradation". This is a key technical design for balancing "conflict prevention" and "timeliness" in minute-level and billion-level data synchronization scenarios.

[0090] Optionally, when information indicating a failure to acquire the distributed lock is detected, a minute-level timeout requirement is also matched to avoid excessive waiting leading to timeout of sub-data synchronization. For example, a "remaining lock duration + 30-second buffer period" (e.g., 2 minutes + 30 seconds = 2 minutes and 30 seconds) is set to ensure that the lock has been released after waiting and that sufficient time is reserved to complete synchronization, thus avoiding failure of secondary requests.

[0091] Optionally, when the information of failure to acquire the distributed lock is detected, the next data table to be synchronized in the source database is determined, and the steps of acquiring the data table to be synchronized and its attribute parameters in the source database are executed based on the next data table to be synchronized. In order to maximize the utilization of synchronization resources in the distributed competition scenario through the "dynamic flow and parallel preparation" mechanism, while ensuring the continuity and efficiency of minute-level and billion-level data synchronization.

[0092] Optionally, for example, if the "Marketing Campaign Table" fails to acquire all instances due to a locking mechanism issue, the instances will automatically redirect to the "User Behavior Table" or "Coupon Table," ensuring that core business data (such as user behavior and coupon status) can still be synchronized on time. This reduces the impact of the anomaly from "full blockage" to "single table delay," thus meeting the rigid demand for "high availability" in synchronizing hundreds of millions of data points.

[0093] Optionally, before initiating a request to the preset distributed resource locking unit to acquire the distributed lock corresponding to the synchronization subtask, it is determined whether the current data is being synchronized. If it is being synchronized, the process waits for the data synchronization to complete; otherwise, it initiates a request to the preset distributed resource locking unit to acquire the distributed lock corresponding to the synchronization subtask. Specifically, the mechanism determines whether the current data is being synchronized by querying the data synchronization status identifier. This determination mechanism relies on real-time updated status records to identify the data processing status in advance, avoiding repeated lock requests for data that is being synchronized, reducing invalid concurrent contention from the source, and preventing update conflicts caused by multiple processes processing the same data simultaneously.

[0094] Optionally, before performing the synchronization operation on the data table to be synchronized, it is determined whether the data to be synchronized has already been synchronized in the target database. Optionally, by determining whether the amount of data to be synchronized in the source database and the target database is consistent, and whether the corresponding update time is consistent, if both the amount of data to be synchronized and the update time are consistent, then no synchronization is required and the corresponding synchronization operation is terminated. If either the amount of data to be synchronized or the update time is inconsistent, then the multi-threaded synchronization operation is continued to be performed, thereby avoiding data duplication in the target database and reducing unnecessary resource consumption.

[0095] In this embodiment, by sending a request to the distributed lock and flexibly executing the corresponding synchronization strategy according to the request result, it not only prevents sub-data synchronization conflicts, but also ultimately ensures the global timeliness and data consistency of minute-level and billion-level data synchronization; through dynamic flow and parallel preparation mechanisms, it ensures the continuity and efficiency of minute-level and billion-level data synchronization.

[0096] Furthermore, based on the content of any of the above embodiments, in some feasible embodiments, after step B40, the following implementation steps C41-C44 are also included.

[0097] Step C41: When a failure message of multi-threaded synchronization is detected, trigger the synchronization mode switching mechanism to switch the multi-threaded synchronization mode to the source database backup full update mode;

[0098] Step C42: Obtain the current full data of the data table to be synchronized from the backup database of the source database. The current full data includes all record files in the data table to be synchronized and the latest status of each record file.

[0099] Step C43: Perform a full comparison between the current full data and the existing data in the corresponding data table of the data table to be synchronized in the target database, and determine the difference data based on the full comparison results;

[0100] Step C44: Perform a full update operation on the target database based on the difference data to obtain a successful synchronization result.

[0101] In this embodiment, in the scenario of synchronizing hundreds of millions of data points on a minute-by-minute basis and multi-threaded synchronization, when a synchronization failure is received, a synchronization mode switching mechanism is triggered to switch the multi-threaded synchronization mode to the source database backup full update mode, and execute the synchronization operation corresponding to the source database backup full update mode. The core purpose is to avoid synchronization interruption or timeout caused by single-thread / partial thread failure or main source database abnormality in the high-complexity scenario of multi-threaded parallel processing of hundreds of millions of data points through "rapid fault detection + automatic backup source switching", ultimately ensuring the achievement of the core goal of synchronizing hundreds of millions of data points on a minute-by-minute basis.

[0102] In this example, the full update method for the source database backup includes obtaining the current full data of the data table to be synchronized from the backup database of the source database. The current full data includes all record files in the data table to be synchronized and the latest status of each record file. The current full data is then compared with the existing data of the corresponding data table in the target database. The difference data is determined based on the comparison results. A full update operation is then performed on the target database based on the difference data to obtain a successful synchronization result. In the scenario of synchronizing hundreds of millions of data points at the minute level, the application to be synchronized and the target synchronization application involved in the multi-threaded interface synchronization are respectively associated with the source database and the target database.

[0103] like Figure 2 As shown, Figure 2 This diagram illustrates the multi-threaded synchronization involved in this application. The source and target databases each contain a primary and a backup database. The primary database serves as the core read / write node, carrying real-time business operations, while the backup database is in a hot standby state, synchronizing data with the primary database in real time. During normal synchronization, the application to be synchronized retrieves changed data from the primary database using multiple threads and sends it to the target synchronization application via an interface. The target synchronization application receives, verifies, and writes the data to the primary database. The backup database synchronizes with the primary database, and the backup database synchronizes with the primary database. The backup database does not participate in the normal process but only serves as redundancy. When interface synchronization fails, including due to sending timeout, receiving verification failure, or writing to the primary database timeout, the failure detection module of the target synchronization application determines the failure through thread heartbeats and data receipts, recording the failed fragment identifier, the last successful synchronization timestamp, and the amount of failed data. At this point, the target synchronization application does not rely on the application to be synchronized to retry. Instead, it actively sends requests to the source and backup databases to obtain incremental change data for the corresponding failed shards based on the recorded timestamps. The source and backup databases return the latest data consistent with the primary database through a parallel query interface. After receiving the data, the target synchronization application prioritizes writing it to the target primary database. If the primary database fails, it temporarily writes it to the target backup database. At the same time, it enables parallel processing with threads matching the number of failed shards. Combined with data compression and dedicated bandwidth control, it ensures that the minute-level synchronization requirement is met. The backup database data is pulled by the target end, and the primary and backup redundant resources are used to ensure timely synchronization of the changed data.

[0104] Optionally, in scenarios involving minute-level or billion-level data synchronization, the current full data of the data table to be synchronized is obtained from the backup database of the source database. The current full data includes all record files in the data table to be synchronized and the latest status of each record file.

[0105] Optionally, it should be noted that all record files of the data to be synchronized refer to all record files that need to be transmitted within the current synchronization cycle of the data table to be synchronized, and must include all existing and incremental data related to business in the table without omission; the latest status of the record files refers to the latest business attributes and data validity status of these record files at the time of synchronization triggering, and must be ensured to be completely consistent with the latest data status of the source database (especially the backup database, which must be aligned with the primary database in real time). The two together constitute the basis for the integrity and timeliness of the synchronized data, in order to meet the accuracy requirements of synchronizing hundreds of millions of data.

[0106] Optionally, in scenarios involving minute-level or billion-level data synchronization, the full comparison of the data to be synchronized with the corresponding data table in the target database requires first defining the comparison scope as all record files of the data to be synchronized and the data table in the target database. The comparison dimensions cover unique data identifiers (such as primary keys and business unique codes) and core business fields (such as status fields, numeric fields, and timestamp fields). Based on the full comparison results, the difference data is obtained to ensure that each record file can be accurately located and the field status can be verified.

[0107] During the comparison process, to adapt to the requirements of hundreds of millions of data points and minute-level timeliness, a sharded parallel comparison strategy is adopted. The data to be synchronized and the data in the target database are split into equal shards according to the unique identifier of the data or the business dimension. Multi-threaded comparison operations are used to perform comparison operations on each shard separately to avoid excessive time consumption caused by full table scans. During the comparison, the data to be synchronized and the data in the target database are first matched by the unique identifier, and then the core field values ​​are checked field by field to see if they are consistent. If the unique identifier does not exist in the target database, or exists but the core field values ​​are inconsistent, the record file is marked as the difference data that needs to be synchronized. The difference data includes, but is not limited to, newly added, modified, and deleted data, thereby providing technical support for improving the stability of minute-level hundreds of millions of data synchronization.

[0108] Optionally, when performing a full update based on the selected difference data, multiple threads are allocated according to the shard dimension, with each thread corresponding to a data shard. The data to be synchronized is written to the target database in batch writing mode. The batch size needs to be dynamically adjusted in combination with the target database's IO (Input / Output) performance and the number of threads to avoid IO overload caused by an excessively large batch or an excessively small batch that increases the number of interactions.

[0109] Optionally, transaction control is enabled during the synchronization process to ensure that all data to be synchronized within a single shard is either written successfully or rolled back completely, preventing data fragmentation in the target database caused by partial data updates succeeding while partial failures occur. At the same time, the synchronization operation will prioritize writing to the target database's primary database. If the primary database experiences a temporary failure, the writing will be switched to the target database's backup database. Once the primary database recovers, the data will be supplemented through the primary-backup synchronization mechanism, ensuring that the synchronization process is uninterrupted and thus guaranteeing the stability of minute-level and billion-level data synchronization.

[0110] Optionally, upon detecting a successful synchronization, the synchronization result is output based on a preset method to ensure that the result can be automatically parsed, manually traced, and meets business monitoring requirements. It should be noted that the preset output method follows predefined rules, formats, and channels, adhering to the principles of "field standardization" (uniform field names and data types for similar results), "granularity adaptation" (e.g., outputting detailed data by shards for hundreds of millions of data points while simultaneously summarizing all results), and "timeliness priority" (e.g., outputting within 10 seconds of synchronization completion). This ensures that the result supports both automated processes and manual investigation needs. The preset method includes, but is not limited to, sending SMS messages to the corresponding investigator's mobile phone or displaying text indicating synchronization success or failure on the synchronization system's interface.

[0111] In this embodiment, in the event of multi-threaded synchronization failure, a full update of the backup database is adopted. Specifically, all discrepancies are located by full comparison to avoid data loss or corruption in the target database caused by partial synchronization failure. The full update of the target database is achieved based on the discrepancies, thus ensuring the timeliness of synchronization, thereby improving the system's fault tolerance, significantly improving the recovery success rate, accuracy and stability after synchronization failure, and reducing the business impact caused by single point of failure.

[0112] Furthermore, based on the content of any of the above embodiments, in some feasible embodiments, the above database synchronization method, after step S20, further includes the following implementation steps D70 to D100.

[0113] Step D70: When the attribute parameters are detected to not meet the preset synchronization conditions, poll the update time of the data table to be synchronized in the source database at preset time intervals, and filter out the record files whose update time is within the current polling period;

[0114] In this embodiment, when the attribute parameters are detected to not meet the preset synchronization conditions, the update time of the data table to be synchronized in the source database is polled according to the preset time period and the record files within the current polling period are filtered to determine that the parameters of the data table to be synchronized are not lightweight data that does not meet the minute-level and billion-level data synchronization scenario. This avoids the waste of resources caused by full synchronization, focuses only on the updated data within the period, reduces the reading pressure of the source database and the amount of data transmission, and adapts to the low-load synchronization requirements when the parameters are not met.

[0115] Optionally, the polling period should be preset based on the business real-time requirements of the data to be synchronized (e.g., a 5-minute interval in low-change-frequency scenarios) and the performance carrying capacity of the source database. This avoids the source database IO from being overloaded frequently due to too short an interval or the data synchronization delay caused by too long an interval. This achieves a balance between source database resource consumption and synchronization real-time performance, ensuring that the source database does not need to continuously respond to synchronization requests when the parameters do not meet the preset synchronization conditions, thus reducing unnecessary performance loss.

[0116] Optionally, filtering record files within the current polling period should be based on the update time field of the data table, using precise SQL (Structured Query Language) conditions (such as...).

[0117] The WHEREUPDATE_TIMEBETWEEN period (start and end times) is used to locate newly added and modified records within the period, excluding data that has not been updated. This significantly reduces the amount of data to be synchronized, avoids redundant data transmission caused by full table scans, and reduces the pressure on subsequent encapsulation, Kafka push, and target database retrieval. It adapts to the lightweight synchronization requirements of scenarios with parameters that do not meet the needs of synchronizing hundreds of millions of data points per minute, ensuring that the synchronization process is efficient and low-cost.

[0118] Step D80: Mark the operation type of the selected record files to obtain the marked record files;

[0119] In this embodiment, the selected record files are marked with operation types. Specifically, the mark is made by distinguishing between record file addition, deletion, and modification operations and the corresponding update time identification rules. This allows the target database to clearly understand the processing method of each record file, avoid blind writing or misoperation, reduce the error rate of data processing in the target database, and ensure synchronization accuracy.

[0120] Step D90: Encapsulate the marked record file to obtain the encapsulated record file;

[0121] Optionally, the marked record file is encapsulated according to a preset data format, which includes at least: an operation type field, a data identifier field, a source database identifier field, an operation timestamp field, and a record file content field. For example, the operation type field includes, but is not limited to, List. <t>, String; Data identifier fields include, but are not limited to, deleted records, newly added and updated records, and table names: Therefore, by encapsulating the marked records and defining a unified data encapsulation format, the format of Kafka message push is defined, so that the records are adapted to the Kafka transmission specification and the target library parsing logic, eliminating transmission interruption or parsing failure caused by format incompatibility, and ensuring that the integrity of the data is not compromised during transmission.

[0122] Step D100: Push each encapsulated log file as a message to the target database.

[0123] In this embodiment, Kafka is used to push each encapsulated record as a message to the target database to complete the synchronization operation. Based on Kafka's asynchronous synchronization link, Kafka's high throughput and message caching capabilities are used to alleviate the pressure of direct interaction between the source and target databases, avoid synchronization congestion during data peaks, and support on-demand subscription and retrieval by the target database to ensure the stability and orderliness of synchronization in scenarios where parameters do not meet the requirements of minute-level or billion-level data synchronization.

[0124] Optionally, the encapsulated log file can be pushed as a message to a preset Kafka topic. This preset Kafka topic is a "classification container" in the chain specifically used to carry messages to be synchronized. The topic needs to be defined in advance according to synchronization requirements (such as data table type, business priority, sharding dimension) to ensure that the messages pushed by the source end can be classified according to the rules, and the target application end can accurately subscribe to the corresponding topic to obtain the required data.

[0125] Optionally, data can be pushed via Kafka, causing the target application to be temporarily busy. Data can be temporarily stored in a Kafka topic and retrieved after the target application recovers, thus decoupling the source and target ends. After receiving the data, the target application updates according to the primary key. The primary key serves as a unique identifier for the data and can accurately locate the record file in the target database that needs to be updated, avoiding erroneous updates due to incorrect field matching. Furthermore, when updating by primary key, transaction control can be combined to ensure the accuracy of data modification, guarantee the consistency between the data in the target database and the updated data in the source database, and avoid duplicate writes or data corruption.

[0126] Optionally, Kafka can be used to push the encapsulated record files as messages to the target database. The data in the target database is updated in the following order: first, record files marked as deletion operations are processed; then, record files marked as addition operations are processed; and finally, record files marked as update operations are processed. This completely avoids data operation conflicts, ensures that the data in the target database is aligned with the state of the source database in real time, and improves the timeliness of data synchronization.

[0127] Optionally, when pushing changed data via Kafka, a threshold for the data volume of each batch of sub-messages should be preset based on Kafka's single message processing capacity and the average size of a single record file. This provides a quantitative standard for batching, preventing batches from being too large and exceeding Kafka's capacity, or too small and increasing the number of interactions, thus laying the foundation for reasonable batching. Subsequently, the marked record files are traversed, counted by cumulative size or quantity, and when the threshold is reached, the current batch is completed and a new batch is started. Simultaneously, record files from the same business unit are grouped into one batch, thereby controlling the data volume of a single batch within a safe range and avoiding subsequent processing chaos caused by splitting business-related data across batches. Next, each batch of record files is encapsulated into sub-messages using a standardized format. This ensures that the sub-messages are compatible with the Kafka transmission protocol and the target application's parsing logic, eliminating push failures or parsing errors caused by format incompatibility. Finally, the messages are pushed sequentially to the Kafka topic in batches, with an ACK (Acknowledgment) confirmation mechanism enabled. In case of failure, only the current batch is retried, which significantly reduces the push failure rate caused by large single messages. Furthermore, batch retries reduce retransmission costs, ensuring the stability, accuracy, and efficiency of pushing hundreds of millions of data points.

[0128] In this embodiment, for scenarios where minute-level or billion-level data synchronization is not feasible, the update time of the data table to be synchronized in the source database is polled according to a preset time period, and the record files within the current polling period are filtered. This determines that the parameters of the data table to be synchronized are lightweight data that does not meet the requirements of minute-level or billion-level data synchronization, avoiding the resource waste caused by full synchronization. By marking the operation type of the filtered record files, outdated data is excluded, significantly reducing the volume of data to be synchronized. By encapsulating the marked record files according to a preset data format, the record files are made compatible with the Kafka transmission specification and the target database parsing logic, ensuring that the data integrity is not compromised during transmission. Data is pushed through Kafka, enabling the target database to subscribe to and pull messages from a preset Kafka topic, achieving consistency between the target database data and the updated data in the source database. This ensures the stability, accuracy, and orderliness of synchronization even when the parameters do not meet the requirements of minute-level or billion-level data synchronization.

[0129] Furthermore, based on the content of any of the above embodiments, in some feasible embodiments, the above database synchronization method, step D100 further includes steps E10-E40.

[0130] Step E10: When a push failure message is detected, trigger the synchronization mode switching mechanism to switch the push synchronization mode to the source database backup full update mode;

[0131] Step E20: Obtain the current full data of the data table to be synchronized from the backup database of the source database. The current full data includes all record files in the data table to be synchronized and the latest status of each record file.

[0132] Step E30: Perform a full comparison between the current full data and the existing data in the corresponding data table of the data table to be synchronized in the target database, and determine the difference data based on the full comparison results;

[0133] Step E40: Perform a full update operation on the target database based on the difference data to obtain a successful synchronization result.

[0134] In this embodiment, when a Kafka push failure message is detected, it is confirmed that Kafka push is unavailable, triggering a synchronization mode switching mechanism. This automatically activates the full update process of the source and backup databases, while simultaneously suspending the Kafka push link. This achieves automatic switching in fault scenarios, reduces the time spent on manual intervention, and avoids long-term data synchronization stagnation. After switching, the status of the source and backup databases is first verified, and the synchronization delay and data integrity between the backup and primary databases are queried to confirm that the backup database data is consistent with the primary database. This avoids performing updates based on abnormal backup database data and ensures the reliability of the data source for the full update.

[0135] Optionally, the source and backup database full update process includes: obtaining the current full data of the data table to be synchronized from the backup database of the source database, wherein the current full data includes all record files in the data table to be synchronized and the latest status of each record file; performing a full comparison between the current full data and the existing data of the corresponding data table in the target database; determining the difference data based on the full comparison result; and performing a full update operation on the target database based on the difference data to obtain a successful synchronization result.

[0136] Optionally, such as Figure 3 As shown, Figure 3 This diagram illustrates the Kafka push functionality involved in this application. The Kafka message queue is a distributed middleware built on Apache Kafka technology for temporarily storing and forwarding data messages. The Kafka message queue's operational logic permeates the entire "message production-storage-consumption" chain. It is responsible for receiving changed data sent by the sender (the application corresponding to the source database), storing it according to preset rules to become consumed changed data, and then forwarding it in an orderly manner to the receiver (the application corresponding to the target database). If Kafka push is determined to be unavailable, a full update of the source database backup is initiated. The principle of this full update is consistent with the principle of the full update of the source database backup when multi-threaded synchronization fails, and will not be elaborated further here.

[0137] Optionally, before step D100, the method further includes sending a test message corresponding to the encapsulated record file to a preset Kafka topic; when the test message is detected to meet a preset fault condition, it is determined that Kafka push is unavailable and Kafka push is started, wherein the preset fault condition is used to determine whether Kafka push is available.

[0138] Optionally, a test message corresponding to the encapsulated record file is sent to a preset Kafka topic. Test data with the same format as the actual business message must be generated and pushed to the target topic via the Kafka producer client of the application to be synchronized. This test message must carry an identifier to distinguish it from the business message, verifying the availability of the Kafka link in advance, avoiding the exposure of push failures when pushing business data in batches, and reducing unnecessary consumption in large-scale data processing. When a test message is detected to meet preset failure conditions, failure conditions must be defined first (e.g., no ACK confirmation is received from the Kafka cluster within 5 seconds after the test message is sent, the cumulative number of message sending failures exceeds 3, the test message is stored in the topic for more than 10 seconds without being detected by the consumer). Alternatively, if the capacity of the Kafka push change table is small, or there is no system traffic at night, the status of the test message is tracked in real time through the monitoring module. When any condition is triggered, Kafka push is determined to be unavailable. This objectively evaluates the Kafka link status through quantitative standards, avoiding misjudgments of availability or unavailability due to vague judgments, and ensuring the accuracy of decision-making.

[0139] Optionally, after determining that Kafka push is unavailable, restarting Kafka push requires first triggering a link self-check (such as checking Kafka node status, network connection, and topic configuration), performing repairs based on the fault type (such as switching to a backup Kafka node if a node crashes, or re-authenticating for permission issues), resending test messages for verification after repair, and continuing until the link is restored before pushing business data. This allows for rapid activation of the self-healing mechanism after a fault occurs, shortening the duration of Kafka push unavailability, ensuring the timeliness of business data synchronization, and avoiding large-scale data backlog caused by link interruption.

[0140] In this embodiment, when a Kafka push failure message is detected, a synchronization mode switching mechanism is triggered to switch to the source database backup full update mode and execute the corresponding synchronization operation steps for the source database backup full update mode. This avoids data synchronization interruption caused by the failure of a single Kafka link. By switching to the source database backup full update mode, since the backup database is synchronized with the primary database in real time, its full data is consistent with the primary database. It can serve as a reliable data source to ensure the integrity of synchronized data and prevent the loss of changed data caused by Kafka push failure.

[0141] In addition, this application also provides a database synchronization device, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the database synchronization device involved in the embodiments of this application. The database synchronization device provided in this application includes:

[0142] The data acquisition module H01 is used to acquire the data table to be synchronized and its attribute parameters in the source database. The attribute parameters include the storage capacity of the data table to be synchronized, the real-time change frequency of the data table to be synchronized, and the amount of changed data of the data table to be synchronized per unit time.

[0143] The condition judgment module H02 is used to determine whether the attribute parameters meet the preset synchronization conditions. The preset synchronization conditions are that the storage capacity, real-time change frequency, and change data volume are all greater than their respective thresholds.

[0144] The synchronization mode acquisition module H03 is used to acquire the preset synchronization mode when the attribute parameter is detected to meet the preset synchronization conditions. The preset synchronization mode includes at least one of multi-threaded synchronization mode and single-threaded synchronization mode.

[0145] The data synchronization module H04 is used to synchronize the data table to be synchronized to the target database based on a preset synchronization method.

[0146] The database synchronization device provided in this application, employing the database synchronization method in the above embodiments, can solve the technical problem of low efficiency in database synchronization devices. Compared with the prior art, the beneficial effects of the database synchronization device provided in this application are the same as those of the database synchronization device method provided in the above embodiments, and other technical features in this database synchronization device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0147] In addition, this application also provides a database synchronization device. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of the database synchronization device involved in the embodiments of this application. Specifically, the device in the embodiments of this application may be a device for running a database synchronization method locally.

[0148] This application provides a database synchronization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the database synchronization device method of the above embodiment 1.

[0149] The following is for reference. Figure 5 , Figure 5 This is a schematic diagram of the structure of a database synchronization device involved in the embodiments of this application, showing a structural schematic diagram suitable for implementing the database synchronization device of the embodiments of this application. The database synchronization device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle terminals (e.g., vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The database synchronization device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0150] like Figure 5 As shown, the database synchronization device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the database synchronization device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following devices can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the database synchronization device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a database synchronization device with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented alternatively.

[0151] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0152] The database synchronization device provided in this application, employing the database synchronization method described in the above embodiments, can solve the technical problem of low efficiency in database synchronization devices. Compared with the prior art, the beneficial effects of the database synchronization device provided in this application are the same as those of the database synchronization method provided in the above embodiments, and other technical features of this database synchronization device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0153] Furthermore, this application provides a computer-readable storage medium. This computer-readable storage medium stores a database synchronization program, which, when executed by a processor, implements the steps of the database synchronization method described above.

[0154] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0155] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0157] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.< / t>

Claims

1. A database synchronization method, characterized in that, The database synchronization method includes: Upon receiving a data synchronization request, the system obtains the data table to be synchronized and its attribute parameters from the source database. The attribute parameters include the storage capacity of the data table to be synchronized, the real-time change frequency of the data table to be synchronized, and the amount of changed data in the data table to be synchronized per unit time. Determine whether the attribute parameters meet the preset synchronization conditions, wherein the preset synchronization conditions are that the storage capacity, the real-time change frequency, and the amount of changed data are all greater than their respective thresholds; When the attribute parameter is detected to meet the preset synchronization condition, a preset synchronization method is obtained, wherein the preset synchronization method includes at least one of a multi-threaded synchronization method and a single-threaded synchronization method; The data table to be synchronized is synchronized to the target database based on the preset synchronization method.

2. The database synchronization method as described in claim 1, characterized in that, The step of determining whether the attribute parameter meets the preset synchronization conditions includes: The storage capacity is compared with a preset capacity threshold, the real-time change frequency is compared with a preset frequency threshold, and the change data volume is compared with a preset data volume threshold. If the storage capacity is greater than a preset capacity threshold, the real-time change frequency is greater than a preset frequency threshold, and the change data volume is greater than a preset data volume threshold, then the attribute parameters are determined to meet the preset synchronization conditions. If the storage capacity is less than or equal to a preset capacity threshold, and / or the real-time change frequency is less than or equal to a preset frequency threshold, and / or the change data volume is less than or equal to a preset data volume threshold, then the attribute parameter is determined to not meet the preset synchronization conditions.

3. The database synchronization method as described in claim 2, characterized in that, The step of obtaining the preset synchronization method when the attribute parameter is detected to meet the preset synchronization condition includes: When the attribute parameter is detected to meet the preset synchronization condition, the business attribute characteristics of the attribute parameter are obtained, wherein the business attribute characteristics include at least one of industry adaptation characteristics and timeliness control characteristics; The data table to be synchronized is divided according to the business attribute characteristics to obtain multiple independent synchronization sub-tasks; Compare the number of the synchronized subtasks with the preset single-threaded processing threshold; If the number of synchronized subtasks is greater than the preset single-threaded processing threshold, then the preset synchronization method is determined to be a multi-threaded synchronization method, wherein the number of threads in the multi-threaded synchronization method is determined based on the number of synchronized subtasks and the processing capacity of each single thread corresponding to the multi-threaded synchronization method.

4. The database synchronization method as described in claim 3, characterized in that, Before the step of synchronizing the data table to be synchronized to the target database based on the preset synchronization method, the method further includes: Upon receiving an instruction to synchronize the execution of the synchronization subtask, a request is sent to a preset distributed resource locking unit to acquire the distributed lock corresponding to the synchronization subtask. When the information indicating that the distributed lock has been successfully acquired is detected, the step of synchronizing the data table to be synchronized to the target database based on the preset synchronization method is executed; When the information indicating a failure to acquire the distributed lock is detected, the next data table to be synchronized in the source database is determined, and the steps of acquiring the data table to be synchronized in the source database and the attribute parameters of the data table to be synchronized are performed based on the next data table to be synchronized.

5. The database synchronization method as described in claim 3, characterized in that, After the step of synchronizing the data table to be synchronized to the target database based on the preset synchronization method, the following steps are included: When a synchronization failure message is detected in the multi-threaded synchronization method, a synchronization method switching mechanism is triggered, and the multi-threaded synchronization method is switched to the source database backup full update method. Obtain the current full data of the data table to be synchronized from the backup database of the source database, wherein the current full data includes all record files in the data table to be synchronized and the latest status of each record file; The current full data is compared with the existing data in the corresponding data table of the data table to be synchronized in the target database, and the difference data is determined based on the full comparison results. A full update operation is performed on the target database based on the difference data, resulting in a successful synchronization.

6. The database synchronization method as described in any one of claims 1-4, characterized in that, After the step of determining whether the attribute parameter meets the preset synchronization conditions, the method further includes: When it is detected that the attribute parameter does not meet the preset synchronization condition, the update time of the data table to be synchronized in the source database is polled at a preset time interval, and the record files whose update time is within the current polling period are selected. The selected record files are marked with operation types to obtain marked record files; The marked record file is encapsulated according to a preset data format to obtain the encapsulated record file; The encapsulated log files are then pushed as messages to the target database.

7. The database synchronization method as described in claim 6, characterized in that, Following the step of pushing each of the encapsulated records as a message to the target database, the following is included: When a push failure message is detected, a synchronization mode switching mechanism is triggered to switch the push synchronization mode to a full update mode for the source database backup database. Obtain the current full data of the data table to be synchronized from the backup database of the source database, wherein the current full data includes all record files in the data table to be synchronized and the latest status of each record file; The current full data is compared with the existing data in the corresponding data table of the data table to be synchronized in the target database, and the difference data is determined based on the full comparison results. A full update operation is performed on the target database based on the difference data, resulting in a successful synchronization.

8. A database synchronization device, characterized in that, The database synchronization device includes: The data acquisition module is used to acquire the data table to be synchronized and the attribute parameters of the data table to be synchronized in the source database. The attribute parameters include the storage capacity of the data table to be synchronized, the real-time change frequency of the data table to be synchronized, and the amount of changed data of the data table to be synchronized per unit time. The condition judgment module is used to determine whether the attribute parameters meet the preset synchronization conditions, wherein the preset synchronization conditions are that the storage capacity, the real-time change frequency, and the change data volume are all greater than their respective thresholds; The synchronization mode acquisition module is used to acquire a preset synchronization mode when the attribute parameter is detected to meet the preset synchronization condition, wherein the preset synchronization mode includes at least one of a multi-threaded synchronization mode and a single-threaded synchronization mode; The data synchronization module synchronizes the data table to be synchronized to the target database based on the preset synchronization method.

9. A database synchronization device, characterized in that, The database synchronization device includes a memory, a processor, and a database synchronization program stored in the memory and executable on the processor. When the processor executes the database synchronization program, it implements the steps of the database synchronization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a database synchronization program, which, when executed by a processor, implements the steps of the database synchronization method as described in any one of claims 1 to 7.

Images