Data synchronization method, device, storage medium and electronic device

By obtaining log data from the MongoDB database to determine incremental data and synchronizing it to the target database, the issues of data synchronization timeliness and source database load are resolved, achieving efficient and stable data transmission.

CN117216159BActive Publication Date: 2024-11-05FEISUANSHUZHI TECHNOLOGY (SHENZHEN) CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202311066707.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-22
Publication Date
2024-11-05
Estimated Expiration
2043-08-22

AI Technical Summary

Technical Problem

In traditional solutions that synchronize data from MongoDB to DreamData, the timeliness of data synchronization is difficult to guarantee, and the load on the source database is not taken into account, which affects the normal operation of the database.

Method used

By acquiring log data from the first database, the latest target time is determined, thereby acquiring incremental data and synchronizing it to the target database, reducing the amount of data to be synchronized, lowering latency, and performing restarts and verifications in case of synchronization failure.

Benefits of technology

It improves the efficiency of data synchronization, ensures the stability of data transmission, reduces the load on the source database, and prevents data loss.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117216159B_ABST
    Figure CN117216159B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a data synchronization method, device, storage medium and electronic device, the method comprising: acquiring log data of a first database; the log data comprising update time and update data of data in the first database; determining the latest target time from the update time; determining incremental data from the first database according to the target time and the log data, the incremental data comprising data generated in the first database after the target time; synchronizing the incremental data to the target database.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular, to a data synchronization method, device, storage medium and electronic device. Background Art

[0002] In big data technology, data from different systems need to be synchronized. In the traditional solution of synchronizing from MongoDB to the Dream database, data synchronization usually synchronizes all data once a day. However, with this method, the timeliness of data synchronization is difficult to guarantee, and the load problem of the source database when extracting data is not considered, which will affect the normal operation of the database. Summary of the invention

[0003] The purpose of the present disclosure is to provide a data synchronization method, device, storage medium and electronic device.

[0004] In order to achieve the above objectives, in a first aspect, the present disclosure provides a data synchronization method, the method comprising:

[0005] Acquire log data of the first database; the log data includes update time and update data of data in the first database;

[0006] determining a latest target time from the update time;

[0007] Determine incremental data from the first database according to the target time and the log data, the incremental data including data in the first database generated after the target time;

[0008] The incremental data is synchronized to the target database.

[0009] Optionally, determining the incremental data from the first database according to the target time and the log data includes:

[0010] The log data is parsed according to the target time to obtain the incremental data of the first database.

[0011] Optionally, synchronizing the incremental data to a target database includes:

[0012] Determine a corresponding pending data table according to the incremental data;

[0013] According to the pending data table, the incremental data is synchronized to the target database.

[0014] Optionally, synchronizing the incremental data to the target database according to the pending data table includes:

[0015] Determine the mapping relationship between the pending data table and the target database according to the preset corresponding relationship;

[0016] According to the mapping relationship, the incremental data corresponding to the pending data table is forwarded to the target database, so that the target database performs data synchronization according to the incremental data.

[0017] Optionally, the method further comprises:

[0018] Obtaining a check time node, where the check time node is the next time adjacent to the target time node;

[0019] Determine the synchronization result of the data synchronization task corresponding to the inspection time node;

[0020] When it is determined that the synchronization result is a synchronization failure, forwarding the incremental data to the target database is stopped.

[0021] Optionally, the determining a synchronization result of the data synchronization task corresponding to the inspection time node includes:

[0022] Performing data verification on the incremental data of the first database and the incremental data of the target database at the check time node to verify whether the incremental data of the first database and the incremental data of the target database are completely identical;

[0023] In a case where the incremental data of the first database and the incremental data of the target database are not completely the same, it is determined that the synchronization result of the data synchronization task corresponding to the inspection time node is a synchronization failure.

[0024] Optionally, the method further comprises:

[0025] If it is determined that the synchronization result is a synchronization failure, generating a first target database according to the first database, wherein the first target database includes all data of the first database;

[0026] The first target database is used as the new target database, and the data synchronization task of the target time node is restarted.

[0027] In a second aspect, the present disclosure provides a data synchronization device, the device comprising:

[0028] An acquisition module, used to acquire log data of the first database; the log data includes update time and update data of the data in the first database;

[0029] A first determination module, used to determine the latest target time from the update time;

[0030] A second determining module, configured to determine incremental data from the first database according to the target time and the log data, wherein the incremental data includes data in the first database generated after the target time;

[0031] The synchronization module is used to synchronize the incremental data to the target database.

[0032] Optionally, the second determination module is used to parse the log data according to the target time to obtain the incremental data of the first database.

[0033] Optionally, the synchronization module includes:

[0034] A first determination submodule, configured to determine a corresponding pending data table according to the incremental data;

[0035] A synchronization submodule is used to synchronize the incremental data to the target database according to the pending data table.

[0036] Optionally, the first determination submodule is used to determine the mapping relationship between the pending data table and the target database according to a preset corresponding relationship; according to the mapping relationship, the incremental data corresponding to the pending data table is forwarded to the target database, so that the target database performs data synchronization according to the incremental data.

[0037] Optionally, the device further comprises:

[0038] A checking module, used for obtaining a checking time node, where the checking time node is the next time node adjacent to the target time node;

[0039] A third determination module is used to determine the synchronization result of the data synchronization task corresponding to the inspection time node;

[0040] The forwarding module is used to stop forwarding the incremental data to the target database when it is determined that the synchronization result is a synchronization failure.

[0041] Optionally, the third determining module includes:

[0042] A verification submodule, configured to perform data verification on the incremental data of the first database and the incremental data of the target database at the check time node to verify whether the incremental data of the first database and the incremental data of the target database are completely identical;

[0043] The second determination submodule is used to determine that the synchronization result of the data synchronization task corresponding to the inspection time node is a synchronization failure when the incremental data of the first database and the incremental data of the target database are not completely the same.

[0044] Optionally, the device further comprises:

[0045] a generating submodule, configured to generate a first target database according to the first database when it is determined that the synchronization result is a synchronization failure, wherein the first target database includes all data of the first database;

[0046] The restart submodule is used to use the first target database as the new target database and restart the data synchronization task of the target time node.

[0047] In a third aspect, the present disclosure provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements the steps of the data synchronization method provided in the first aspect of the present disclosure.

[0048] In a fourth aspect, the present disclosure provides an electronic device, including:

[0049] a memory having a computer program stored thereon;

[0050] A processor is used to execute the computer program in the memory to implement the steps of the data synchronization method provided in the first aspect of the present disclosure.

[0051] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0052] By obtaining the log data of the first database; the log data includes the update time and update data of the data in the first database; determining the latest target time from the update time; determining incremental data from the first database according to the target time and the log data, the incremental data includes the data generated in the first database after the target time; synchronizing the incremental data to the target database. In this way, by determining the incremental data according to the target time of each update and synchronizing the incremental data to the target database, the amount of synchronized data can be reduced, the delay time of data synchronization can be reduced, and compensation and integration of databases can be achieved, thereby improving the efficiency of data synchronization and ensuring the stability of data transmission.

[0053] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0055] Figure 1The present invention is a flowchart of a data synchronization method according to an exemplary embodiment.

[0056] Figure 2 The present invention is a flowchart of a data synchronization method according to an exemplary embodiment.

[0057] Figure 3 The present invention is a flowchart of a data synchronization method according to an exemplary embodiment.

[0058] Figure 4 The invention is a block diagram of a data synchronization device according to an exemplary embodiment.

[0059] Figure 5 is based on Figure 4 The illustrated embodiment shows a block diagram of a synchronization module.

[0060] Figure 6 is based on Figure 4 The illustrated embodiment shows a block diagram of another data synchronization device.

[0061] Figure 7 is based on Figure 6 The illustrated embodiment shows a block diagram of a third determination module.

[0062] Figure 8 is based on Figure 6 The illustrated embodiment shows a block diagram of another data synchronization device.

[0063] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0064] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0065] First, the application scenario of the present disclosure can be introduced. The present disclosure can be applied to the scenario of data synchronization between different databases, especially to the traditional solution of synchronizing from MongoDB to the Dameng database. In this scenario, the data of the Mongodb database needs to be synchronized to the target database, such as the Dameng database. In the related technology, the full amount of data is usually synchronized once a day.

[0066] However, with this method, the timeliness of data synchronization is low due to the large amount of full data and the long synchronization time. In addition, in the process of extracting data, the load problem of the source database is not taken into consideration. This will easily cause the source database CPU (Central Processing Unit), memory, IO (Input / Output) and other resources to be overloaded, thereby indirectly or directly affecting the normal operation of the database. In addition, in the traditional ETL solution, the creation of the target class table of the database requires manual creation of the target class table, and the table cannot be used normally during the derivation process.

[0067] Therefore, the present disclosure provides a data synchronization method, device, storage medium and electronic device; by obtaining log data of a first database; the log data includes the update time and update data of the data in the first database; from the update time, the latest target time is determined; according to the target time and the log data, incremental data is determined from the first database, and the incremental data includes the data generated in the first database after the target time; the incremental data is synchronized to the target database. In this way, by determining the incremental data according to the target time of an update, and synchronizing the incremental data to the target database, the amount of synchronized data can be reduced, and the delay time of data synchronization can be reduced; compensation and integration of databases can be achieved, thereby improving the efficiency of data synchronization and ensuring the stability of data transmission.

[0068] The present disclosure is described below in conjunction with specific embodiments.

[0069] Figure 1 FIG. 1 is a flow chart of a data synchronization method according to an exemplary embodiment. Figure 1 As shown, the method may include the following steps.

[0070] In step S11, log data of the first database is obtained.

[0071] The log data includes the update time and update data of the data in the first database. The first database may be MongoDB (an open source database system based on distributed file storage).

[0072] In the related art, the MongoDB database is a database based on distributed file storage, which can provide a scalable, high-performance data storage solution for WEB applications. However, when MongoDB is currently performing full data migration, it achieves synchronous full data migration by pulling the operation log (oplog). And the oplog itself is a limited space management implemented by a circular queue. When the business write is at its peak, if the stock data migration is slow, the earliest generated oplog may be overwritten by the latest generated oplog. In this way, when the stock data migration is completed, when the incremental migration starts, it will be found that the earlier generated oplog to be pulled has been overwritten, resulting in a fault in the migration, which in turn causes the failure of the data migration.

[0073] Therefore, in the disclosed technical solution, the log data of the first database may be obtained first, and then multiple update times of the first database may be determined according to the log data.

[0074] Here, the first database refers to a database that can directly provide the required original information or specific data in the data migration scenario, wherein the type of the source database can be MongoDB, etc., which is not limited here, and in some scenarios, it can also include SQL Server, PostgreSQL, mySQL and other databases. In this embodiment, the target database corresponds to the first database, which refers to the database object to which the data is migrated, and the type of the target database is also not limited.

[0075] In step S12, the latest target time is determined from the update time.

[0076] In this embodiment, during the actual full data migration, since the first database may continue to store new data, the oplog may change dynamically. In one possible implementation, the latest operation log in the first database may be obtained, and then the latest target time may be determined from the latest operation log. For example, assuming that the full data migration of the first database A is being prepared at the current moment, the latest operation log in the first database at the current moment may be obtained.

[0077] In step S13, incremental data is determined from the first database according to the target time and the log data.

[0078] The incremental data includes data in the first database generated after the target time.

[0079] In this step, the incremental data is data in which data changes occur in the first database. For example, the incremental data may be data obtained by performing one or more of an add operation, a delete operation, and a modify operation on the data in the first database.

[0080] Optionally, the log data may be parsed according to the target time to obtain the incremental data of the first database.

[0081] In step S14, the incremental data is synchronized to the target database.

[0082] Through the above technical solution, the log data of the first database is obtained; the log data includes the update time and update data of the data in the first database; the latest target time is determined from the update time; according to the target time and the log data, the incremental data is determined from the first database, and the incremental data includes the data generated after the target time in the first database; the incremental data is synchronized to the target database. In this way, by determining the incremental data according to the target time of an update and synchronizing the incremental data to the target database, the amount of synchronized data can be reduced, and the delay time of data synchronization can be reduced; compensation and integration of databases can be achieved, thereby improving the efficiency of data synchronization and ensuring the stability of data transmission.

[0083] In some embodiments, Figure 2 As shown, step S14 may include the following steps.

[0084] In step S141, the corresponding pending data table is determined according to the incremental data.

[0085] In this step, a pending data table may be firstly established according to the incremental data.

[0086] In step S142, the incremental data is synchronized to the target database according to the pending data table.

[0087] Optionally, the mapping relationship between the pending data table and the target database can be determined first according to a preset corresponding relationship, and then the incremental data corresponding to the pending data table can be forwarded to the target database according to the mapping relationship, so that the target database synchronizes data according to the incremental data.

[0088] Considering the problem of supplementing and repairing the historical data of the real-time data table, a real-time compensation mechanism can be used. In some embodiments, the target database can also perform data synchronization according to the incremental data, including the following real-time compensation mechanism.

[0089] Optionally, the incremental data can also be synchronized to the pending data table; the target data table corresponding to the source data table is deleted, and the name of the pending data table is changed through a preset naming mechanism to obtain a new target data table, and the target data table is sent to the target database.

[0090] For example, the preset naming mechanism may be a RENAME mechanism; the name of the pending data table may be changed to the target data table through the RENAME mechanism.

[0091] By adopting the above technical solution, the incremental data can be sent to the pending data table and the pending data table name can be changed to the target data table by renaming the table name mechanism, thereby completing the data synchronization of the incremental data. In this way, the access to the target database during the data repair process can be unaffected.

[0092] In some embodiments, Figure 3 As shown, the method also includes the following steps.

[0093] In step S15, the inspection time node is obtained.

[0094] The inspection time node is the next time adjacent to the target time node.

[0095] In this embodiment, after the target database performs data synchronization according to the incremental data, the next update time node adjacent to the target time node may be determined according to the log data of the first database.

[0096] In step S16, the synchronization result of the data synchronization task corresponding to the inspection time node is determined.

[0097] Optionally, the incremental data of the first database and the incremental data of the target database can first be verified at the inspection time node to verify whether the incremental data of the first database and the incremental data of the target database are exactly the same; then, if the incremental data of the first database and the incremental data of the target database are not exactly the same, the synchronization result of the data synchronization task corresponding to the inspection time node can be determined to be a synchronization failure; if the incremental data of the first database and the incremental data of the target database are exactly the same, the synchronization result of the data synchronization task corresponding to the inspection time node can be determined to be a synchronization success.

[0098] In step S17, when it is determined that the synchronization result is a synchronization failure, forwarding the incremental data to the target database is stopped.

[0099] Optionally, when it is determined that the synchronization result is a synchronization failure, a first target database can be generated based on the first database, and the first target database includes all the data of the first database; then the first target database can be used as the new target database, and the data synchronization task of the target time node can be restarted.

[0100] By adopting the above technical solution, after the target database synchronizes data according to the incremental data, the synchronization result can be synchronously checked at the check time node, and the synchronization can be restarted in case of synchronization failure, thereby preventing data loss caused by data synchronization failure.

[0101] Figure 4 FIG. 1 is a block diagram of a data synchronization device according to an exemplary embodiment. Figure 4 As shown, the device 400 includes an acquisition module 401 , a first determination module 402 , a second determination module 403 and a synchronization module 404 .

[0102] The acquisition module 401 is used to acquire log data of the first database; the log data includes the update time and update data of the data in the first database;

[0103] A first determination module 402, configured to determine the latest target time from the update time;

[0104] A second determining module 403 is used to determine incremental data from the first database according to the target time and the log data, where the incremental data includes data generated in the first database after the target time;

[0105] The synchronization module 404 is used to synchronize the incremental data to the target database.

[0106] The second determination module 403 is used to parse the log data according to the target time to obtain the incremental data of the first database.

[0107] Figure 5 is based on Figure 4 The embodiment shown is a block diagram of a synchronization module. Figure 5 As shown, the synchronization module 404 includes:

[0108] A first determination submodule 4041 is used to determine a corresponding pending data table according to the incremental data;

[0109] The synchronization submodule 4042 is used to synchronize the incremental data to the target database according to the pending data table.

[0110] Optionally, the first determination submodule 4041 is used to determine the mapping relationship between the pending data table and the target database according to a preset corresponding relationship; according to the mapping relationship, the incremental data corresponding to the pending data table is forwarded to the target database, so that the target database synchronizes data according to the incremental data.

[0111] Figure 6 is based on Figure 4 The embodiment shown is a block diagram of another data synchronization device. Figure 6 As shown, the device also includes:

[0112] The inspection module 405 is used to obtain an inspection time node, where the inspection time node is the next time adjacent to the target time node;

[0113] The third determination module 406 is used to determine the synchronization result of the data synchronization task corresponding to the inspection time node;

[0114] The forwarding module 407 is used to stop forwarding the incremental data to the target database when it is determined that the synchronization result is a synchronization failure.

[0115] Figure 7 is based on Figure 6 The embodiment shown is a block diagram of a third determination module. Figure 7 As shown, the third determination module 406 includes:

[0116] The verification submodule 4061 is used to perform data verification on the incremental data of the first database and the incremental data of the target database at the check time node to verify whether the incremental data of the first database and the incremental data of the target database are completely the same;

[0117] The second determining submodule 4062 is used to determine that the synchronization result of the data synchronization task corresponding to the check time node is a synchronization failure when the incremental data of the first database and the incremental data of the target database are not completely the same.

[0118] Figure 8 is based on Figure 6 The embodiment shown is a block diagram of another data synchronization device. Figure 8 As shown, the device also includes:

[0119] A generating submodule 408 is configured to generate a first target database according to the first database when it is determined that the synchronization result is a synchronization failure, wherein the first target database includes all data of the first database;

[0120] The restart submodule 409 is used to use the first target database as the new target database and restart the data synchronization task of the target time node.

[0121] By obtaining the log data of the first database; the log data includes the update time and update data of the data in the first database; determining the latest target time from the update time; determining incremental data from the first database according to the target time and the log data, the incremental data includes the data generated in the first database after the target time; synchronizing the incremental data to the target database. In this way, by determining the incremental data according to the target time of an update and synchronizing the incremental data to the target database, the amount of synchronized data can be reduced, the delay time of data synchronization can be reduced, and compensation and integration of databases can be achieved, thereby improving the efficiency of data synchronization and ensuring the stability of data transmission.

[0122] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0123] Fig. 9 FIG. 9 is a block diagram of an electronic device 900 according to an exemplary embodiment. Fig. 9 As shown, the electronic device 900 may include: a processor 901 and a memory 902. The electronic device 900 may also include one or more of a multimedia component 903, an input / output (I / O) interface 904, and a communication component 905.

[0124] The processor 901 is used to control the overall operation of the electronic device 900 to complete all or part of the steps in the above-mentioned data synchronization method. The memory 902 is used to store various types of data to support the operation of the electronic device 900. For example, these data may include instructions for any application or method used to operate on the electronic device 900, and application-related data, such as contact data, messages sent and received, pictures, audio, video, etc. The memory 902 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 903 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 902 or sent through the communication component 905. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 904 provides an interface between the processor 901 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 905 is used for wired or wireless communication between the electronic device 900 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 905 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

[0125] In an exemplary embodiment, the electronic device 900 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned data synchronization method.

[0126] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned data synchronization method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 902 including program instructions, and the above-mentioned program instructions can be executed by the processor 901 of the electronic device 900 to complete the above-mentioned data synchronization method.

[0127] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above data synchronization method when executed by the programmable device.

[0128] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0129] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0130] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A data synchronization method, characterized in that: The method comprises: Acquire log data of the first database; the log data includes update time and update data of data in the first database; determining a latest target time from the update time; Determine incremental data from the first database according to the target time and the log data, the incremental data including data in the first database generated after the target time; Synchronize the incremental data to the target database; The step of synchronizing the incremental data to the target database includes: Synchronize the incremental data to the pending data table corresponding to the incremental data, and delete the target data table corresponding to the source data table; Changing the name of the pending data table through a preset naming mechanism to obtain a new target data table, and sending the new target data table to a target database; The method further comprises: Acquire a check time node, where the check time node is a next update time node adjacent to the target time node determined according to the log data of the first database; Determine the synchronization result of the data synchronization task corresponding to the inspection time node; When it is determined that the synchronization result is a synchronization failure, stop forwarding the incremental data to the target database; When it is determined that the synchronization result is a synchronization failure, a first target database is generated based on the first database, and the first target database includes all the data of the first database; and the first target database is used as the new target database, and the data synchronization task of the target time node is restarted.

2. The method according to claim 1, characterized in that Determining the incremental data from the first database according to the target time and the log data includes: The log data is parsed according to the target time to obtain the incremental data of the first database.

3. The method according to claim 1, characterized in that The step of synchronizing the incremental data to the target database includes: Determine a corresponding pending data table according to the incremental data; According to the pending data table, the incremental data is synchronized to the target database.

4. The method according to claim 3, characterized in that The step of synchronizing the incremental data to the target database according to the pending data table includes: Determine the mapping relationship between the pending data table and the target database according to the preset corresponding relationship; According to the mapping relationship, the incremental data corresponding to the pending data table is forwarded to the target database, so that the target database performs data synchronization according to the incremental data.

5. The method according to claim 1, characterized in that The determination of the synchronization result of the data synchronization task corresponding to the inspection time node includes: Performing data verification on the incremental data of the first database and the incremental data of the target database at the check time node to verify whether the incremental data of the first database and the incremental data of the target database are completely identical; In a case where the incremental data of the first database and the incremental data of the target database are not completely the same, it is determined that the synchronization result of the data synchronization task corresponding to the inspection time node is a synchronization failure.

6. A data synchronization device, characterized in that: The device comprises: An acquisition module, used to acquire log data of the first database; the log data includes update time and update data of the data in the first database; A first determination module, used to determine the latest target time from the update time; A second determining module, configured to determine incremental data from the first database according to the target time and the log data, wherein the incremental data includes data in the first database generated after the target time; A synchronization module, used to synchronize the incremental data to a target database; The synchronization module is used to synchronize the incremental data to the pending data table corresponding to the incremental data, and delete the target data table corresponding to the source data table; change the name of the pending data table through a preset naming mechanism to obtain a new target data table, and send the new target data table to the target database; The device also includes: A checking module, used for obtaining a checking time node, where the checking time node is a next update time node adjacent to the target time node determined according to the log data of the first database; A third determination module is used to determine the synchronization result of the data synchronization task corresponding to the inspection time node; A forwarding module, configured to stop forwarding the incremental data to the target database when determining that the synchronization result is a synchronization failure; a generating module, configured to generate a first target database according to the first database when it is determined that the synchronization result is a synchronization failure, wherein the first target database includes all data of the first database; A restart module is used to use the first target database as the new target database and restart the data synchronization task of the target time node.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

8. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Database migration method and system based on log synchronization

    CN113051247A

  • Data updating method and device

    CN113326232A

  • Snapshot processing method and device based on object storage bucket and electronic equipment

    CN115292244A

  • Data synchronization method for fusing stock data and incremental data based on message queue

    CN115599870A

  • Heterogeneous relational database rapid migration method and device

    CN115934681A