Data synchronization method and system, electronic equipment and storage medium

Abstract

The invention discloses a data synchronization method and system, electronic equipment and a storage medium, and relates to the technical field of data synchronism. The method comprises the steps that data sets are inquired in a source library and a target library respectively; comparing the data sets respectively queried in the source library and the target library, and determining a data variable quantity; and performing data synchronization on the source library and the target library according to the data variation. According to the method, bidirectional data synchronization can be simultaneously carried out on the source library and the target library according to the data variable quantity, data synchronization can be realized among different databases, the situation that one synchronization link needs to be maintained for each table is avoided, and the cost is reduced.

Description

Background Art

[0002] In traditional data synchronization scenarios, data synchronization is usually achieved at the database level. This synchronization method is usually achieved by replaying logs. That is, the script executed in database A is executed again in database B. This synchronization method has the following limitations:

[0003] 1) Traditional data synchronization can only achieve one-way data synchronization. Specifically, the data to be synchronized is synchronized from database A for writing to database B for reading, or the data to be synchronized is synchronized from database B for writing to database A for reading. However, the data to be synchronized cannot be synchronized from database A for writing to database B for writing, nor can the data to be synchronized from database B for writing to database A for writing.

[0004] 2) Only synchronization between the same databases is supported, and data synchronization between different databases is not supported.

[0005] 3) Each table needs to maintain a synchronization link, which is costly. Summary of the invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a data synchronization method, system, electronic device and storage medium, as follows:

[0007] 1) In the first aspect, the present invention provides a data synchronization method, and the specific technical solution is as follows:

[0008] Query the data sets in the source and target databases respectively;

[0009] Compare the data sets queried in the source database and the target database to determine the amount of data change;

[0010] Synchronize the source and target databases based on the amount of data changes.

[0011] The beneficial effects of a data synchronization method provided by the present invention are as follows:

[0012] According to the amount of data changes, bidirectional data synchronization can be performed on the source database and the target database at the same time. Moreover, data synchronization can be achieved between different databases, avoiding the situation where "each table needs to maintain a synchronization link" and reducing costs.

[0013] Based on the above solution, a data synchronization method of the present invention can also be improved as follows.

[0014] Furthermore, according to the amount of data changes, the source database and the target database are synchronized, including:

[0015] Generate DML statements corresponding to the data changes, and synchronize data between the source database and the target database based on the DML statements corresponding to the data changes.

[0016] Furthermore, it also includes:

[0017] Collect DML statements generated when operating the source and target databases;

[0018] Convert the collected DML statements into data of standard structure, which at least includes: the data range in the source database and the data range in the target database;

[0019] Query the data sets in the source and target databases respectively, including:

[0020] According to the data range in the source database, the data set is queried in the source database, and according to the data range in the target database, the data set is queried in the target database.

[0021] Furthermore, the DML statements generated when operating the source and target databases are collected, including:

[0022] Use the druid data change collection method, probe data change collection method, JDBC proxy method, DAO layer interception method, embedded code sending method or manual page trigger method to collect DML statements generated when operating the source and target libraries.

[0023] 2) In a second aspect, the present invention also provides a data synchronization system, and the specific technical solution is as follows:

[0024] Including data set query module, data comparison module and data synchronization module;

[0025] The dataset query module is used to query datasets in the source and target libraries respectively;

[0026] The data comparison module is used to compare the data sets queried in the source database and the target database respectively to determine the amount of data changes;

[0027] The data synchronization module is used to synchronize the source database and the target database according to the amount of data changes.

[0028] Based on the above solution, a data synchronization system of the present invention can also be improved as follows.

[0029] Furthermore, the data synchronization module is specifically used for:

[0030] Generate DML statements corresponding to the data changes, and synchronize data between the source database and the target database based on the DML statements corresponding to the data changes.

[0031] Further, it also includes a collection module and a sentence conversion module;

[0032] The collection module is used to collect DML statements generated when operating the source and target libraries;

[0033] The statement conversion module is used to convert the collected DML statements into data of a standard structure, where the data of the standard structure at least includes: the data range in the source database and the data range in the target database;

[0034] The data set query module is specifically used to query the data set in the source library according to the data range in the source library, and query the data set in the target library according to the data range in the target library.

[0035] Furthermore, the collection module is specifically used to collect DML statements generated when operating the source library and the target library by using the druid data change collection method, the probe data change collection method, the JDBC proxy method, the DAO layer interception method, the embedded point coding sending method or the manual page trigger method.

[0036] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor, so that the electronic device implements any of the above-mentioned data synchronization methods.

[0037] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor so that the computer implements any of the above-mentioned data synchronization methods.

[0038] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0040] Figure 1 A schematic diagram of a data synchronization method according to an embodiment of the present invention;

[0041] Figure 2 A structural diagram of a data synchronization system according to an embodiment of the present invention;

[0042] Figure 3 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] like Figure 1 As shown, a data synchronization method according to an embodiment of the present invention comprises the following steps:

[0045] S1, query the data set in the source database and the target database respectively;

[0046] Among them, the source database and the target database are both databases specified by the user according to actual conditions.

[0047] S2. Compare the data sets queried in the source database and the target database respectively to determine the amount of data change;

[0048] S3. Synchronize the source database and the target database based on the amount of data change.

[0049] The data change amount is used to describe: data that is different between the data set queried in the source database and the data set queried in the target database. For example, the data set queried in the source database has one more data than the data set queried in the target database. For example, the specific data of the same data item in the data set queried in the source database and the data set queried in the target database are different.

[0050] Optionally, in S3, data synchronization is performed on the source database and the target database according to the amount of data change, including:

[0051] S30: Generate a DML statement corresponding to the data change amount, and synchronize data between the source database and the target database according to the DML statement corresponding to the data change amount.

[0052] Optionally, in the above technical solution, it also includes:

[0053] S01. Collect DML statements generated when operating the source database and the target database;

[0054] S02, converting the collected DML statements into data of a standard structure, where the data of the standard structure at least includes: a data range in a source database and a data range in a target database;

[0055] The data of the standard structure may also include source library information and target library information. The source library information includes the name and storage location of the source library, and the target library information includes the name and storage location of the target library. The standard structure may be set according to actual conditions.

[0056] In S1, the data sets are queried in the source and target databases respectively, including:

[0057] S10. According to the data range in the source database, query the data set in the source database, and according to the data range in the target database, query the data set in the target database.

[0058] Optionally, in S01, DML statements generated when operating the source database and the target database are collected, including:

[0059] S010. Use the druid data change collection method, the probe data change collection method, the JDBC proxy method, the DAO layer interception method, the embedded code sending method or the manual page trigger method to collect the DML statements generated when operating the source database and the target database.

[0060] The method for collecting druid data changes is to use the filter of the druid connection pool to collect the DML statements generated when operating the source and target libraries through the underlying JDBC proxy.

[0061] Among them, the probe data change collection method is: using javaAgent technology to collect data changes in the underlying druid of mybatis.

[0062] In another embodiment, comprising:

[0063] Use the druid data change collection method or the probe data change collection method to collect the DML statements generated when operating the source and target libraries. The druid data change collection method is to use the filter of the druid connection pool to collect the DML statements generated when operating the source and target libraries through the underlying JDBC agent. The probe data change collection method is to use the javaAgent technology to collect the druid data change collection method at the bottom of mybatis. Push data to the data synchronization platform through http or MQ. The data synchronization platform first queries the changed data sets in the source and target libraries, dynamically compiles the PO entity class according to the queried data sets, and then compares the changes in the source and target data sets, generates DML statements, and executes them in the target library.

[0064] In another embodiment, comprising:

[0065] According to the received data sets that may change, the PO entity class is dynamically compiled to receive data. The source data set and the target data set are obtained, and the insert and delete data are obtained by comparing the primary keys. On the basis of the same primary keys, all fields are compared to find the data with the same primary keys but different data, which is the update data. Through the three data sets generated after comparison, the DML statements are reversely generated and executed in the target database to achieve incremental data synchronization.

[0066] The beneficial effects of the present invention are as follows:

[0067] 1) Designed non-intrusive deployment methods for popular programming frameworks, including probe deployment and durid plug-in deployment, and also provided a manual SDK call method.

[0068] 2) Data synchronization is implemented at the application layer, which can be flexibly synchronized according to business scenarios.

[0069] 3) Synchronize using incremental comparison, automatically generate DML statements based on the current data in the source and target databases, and avoid primary key conflict errors in log replay synchronization, which may cause synchronization to stop.

[0070] 4) Dynamic compilation of PO entity objects avoids the phenomenon of field loss caused by manual maintenance of entity classes and DDL statements.

[0071] 5) Provides a solution for two-way data synchronization, which can realize multi-database writing and truly achieve two-way data synchronization between applications and databases.

[0072] 6) Based on the amount of data changes, bidirectional data synchronization can be performed on the source database and the target database at the same time. Moreover, data synchronization can be achieved between different databases, avoiding the situation where "each table needs to maintain a synchronization link" and reducing costs.

[0073] Bidirectional data synchronization means that the data to be synchronized can be synchronized from the database used for writing to the database used for writing.

[0074] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0075] like Figure 2 As shown, a data synchronization system 200 according to an embodiment of the present invention includes a data set query module 201, a data comparison module 202 and a data synchronization module 203;

[0076] The data set query module 201 is used to query the data sets in the source database and the target database respectively;

[0077] The data comparison module 202 is used to compare the data sets queried in the source database and the target database respectively to determine the data change amount;

[0078] The data synchronization module 203 is used to synchronize the source database and the target database according to the amount of data change.

[0079] Optionally, in the above technical solution, the data synchronization module 203 is specifically used for:

[0080] Generate DML statements corresponding to the data changes, and synchronize data between the source database and the target database based on the DML statements corresponding to the data changes.

[0081] Optionally, in the above technical solution, it also includes a collection module and a sentence conversion module;

[0082] The collection module is used to collect DML statements generated when operating the source and target libraries;

[0083] The statement conversion module is used to convert the collected DML statements into data of a standard structure, where the data of the standard structure at least includes: the data range in the source database and the data range in the target database;

[0084] The data set query module 201 is specifically used to query the data set in the source library according to the data range in the source library, and query the data set in the target library according to the data range in the target library.

[0085] Optionally, in the above technical solution, the collection module is specifically used to: utilize the druid data change collection method, the probe data change collection method, the JDBC proxy method, the DAO layer interception method, the embedded point coding sending method or the manual page trigger method to collect the DML statements generated when operating the source library and the target library.

[0086] It should be noted that the beneficial effects of a data synchronization system 200 provided in the above embodiment are the same as the beneficial effects of the above data synchronization method, which will not be described in detail here. In addition, when the system provided in the above embodiment realizes its functions, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described in detail here.

[0087] like Figure 3 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above data synchronization methods, specifically:

[0088] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements any of the data synchronization methods provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for realizing device functions, which will not be described in detail here. The electronic device may specifically be a computer, etc.

[0089] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above-mentioned data synchronization methods.

[0090] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0091] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned data synchronization methods.

[0092] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0093] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0094] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0095] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A data synchronization method, characterized in that: include: Query the data sets in the source and target databases respectively; Compare the data sets queried in the source database and the target database respectively to determine the amount of data change; According to the data change amount, data synchronization is performed on the source database and the target database.

2. A data synchronization method according to claim 1, characterized in that: Synchronizing the source database and the target database according to the data change amount includes: A DML statement corresponding to the data change amount is generated, and data synchronization is performed on the source database and the target database according to the DML statement corresponding to the data change amount.

3. A data synchronization method according to claim 1 or 2, characterized in that: Also includes: Collecting DML statements generated when operating the source database and the target database; Convert the collected DML statements into data of a standard structure, where the data of the standard structure at least includes: a data range in the source database and a data range in the target database; Query the data sets in the source and target databases respectively, including: According to the data range in the source library, a data set is queried in the source library, and according to the data range in the target library, a data set is queried in the target library.

4. A data synchronization method according to claim 3, characterized in that: Collecting DML statements generated when operating the source database and the target database, including: The DML statements generated when operating the source database and the target database are collected by using the druid data change collection method, the probe data change collection method, the JDBC proxy method, the DAO layer interception method, the embedded code sending method or the manual page triggering method.

5. A data synchronization system, characterized in that: Including data set query module, data comparison module and data synchronization module; The data set query module is used to query the data sets in the source database and the target database respectively; The data comparison module is used to compare the data sets queried in the source database and the target database respectively to determine the data change amount; The data synchronization module is used to synchronize the data between the source database and the target database according to the data change amount.

6. A data synchronization system according to claim 5, characterized in that: The data synchronization module is specifically used for: A DML statement corresponding to the data change amount is generated, and data synchronization is performed on the source database and the target database according to the DML statement corresponding to the data change amount.

7. A data synchronization system according to claim 5 or 6, characterized in that: It also includes a collection module and a sentence conversion module; The collection module is used to: collect DML statements generated when operating the source database and the target database; The statement conversion module is used to convert the collected DML statements into data of a standard structure, where the data of the standard structure at least includes: a data range in the source database and a data range in the target database; The data set query module is specifically used to query the data set in the source library according to the data range in the source library, and query the data set in the target library according to the data range in the target library.

8. A data synchronization system according to claim 7, characterized in that: The collection module is specifically used to collect the DML statements generated when operating the source library and the target library using the druid data change collection method, the probe data change collection method, the JDBC proxy method, the DAO layer interception method, the embedded code sending method or the manual page trigger method.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements a data synchronization method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer readable storage medium At least one computer program is stored, and the at least one computer program is loaded and executed by a processor, So that a computer can implement a data synchronization method as described in any one of claims 1 to 4.

Images