Data synchronization method and device, equipment and storage medium
By implementing the data synchronization method in the database, reading the set to be synchronized and writing it to the target database, the data loss and access problems caused by database crashes are solved, and the full synchronization and security of data are achieved.
Patent Information
- Application Number
- CN202510007253.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-03
- Publication Date
- 2025-04-25
AI Technical Summary
The prior art is difficult to effectively solve the data loss and data access problems caused by database crashes, especially in the scenario of heterogeneous basic data.
Provides a data synchronization method, by starting a data synchronization task, reading the set to be synchronized in the database to be synchronized, writing to the data load queue, and reading and writing to the target database from the queue until all data synchronization is completed.
The full data synchronization of the synchronized database is realized, avoiding data loss and access problems, and ensuring data security and reliability.
Smart Images
Figure CN119884251A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of big data technology, and in particular to a data synchronization method, apparatus, device and storage medium. Background Art
[0002] With the development of informatization and the rapid development of computer technology, governments and enterprises have accumulated a large amount of heterogeneous basic data, which are scattered in the database. However, if only one copy of the data is stored, there is a high probability that some problems will occur. For example, the database crashes and causes data loss, and the desired data may not be accessible. Therefore, it is necessary to back up important data to other databases.
[0003] Based on the above background, the present application proposes a data synchronization method to synchronize the data of a database to be synchronized to a target database, thereby realizing data synchronization of the database to be synchronized. Summary of the invention
[0004] The present invention provides a data synchronization method, device, equipment and storage medium to achieve data synchronization of a database to be synchronized.
[0005] In a first aspect, an embodiment of the present invention provides a data synchronization method, including:
[0006] Start a data synchronization task for the database to be synchronized, and read the set to be synchronized in the database to be synchronized;
[0007] Writing the read set to be synchronized into the data loading queue;
[0008] The to-be-synchronized set is read from the data loading queue, and the read to-be-synchronized set is written into the target database until all the to-be-synchronized sets are written into the target database.
[0009] The technical solution of the embodiment of the present invention provides a data synchronization method, including: starting a data synchronization task for a database to be synchronized, reading a set to be synchronized in the database to be synchronized; writing the read set to be synchronized into a data loading queue; reading the set to be synchronized from the data loading queue, and writing the read set to be synchronized into a target database, until all sets to be synchronized are written into the target database. In the above technical solution, after starting a data synchronization task for a database to be synchronized, based on a data reading thread, reads the set to be synchronized in the database to be synchronized from a queue storing sets to be synchronized by the database to be synchronized, and writes the read set to be synchronized into the data loading queue, so as to realize the successive reading and writing of the set to be synchronized, and avoid the omission of the set. While the data reading thread writes the read set to be synchronized into the data loading queue, based on the data loading thread, reads the set to be synchronized from the data loading queue, and writes the read set to be synchronized into a corresponding storage location in the target database, so as to realize the synchronization of the set to be synchronized to the target database. Based on the same method, the set to be synchronized in the database to be synchronized can be synchronized to the target database, so as to realize the synchronization of all sets to be synchronized in the database to be synchronized to the target database, and then realize the full data synchronization of the database to be synchronized.
[0010] Furthermore, before starting the data synchronization task for the database to be synchronized, the method further includes:
[0011] By traversing each set of the database to be synchronized, the set to be synchronized is written into the data storage queue;
[0012] Accordingly, reading the to-be-synchronized set in the to-be-synchronized database includes:
[0013] The to-be-synchronized set is read from the data storage queue.
[0014] Furthermore, writing the read set to be synchronized into the target database includes:
[0015] The to-be-synchronized set is written into the target database according to the mapping relationship corresponding to the to-be-synchronized set.
[0016] Furthermore, it also includes:
[0017] The start time of the data synchronization task and the reading time of each set to be synchronized are recorded.
[0018] Furthermore, after writing the read set to be synchronized into the target database, the method further includes:
[0019] Filtering the operation logs between the current time and the start time to obtain the target log, and writing the target log into the queue to be distributed;
[0020] Each log in the queue to be distributed is read in sequence, and corresponding data operations are performed on the target database according to the operation type and operation content of each log.
[0021] Further, the target log is obtained by filtering the operation logs between the current time and the startup time, including:
[0022] Filter the heartbeat log and parameter log in the operation log according to the log type to obtain the intermediate log;
[0023] According to the startup time and the read time of each set to be synchronized, the logs between the startup time and the read time corresponding to each set to be synchronized in the intermediate log are filtered to obtain the target log.
[0024] Furthermore, after writing the target log into the queue to be distributed, the method further includes:
[0025] Writing each log into the corresponding data loading queue according to the operation set corresponding to each log in the queue to be distributed;
[0026] Accordingly, each log in the queue to be distributed is read in sequence, including:
[0027] Read each log in the data loading queue in sequence.
[0028] In a second aspect, an embodiment of the present invention further provides a data synchronization device, including:
[0029] A starting module, used to start a data synchronization task for a database to be synchronized, and read a set to be synchronized in the database to be synchronized;
[0030] A writing module, used for writing the read set to be synchronized into a data loading queue;
[0031] The synchronization module is used to read the to-be-synchronized set from the data loading queue, and write the read to-be-synchronized set into the target database until all the to-be-synchronized sets are written into the target database.
[0032] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:
[0033] at least one processor; and a memory communicatively coupled to the at least one processor;
[0034] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method as described in any one of the first aspects.
[0035] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the data synchronization method as described in any one of the first aspects.
[0036] In a fifth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed on a computer, the computer executes the data synchronization method provided in the first aspect.
[0037] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the data synchronization device, or may be packaged separately from the processor of the data synchronization device, which is not limited in this application.
[0038] The description of the second, third, fourth and fifth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third, fourth and fifth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.
[0039] In this application, the name of the above-mentioned data synchronization device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.
[0040] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0042] Figure 1 A flowchart of a data synchronization method provided by an embodiment of the present invention;
[0043] Figure 2 A flowchart of another data synchronization method provided by an embodiment of the present invention;
[0044] Figure 3 A schematic diagram of data interaction of cache files in another data synchronization method provided by an embodiment of the present invention;
[0045] Figure 4 A schematic diagram of the structure of a data synchronization device provided by an embodiment of the present invention;
[0046] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0047] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0048] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0049] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.
[0050] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0051] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.
[0052] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0053] In the description of the present application, unless otherwise specified, “plurality” means two or more.
[0054] Figure 1 A flowchart of a data synchronization method provided by an embodiment of the present invention is provided. This embodiment is applicable to a situation where data stored in a database needs to be synchronized to a target database. The method can be executed by a data synchronization device, such as Figure 1 As shown, the specific steps include:
[0055] Step 110: Start a data synchronization task for the database to be synchronized, and read the set to be synchronized in the database to be synchronized.
[0056] Before performing data synchronization, you need to traverse the collections stored in the database to be synchronized, write the collections to be synchronized into a queue for storage, and trigger the data synchronization task after traversing all collections.
[0057] Specifically, when it is determined that the data synchronization task for the database to be synchronized is triggered, the set to be synchronized in the database to be synchronized can be read from the queue that stores the set that needs to be synchronized by the database to be synchronized. Specifically, the set to be synchronized that is first stored in the queue can be read in the queue according to the first-in-first-out principle of the queue. For example, the set to be synchronized that is first stored in the queue can be read based on the data reading thread to obtain the set to be synchronized, thereby realizing the reading of the set to be synchronized in the database to be synchronized.
[0058] Step 120: Write the read set to be synchronized into the data loading queue.
[0059] Specifically, the read set to be synchronized can be written into the data loading queue
[0060] In an embodiment of the present invention, after starting a data synchronization task for a database to be synchronized, a set to be synchronized in the database to be synchronized is read in a queue based on a data reading thread, and the read set to be synchronized is written into a data loading queue, thereby realizing successive reading and writing of the set to be synchronized and avoiding omission of the set.
[0061] Step 130: Read the to-be-synchronized set from the data loading queue, and write the read to-be-synchronized set into the target database until all the to-be-synchronized sets are written into the target database.
[0062] Specifically, when it is determined that there is a to-be-synchronized set in the data loading queue, the to-be-synchronized set can be read from the data loading queue based on the data loading thread. Specifically, the data loading thread can read the to-be-synchronized set that is first stored in the queue from the data loading queue according to the first-in-first-out principle of the queue.
[0063] For each to-be-synchronized set in the to-be-synchronized database, a storage location corresponding to the target database may be pre-configured for each to-be-synchronized set.
[0064] Therefore, after reading the to-be-synchronized set, the data loading thread can write the to-be-synchronized set into the corresponding storage location in the target database. Of course, the data loading thread can also continue to write other to-be-synchronized sets into the corresponding storage locations in the target database until all to-be-synchronized sets in the to-be-synchronized database are written into the target database, thereby synchronizing the to-be-synchronized sets in the to-be-synchronized database to the target database and realizing data synchronization of the to-be-synchronized database.
[0065] In an embodiment of the present invention, while the data reading thread writes the read set to be synchronized into the data loading queue, the data loading thread reads the set to be synchronized from the data loading queue and writes the read set to be synchronized into the corresponding position in the target database, thereby synchronizing the set to be synchronized to the target database, and further realizing data synchronization of all sets to be synchronized in the database to be synchronized.
[0066] The data synchronization method provided by the embodiment of the present invention includes: starting a data synchronization task for a database to be synchronized, reading a set to be synchronized in the database to be synchronized; writing the read set to be synchronized into a data loading queue; reading the set to be synchronized from the data loading queue, and writing the read set to be synchronized into a target database, until all sets to be synchronized are written into the target database. In the above technical solution, after starting a data synchronization task for a database to be synchronized, based on a data reading thread, reads the set to be synchronized in the database to be synchronized from a queue storing sets to be synchronized by the database to be synchronized, and writes the read set to be synchronized into the data loading queue, so as to realize the successive reading and writing of the set to be synchronized, and avoid the omission of the set. While the data reading thread writes the read set to be synchronized into the data loading queue, based on the data loading thread, reads the set to be synchronized from the data loading queue, and writes the read set to be synchronized into a corresponding storage location in the target database, so as to realize the synchronization of the set to be synchronized to the target database. Based on the same method, the set to be synchronized in the database to be synchronized can be synchronized to the target database, so as to realize the synchronization of all sets to be synchronized in the database to be synchronized to the target database, and then realize the full data synchronization of the database to be synchronized.
[0067] Figure 2 This is a flow chart of another data synchronization method provided by an embodiment of the present invention. This embodiment is specific based on the above embodiment. Figure 2 As shown, in this embodiment, the method may further include:
[0068] Step 210: traverse each set of the database to be synchronized, and write the set to be synchronized into a data storage queue.
[0069] The database to be synchronized stores a large amount of data based on multiple data sets, and each data set includes at least one data. In order to improve the security and reliability of the database, the data of the database to be synchronized needs to be synchronized. Considering the problem of data volume, some data of the database to be synchronized can be synchronized. Databases usually store data based on data sets. Therefore, an identification information can be pre-configured for each data set in the database. The identification information can indicate whether the data set needs to be synchronized. The value of the identification information can be 0 or 1. When the value is 0, it indicates that the data set does not need to be synchronized, and when the value is 1, it indicates that the data set needs to be synchronized.
[0070] Specifically, first, each set of the database to be synchronized can be traversed, and the identification information of each set can be determined during the traversal process. If the identification information of the set is 1, indicating that data synchronization of the set is required, the set can be determined as the set to be synchronized, and then the set to be synchronized can be written into the data storage queue corresponding to the database to be synchronized. If the identification information of the set is 0, indicating that data synchronization of the set is not required, the set does not need to be written into the data storage queue corresponding to the database to be synchronized.
[0071] It should be noted that writing the to-be-synchronized set into the data storage queue corresponding to the to-be-synchronized database here means copying the to-be-synchronized set and writing it into the data storage queue corresponding to the to-be-synchronized database. While writing the to-be-synchronized set into the data storage queue corresponding to the to-be-synchronized database, the to-be-synchronized set in the to-be-synchronized database does not change.
[0072] It should also be noted that the to-be-synchronized set can be determined by traversing each set of the to-be-synchronized database, without writing the to-be-synchronized set into the data storage queue. After traversing all sets of the to-be-synchronized database, the data synchronization task for the to-be-synchronized database is triggered.
[0073] In an embodiment of the present invention, by traversing each set of the database to be synchronized, the set to be synchronized that needs to be synchronized in the database to be synchronized is determined to avoid omission of the set, and the set to be synchronized is written into the data storage queue to achieve temporary storage of the set to be synchronized, further avoiding data synchronization failure caused by the loss of the set to be synchronized.
[0074] Step 220: Start a data synchronization task for the database to be synchronized, and read the set to be synchronized from the data storage queue.
[0075] Step 230: Write the read set to be synchronized into the data loading queue.
[0076] Specifically, when it is determined that the data synchronization task for the database to be synchronized is triggered, the data synchronization task for the database to be synchronized is started, and at this time, the to-be-synchronized set in the database to be synchronized can be read in the data storage queue. Specifically, the to-be-synchronized set first stored in the data storage queue can be read in the data storage queue according to the first-in-first-out principle of the queue. When reading the to-be-synchronized set, first, the set name stored in the queue can be obtained based on the data reading thread, and then the data corresponding to the set name can be read in the queue according to the set name, so as to realize the reading of the to-be-synchronized set in the database to be synchronized. Then, the read to-be-synchronized set can be written into the data loading queue.
[0077] In one implementation, when starting a data synchronization task for a database to be synchronized, the process further includes:
[0078] Record the start time of the data synchronization task.
[0079] Figure 3 A schematic diagram of data interaction of cache files in another data synchronization method provided by an embodiment of the present invention, such as Figure 3 As shown, when starting a data synchronization task for a database to be synchronized, the start time of the data synchronization task can be recorded and stored in a cache file. The cache file can store the corresponding start time according to the task identifier of the data synchronization task.
[0080] In one implementation, when reading the to-be-synchronized set in the to-be-synchronized database, the method further includes:
[0081] Record the read time of each collection to be synchronized.
[0082] like Figure 3 As shown, when reading the to-be-synchronized set from the data storage queue, the reading time of each to-be-synchronized set can be recorded and stored in the cache file. The cache file can store the corresponding reading time according to the set identifier of the to-be-synchronized set.
[0083] In an embodiment of the present invention, after starting a data synchronization task for a database to be synchronized, a to-be-synchronized set in the database to be synchronized is read in a data storage queue based on a data reading thread, so that the to-be-synchronized set is read and written one after another, thereby avoiding omission of a set. In addition, the start time of the data synchronization task and the read time of each to-be-synchronized set are recorded and stored, so as to provide a time reference for incremental synchronization after the full synchronization is completed, thereby facilitating determination of the start time of incremental synchronization.
[0084] Step 240: Read the to-be-synchronized set from the data loading queue, and write the read to-be-synchronized set into the target database.
[0085] In one implementation, writing the read set to be synchronized into a target database includes:
[0086] The to-be-synchronized set is written into the target database according to the mapping relationship corresponding to the to-be-synchronized set.
[0087] Specifically, the data reading thread writes the read set to be synchronized into the data loading queue, so that it can be determined that there is a set to be synchronized in the data loading queue. Therefore, while the data reading thread writes the read set to be synchronized into the data loading queue, the set to be synchronized can be read from the data loading queue based on the data loading thread. Specifically, the set to be synchronized that is first stored in the queue can be read from the data loading queue according to the first-in-first-out principle of the queue.
[0088] For each to-be-synchronized set in the database to be synchronized, a storage location in the target database is pre-configured for each to-be-synchronized set in the form of a mapping relationship. Therefore, after the data loading thread reads the to-be-synchronized set, it can write the to-be-synchronized set into the target database according to the mapping relationship corresponding to the to-be-synchronized set. After the data loading thread writes the current to-be-synchronized set into the corresponding storage location in the target database, it can continue to read the to-be-synchronized set that was first stored in the queue from the data loading queue according to the first-in-first-out principle of the queue, and write the to-be-synchronized set into the corresponding storage location in the target database according to the mapping relationship corresponding to the to-be-synchronized set, until all the to-be-synchronized sets in the database to be synchronized are written into the target database, thereby synchronizing the to-be-synchronized sets in the database to be synchronized to the target database, and realizing data synchronization of the database to be synchronized.
[0089] In an embodiment of the present invention, while the data reading thread writes the read set to be synchronized into the data loading queue, the data loading thread reads the set to be synchronized from the data loading queue and writes the read set to be synchronized into the corresponding position in the target database, thereby synchronizing the set to be synchronized to the target database, and further realizing full data synchronization of all sets to be synchronized in the database to be synchronized.
[0090] Step 250: Filter the operation logs between the current time and the startup time to obtain a target log, and write the target log into a queue to be distributed.
[0091] In one implementation, the target log is obtained by filtering the operation logs between the current time and the startup time, including:
[0092] Filter the heartbeat log and parameter log in the operation log according to the log type to obtain the intermediate log; filter the logs between the startup time and the reading time corresponding to each of the sets to be synchronized in the intermediate log according to the startup time and the reading time to obtain the target log.
[0093] After full data synchronization is completed, as the database runs, the data in the database will inevitably change. The specific details of these changes are stored in the operation log. Therefore, you can continue to perform incremental synchronization on the database based on the operation log.
[0094] The operation log may include a heartbeat log, a parameter log, and an actual business log. The actual business log may include the operation type and operation content on the database, for example, an add operation and added data, a delete operation and deleted data.
[0095] Since there are some heartbeat logs and parameter logs in the operation log that are not related to the operation content, these logs need to be filtered to reduce the data volume of the operation log. In addition, through full synchronization, the data before the start time of the data synchronization task has completed data synchronization, and the data before the reading time corresponding to each collection has completed data synchronization. The operation log needs to be further filtered to reduce the data volume of the operation log again.
[0096] Through the above analysis, we can first obtain the operation log of the database to be synchronized, filter the heartbeat log and parameter log in the operation log according to the log type, obtain the intermediate log, and implement preliminary filtering of the operation log. Figure 3 The start time of the data synchronization task and the read time of each collection are obtained from the cache file shown. The intermediate logs are filtered based on the start time of the data synchronization task and the read time of each collection. Specifically, the intermediate logs can be filtered based on the start time first, and then the logs corresponding to each collection in the filtered intermediate logs can be filtered based on the read time of each collection to obtain the target log, thereby filtering the operation logs again. Through multiple filtering, invalid logs and duplicate logs can be filtered out.
[0097] Each log record of the target log is the operation type and operation content of each set from the reading time to the current time. After obtaining the target log, the target log can be written into the queue to be distributed to realize the sequential storage of each log record in the target log.
[0098] In the embodiment of the present invention, by filtering the operation logs of the database to be synchronized multiple times, invalid logs and duplicate logs are filtered out, a target log including actual operations is obtained, and sequential storage of the target log is achieved.
[0099] Step 260: Write each log into the corresponding data loading queue according to the operation set corresponding to each log in the queue to be distributed.
[0100] Specifically, each log record of the target log in the queue to be distributed can be read based on the data reading thread, and each log record can be distributed and written into different data loading queues based on the data distribution thread. That is, the queue to be distributed corresponding to each log record can be determined according to the set name or set identifier of the operation set corresponding to each log record. For example, the set name or set identifier of the operation set corresponding to each log record can be hashed, and the data loading queue corresponding to each log record can be determined according to the calculation result. Then, each log record can be distributed and written into different data loading queues.
[0101] In the embodiment of the present invention, the log records corresponding to each set are written into the same data loading queue, so as to facilitate continuous incremental synchronization of each set.
[0102] Step 270: read each log in the data loading queue in sequence, and perform corresponding data operations on the target database according to the operation type and operation content of each log, until all the sets to be synchronized are written into the target database.
[0103] Specifically, each data loading queue stores a log corresponding to at least one set. For each data loading queue, each log record can be read in sequence, and corresponding data operations can be performed on the corresponding set in the target database according to the operation type and operation content of each log, so as to achieve incremental synchronization of the corresponding set in the target database. After performing data operations corresponding to all log records in all data loading queues on the target database, all sets to be synchronized can be written to the target database, so as to achieve incremental synchronization of the database to be synchronized.
[0104] In the embodiment of the present invention, incremental data synchronization of all to-be-synchronized sets in the to-be-synchronized database is achieved by reading and executing data operations corresponding to all logs in all data loading queues on the target database.
[0105] The data synchronization method provided by the embodiment of the present invention includes: traversing each set of the database to be synchronized, and writing the set to be synchronized into a data storage queue; starting a data synchronization task for the database to be synchronized, and reading the set to be synchronized from the data storage queue; writing the set to be synchronized into a data loading queue; reading the set to be synchronized from the data loading queue, and writing the read set to be synchronized into a target database; obtaining a target log by filtering operation logs between the current time and the start time, and writing the target log into a queue to be distributed; writing each log into a corresponding data loading queue according to the operation set corresponding to each log in the queue to be distributed; reading each log in each data loading queue in turn, and performing corresponding data operations on the target database according to the operation type and operation content of each log until all the sets to be synchronized are written into the target database. The above technical solution can first traverse each set of the database to be synchronized, determine the set to be synchronized that needs to be synchronized in the database to be synchronized, avoid omission of the set, write the set to be synchronized into the data storage queue, realize temporary storage of the set to be synchronized, and avoid data synchronization failure caused by the loss of the set to be synchronized. After traversing all the sets of the database to be synchronized, trigger the data synchronization task for the database to be synchronized. After starting the data synchronization task for the database to be synchronized, read the set to be synchronized in the database to be synchronized in the data storage queue based on the data reading thread, and write the read set to be synchronized into the data loading queue, realize the successive reading and writing of the set to be synchronized, and avoid omission of the set. While the data reading thread writes the read set to be synchronized into the data loading queue, the data loading thread reads the set to be synchronized from the data loading queue, and writes the set to be synchronized into the corresponding storage location in the target database according to the mapping relationship corresponding to the set to be synchronized, so as to synchronize the set to be synchronized to the target database. Based on the same method, the set to be synchronized in the database to be synchronized can be synchronized to the target database, so as to synchronize all the sets to be synchronized in the database to be synchronized to the target database, thereby realizing full data synchronization of the database to be synchronized.
[0106] After the full data synchronization is completed, the operation log of the database to be synchronized can be obtained. By filtering the operation log of the database to be synchronized multiple times, invalid logs and duplicate logs are filtered out, and the target log including the actual operation is obtained. The target log is written into the queue to be distributed to realize the sequential storage of the target log. Then, the log records corresponding to each collection can be written into the same data loading queue, which is convenient for continuous incremental synchronization of each collection. The data operations corresponding to all logs in all data loading queues are read and executed on the target database, so as to realize the incremental data synchronization of all the collections to be synchronized in the database to be synchronized, and then realize the complete data synchronization of the database to be synchronized.
[0107] Figure 4 This is a schematic diagram of the structure of a data synchronization device provided by an embodiment of the present invention, which can be applied to synchronize data stored in a database to a target database. The device can be implemented by software and / or hardware and is generally integrated in an electronic device, such as a computer device.
[0108] like Figure 4 As shown, the device comprises:
[0109] A starting module 410 is used to start a data synchronization task for a database to be synchronized, and read a set to be synchronized in the database to be synchronized;
[0110] A writing module 420 is used to write the read set to be synchronized into a data loading queue;
[0111] The synchronization module 430 is used to read the to-be-synchronized set from the data loading queue, and write the read to-be-synchronized set into the target database until all the to-be-synchronized sets are written into the target database.
[0112] The data synchronization device provided in this embodiment starts a data synchronization task for a database to be synchronized, reads a set to be synchronized in the database to be synchronized, writes the read set to be synchronized into a data loading queue, reads the set to be synchronized from the data loading queue, and writes the read set to be synchronized into a target database until all sets to be synchronized are written into the target database. The above technical solution, after starting a data synchronization task for a database to be synchronized, reads the set to be synchronized in the database to be synchronized from a queue storing sets to be synchronized by the database to be synchronized based on a data reading thread, and writes the read set to be synchronized into a data loading queue, thereby realizing the sequential reading and writing of the set to be synchronized, and avoiding the omission of the set. While the data reading thread writes the read set to be synchronized into the data loading queue, the data loading thread reads the set to be synchronized from the data loading queue, and writes the read set to be synchronized into a corresponding storage location in the target database, thereby realizing the synchronization of the set to be synchronized to the target database. Based on the same method, the set to be synchronized in the database to be synchronized can be synchronized to the target database, thereby realizing the synchronization of all sets to be synchronized in the database to be synchronized to the target database, thereby realizing the full data synchronization of the database to be synchronized.
[0113] Based on the above embodiment, the device further includes:
[0114] A preprocessing module, configured to traverse each set of the to-be-synchronized database and write the to-be-synchronized set into a data storage queue;
[0115] Accordingly, the startup module 410 is specifically configured to:
[0116] A data synchronization task for the database to be synchronized is started, and the set to be synchronized is read from the data storage queue.
[0117] Based on the above embodiment, the synchronization module 430 is specifically used for:
[0118] The to-be-synchronized set is read from the data loading queue, and the to-be-synchronized set is written into the target database according to the mapping relationship corresponding to the to-be-synchronized set, until all the to-be-synchronized sets are written into the target database.
[0119] Based on the above embodiment, the startup module 410 is further used for:
[0120] When starting a data synchronization task for a database to be synchronized, the start time of the data synchronization task is recorded.
[0121] Based on the above embodiment, the synchronization module 430 is further used for:
[0122] When reading the to-be-synchronized set from the data loading queue, the reading time of each to-be-synchronized set is recorded.
[0123] Based on the above embodiment, the device further includes:
[0124] The filtering module is used to obtain a target log by filtering the operation logs between the current time and the startup time, and write the target log into a queue to be distributed; read each log in the queue to be distributed in sequence, and perform corresponding data operations on the target database according to the operation type and operation content of each log.
[0125] In one implementation, the target log is obtained by filtering the operation logs between the current time and the startup time, including:
[0126] Filter the heartbeat log and parameter log in the operation log according to the log type to obtain the intermediate log; filter the logs between the startup time and the reading time corresponding to each of the sets to be synchronized in the intermediate log according to the startup time and the reading time to obtain the target log.
[0127] Based on the above embodiment, the filtering module is specifically used for:
[0128] The target log is obtained by filtering the operation logs between the current time and the startup time, and the target log is written into the queue to be distributed; each log is written into the corresponding data loading queue according to the operation set corresponding to each log in the queue to be distributed; each log in the data loading queue is read in turn, and corresponding data operations are performed on the target database according to the operation type and operation content of each log.
[0129] The data synchronization device provided in the embodiment of the present invention can execute the data synchronization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data synchronization method.
[0130] It is worth noting that in the embodiment of the above-mentioned data synchronization device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific name of each functional unit is only for the convenience of distinguishing each other, and is not used to limit the scope of protection of the present invention.
[0131] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 5 A block diagram of an exemplary electronic device 5 suitable for implementing embodiments of the present invention is shown. Figure 5 The electronic device 5 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0132] like Figure 5 As shown, the electronic device 5 is in the form of a general purpose computing electronic device. The components of the electronic device 5 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).
[0133] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0134] The electronic device 5 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 5, including volatile and non-volatile media, removable and non-removable media.
[0135] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 5 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.
[0136] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0137] The electronic device 5 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the electronic device 5, and / or any device that enables the electronic device 5 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 5 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. Figure 5 As shown, the network adapter 20 communicates with other modules of the electronic device 5 via the bus 18. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 5, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0138] The processing unit 16 executes various functional applications and page displays by running the programs stored in the system memory 28, for example, implementing the data synchronization method provided in the embodiment of the present invention, which includes:
[0139] Start a data synchronization task for the database to be synchronized, and read the set to be synchronized in the database to be synchronized;
[0140] Writing the read set to be synchronized into the data loading queue;
[0141] The to-be-synchronized set is read from the data loading queue, and the read to-be-synchronized set is written into the target database until all the to-be-synchronized sets are written into the target database.
[0142] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data synchronization method provided by any embodiment of the present invention.
[0143] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, for example, a data synchronization method provided by an embodiment of the present invention is implemented. The method includes:
[0144] Start a data synchronization task for the database to be synchronized, and read the set to be synchronized in the database to be synchronized;
[0145] Writing the read set to be synchronized into the data loading queue;
[0146] The to-be-synchronized set is read from the data loading queue, and the read to-be-synchronized set is written into the target database until all the to-be-synchronized sets are written into the target database.
[0147] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0148] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0149] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0150] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0151] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0152] In addition, the acquisition, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations.
[0153] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A data synchronization method, characterized in that: include: Initiate a data synchronization task for the database to be synchronized, and read the set to be synchronized in the database to be synchronized; Writing the read set to be synchronized into the data loading queue; The to-be-synchronized set is read from the data loading queue, and the read to-be-synchronized set is written into the target database until all the to-be-synchronized sets are written into the target database.
2. The data synchronization method according to claim 1, characterized in that: Before starting the data synchronization task for the database to be synchronized, the following steps are also included: By traversing each set of the database to be synchronized, the set to be synchronized is written into the data storage queue; Accordingly, reading the to-be-synchronized set in the to-be-synchronized database includes: The to-be-synchronized set is read from the data storage queue.
3. The data synchronization method according to claim 1, characterized in that: Writing the read set to be synchronized into the target database includes: The to-be-synchronized set is written into the target database according to the mapping relationship corresponding to the to-be-synchronized set.
4. The data synchronization method according to claim 1, characterized in that: Also includes: The start time of the data synchronization task and the reading time of each set to be synchronized are recorded.
5. The data synchronization method according to claim 4, characterized in that: After writing the read set to be synchronized into the target database, the method further includes: Filtering the operation logs between the current time and the start time to obtain the target log, and writing the target log into the queue to be distributed; Each log in the queue to be distributed is read in sequence, and corresponding data operations are performed on the target database according to the operation type and operation content of each log.
6. The data synchronization method according to claim 5, characterized in that: The target log is obtained by filtering the operation logs between the current time and the startup time, including: Filter the heartbeat log and parameter log in the operation log according to the log type to obtain the intermediate log; According to the startup time and the read time of each set to be synchronized, the logs between the startup time and the read time corresponding to each set to be synchronized in the intermediate log are filtered to obtain the target log.
7. The data synchronization method according to claim 5, characterized in that: After the target log is written into the queue to be distributed, the method further includes: Writing each log into the corresponding data loading queue according to the operation set corresponding to each log in the queue to be distributed; Accordingly, each log in the queue to be distributed is read in sequence, including: Read each log in the data loading queue in sequence.
8. A data synchronization device, characterized in that: include: A starting module, used to start a data synchronization task for a database to be synchronized, and read a set to be synchronized in the database to be synchronized; A writing module, used for writing the read set to be synchronized into a data loading queue; The synchronization module is used to read the to-be-synchronized set from the data loading queue, and write the read to-be-synchronized set into the target database until all the to-be-synchronized sets are written into the target database.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method as described in any one of claims 1-7.
10. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the data synchronization method as described in any one of claims 1-7.