Data synchronization method and device and electronic equipment

Abstract

The invention provides a data synchronization method and device and electronic equipment, which can be applied to the technical fields of computers and databases. The method comprises the steps that the initial value of a fragment count value is determined based on the fragment number of a to-be-synchronized data table in a source database, the to-be-synchronized data table comprises one or more fragments, and each fragment comprises to-be-synchronized sub-data of a first value; for each fragment, writing the to-be-synchronized sub-data into a synchronous memory corresponding to the to-be-synchronized data table; reducing the fragment count value by a first value; and under the condition that the fragment count value is the second value, performing data synchronization on the target database based on the data stored in the synchronous memory.

Description

Technical Field

[0001] The present invention relates to the fields of computer technology and database technology, and in particular to a data synchronization method, device and electronic equipment. Background Art

[0002] In the process of using the database to save data, the data in the database can be distributed and stored on multiple physical machines through data sharding technology to alleviate the storage pressure of the database. When synchronizing the data of the database stored in the form of shards, the synchronization can be completed by synchronizing the data in each shard to the target database separately.

[0003] In the process of implementing the present invention, it is found that the prior art has at least the following problems: in the process of synchronizing data shards in multiple shards to the target database, there may be a problem of missing shards and not being synchronized, resulting in incomplete data in the target database, affecting the accuracy of the data. In addition, in the process of synchronizing data to multiple target databases, each target database needs to write data in multiple data shards, and the data synchronization efficiency is low. Summary of the invention

[0004] In view of the above problems, the present invention provides a data synchronization method, apparatus, device, medium and program product.

[0005] According to a first aspect of the present invention, a data synchronization method is provided, comprising: determining an initial value of a shard count value based on the number of shards of a data table to be synchronized in a source database, wherein the data table to be synchronized includes one or more shards, and each shard includes sub-data to be synchronized of a first value; for each shard, writing the sub-data to be synchronized into a synchronization memory corresponding to the data table to be synchronized; reducing the shard count value by a first value; and when the shard count value is a second value, synchronizing data on a target database based on the data stored in the synchronization memory.

[0006] According to an embodiment of the present invention, the data synchronization method also includes: adding a write lock and a read lock to the source database at the beginning of data synchronization; releasing the write lock after the sub-data to be synchronized in each shard is written into the synchronization memory; and releasing the read lock after the data stored in the synchronization memory is written into the target database.

[0007] According to an embodiment of the present invention, based on the number of shards of the data table to be synchronized in the source database, an initial value of the shard count value is determined, including: based on the data table to be synchronized, determining the number of shards from the log file of the source database; based on the number of shards, setting the initial value of the shard count value.

[0008] According to an embodiment of the present invention, the data synchronization method also includes: in response to a data synchronization pause instruction submitted by a user, pausing data synchronization by using a scheduling policy script and multiple binary synchronization tools, wherein the scheduling policy script is used to control multiple binary synchronization tools, and the multiple binary synchronization tools are used to parse and synchronize the data to be synchronized in the source database, and the data to be synchronized includes multiple data tables to be synchronized; based on the scheduling policy script, determining the synchronized data; based on the data to be synchronized and the synchronized data, determining the unsynchronized data; in response to the data synchronization continue instruction submitted by the user, synchronizing the unsynchronized data to the synchronization memory by using the scheduling policy script and multiple binary synchronization tools.

[0009] According to an embodiment of the present invention, the data synchronization method further includes: for each sub-data to be synchronized, after synchronization of the sub-data to be synchronized is completed, creating a synchronization record corresponding to the data tag of the sub-data to be synchronized in the synchronization file of the synchronization memory.

[0010] According to an embodiment of the present invention, based on the scheduling strategy script, the synchronized data is determined, including: determining the data tags of multiple sub-data to be synchronized; determining one or more data tags that have been synchronized based on one or more synchronization records stored in the synchronization file of the synchronization memory; and determining the synchronized data based on the one or more data tags that have been synchronized.

[0011] According to an embodiment of the present invention, based on the data to be synchronized and the synchronized data, the unsynchronized data is determined, including: calculating the difference between the data tags of all the sub-data to be synchronized and the data tags included in the synchronized data to determine the data tags that are not included in the synchronization record; and determining the unsynchronized data based on the data tags that are not included in the synchronization record.

[0012] According to an embodiment of the present invention, data synchronization is suspended using a scheduling policy script and multiple binary synchronization tools, including: determining multiple binary synchronization tools currently running; and stopping data synchronization of the scheduling policy script and the multiple binary synchronization tools.

[0013] A second aspect of the present invention provides a data synchronization device, comprising:

[0014] An initial value determination module, configured to determine an initial value of a shard count value based on the number of shards of a data table to be synchronized in a source database, wherein the data table to be synchronized includes one or more shards, and each shard includes sub-data to be synchronized of a first value;

[0015] Data writing module, used for each shard,

[0016] Write the sub-data to be synchronized into the synchronization memory corresponding to the data table to be synchronized;

[0017] A count value updating module, used to reduce the slice count value by a first value;

[0018] The data synchronization module is used to synchronize data on the target database based on the data stored in the synchronization memory when the shard count value is the second value.

[0019] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0020] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0021] The fifth aspect of the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0022] According to an embodiment of the present invention, based on the number of shards of the data table to be synchronized, the initial value of the shard count value is determined. After the sub-data to be synchronized in each shard of the data table to be synchronized is written into the synchronization memory, the initial value is reduced until the initial value is reduced to a second value, and it is determined that the sub-data to be synchronized in all shards have been written. By setting the initial value and reducing the initial value after the writing is completed, it can be ensured that after the initial value is reduced to the second value, the synchronization memory includes the sub-data to be synchronized in all shards, thereby ensuring that all data of the data table to be synchronized is stored in the synchronization memory, thereby improving data accuracy. The data in the synchronization memory is fully synchronized to one or more target databases. Since the data written to the target database is obtained from the synchronization memory, it can ensure that the data written to the target database is consistent and correct, and it can avoid the situation in which data errors and data inconsistencies between multiple target databases are caused by synchronization omissions during data synchronization from data shards to multiple target databases, thereby improving the accuracy and efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0024] Figure 1 An application scenario diagram of a data synchronization method, apparatus, device, medium, and program product according to an embodiment of the present invention is shown.

[0025] Figure 2 A flow chart of a data synchronization method according to an embodiment of the present invention is shown.

[0026] Figure 3 A schematic diagram of the synchronization process of the data synchronization method according to an embodiment of the present invention is shown.

[0027] Figure 4 A structural block diagram of a data synchronization device according to an embodiment of the present invention is shown.

[0028] Figure 5 A block diagram of an electronic device suitable for implementing a data synchronization method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0031] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0033] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, invention and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0034] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present invention provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0035] An embodiment of the present invention provides a data synchronization method, comprising: determining an initial value of a shard count value based on the number of shards of a data table to be synchronized in a source database, wherein the data table to be synchronized includes one or more shards, and each shard includes sub-data to be synchronized of a first value; for each shard, writing the sub-data to be synchronized into a synchronization memory corresponding to the data table to be synchronized; reducing the shard count value by a first value; and when the shard count value is a second value, synchronizing data on a target database based on the data stored in the synchronization memory.

[0036] Figure 1 An application scenario diagram of a data synchronization method, apparatus, device, medium, and program product according to an embodiment of the present invention is shown.

[0037] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0038] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0039] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0040] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0041] It should be noted that the data synchronization method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data synchronization device provided in the embodiment of the present invention can generally be set in the server 105. The data synchronization method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data synchronization device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0042] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0043] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The data synchronization method according to the embodiment of the present invention is described in detail.

[0044] Figure 2 A flow chart of a data synchronization method according to an embodiment of the present invention is shown.

[0045] like Figure 2As shown, the data synchronization method of this embodiment includes operations S210 to S240.

[0046] In operation S210, an initial value of a shard count value is determined based on the number of shards of a data table to be synchronized in a source database, wherein the data table to be synchronized includes one or more shards, and each shard includes sub-data to be synchronized of a first value.

[0047] For each shard, perform operations S220 to S230.

[0048] In operation S220, the sub-data to be synchronized is written into a synchronization memory corresponding to the data table to be synchronized.

[0049] In operation S230, the slice count value is reduced by a first value.

[0050] In operation S240, when the shard count value is the second value, data synchronization is performed on the target database based on the data stored in the synchronization memory.

[0051] According to an embodiment of the present invention, since a data table may include a large amount of data, storing all the data in the same data table in the same physical location may cause excessive load on the physical machine. Therefore, the data table can be stored in the database in the form of multiple data shards to reduce the load on a single server and improve the speed of query and update operations.

[0052] According to an embodiment of the present invention, the number of shards of a data table to be synchronized in a source database is determined, and based on the number of shards, an initial value of a shard count value is determined, wherein the data table to be synchronized is a data table that needs to be synchronized to a target database, the source database is a database storing the data table to be synchronized, and the target database is a database for storing the synchronized data table to be synchronized. The shard count value can be used to characterize the number of shards of the data table to be synchronized that still need to be synchronized after synchronization is completed. The initial value of the shard count value is the same as the number of shards of the data table to be synchronized. The data table to be synchronized includes one or more shards, each shard includes sub-data to be synchronized of a first value, and preferably, the first value can be 1.

[0053] According to an embodiment of the present invention, for each shard, the sub-data to be synchronized is written into the synchronization memory corresponding to the data table to be synchronized, wherein the sub-data to be synchronized is the data stored in the shard, and the synchronization memory can be used to temporarily store all the sub-data to be synchronized in the data table to be synchronized.

[0054] According to an embodiment of the present invention, after the sub-data to be synchronized is written into the synchronous memory, the fragment count value is reduced by a first value, indicating that the sub-data to be synchronized of the first value has been written into the synchronous memory.

[0055] According to an embodiment of the present invention, when the shard count value is a second value, data synchronization is performed on the target database based on the data stored in the synchronous memory. Preferably, the second value can be 0. When the shard count value is a second value, it indicates that all the data tables to be synchronized have been written into the synchronous memory.

[0056] According to an embodiment of the present invention, when the shard count value is a second value, data synchronization is performed on the target database based on the data stored in the synchronization memory, wherein the target database may include multiple databases, and the data stored in the synchronization memory may be fully synchronized to the multiple target databases to complete the data synchronization of the multiple target databases.

[0057] According to an embodiment of the present invention, based on the number of shards of the data table to be synchronized, the initial value of the shard count value is determined. After the sub-data to be synchronized in each shard of the data table to be synchronized is written into the synchronization memory, the initial value is reduced until the initial value is reduced to a second value, and it is determined that the sub-data to be synchronized in all shards have been written. By setting the initial value and reducing the initial value after the writing is completed, it can be ensured that after the initial value is reduced to the second value, the synchronization memory includes the sub-data to be synchronized in all shards, thereby ensuring that all data of the data table to be synchronized is stored in the synchronization memory, thereby improving data accuracy. The data in the synchronization memory is fully synchronized to one or more target databases. Since the data written to the target database is obtained from the synchronization memory, it can ensure that the data written to the target database is consistent and correct, and it can avoid the situation in which data errors and data inconsistencies between multiple target databases are caused by synchronization omissions during data synchronization from data shards to multiple target databases, thereby improving the accuracy and efficiency of data synchronization.

[0058] According to an embodiment of the present invention, the data synchronization method also includes: adding a write lock and a read lock to the source database at the beginning of data synchronization; releasing the write lock after the sub-data to be synchronized in each shard is written into the synchronization memory; and releasing the read lock after the data stored in the synchronization memory is written into the target database.

[0059] According to an embodiment of the present invention, when performing read and write operations on a database, in order to ensure data consistency and integrity and prevent problems caused by concurrent operations, the database may be locked, wherein the locking includes a read lock and a write lock.

[0060] According to an embodiment of the present invention, at the beginning of data synchronization, a write lock and a read lock are added to the source database to ensure data consistency. After the sub-data to be synchronized in each shard is written into the synchronization memory, it indicates that all the contents of the data table to be synchronized have been written into the synchronization memory, and this data synchronization no longer needs to write data from any shard to the synchronization memory, so the write lock can be released. After the data stored in the synchronization memory is written into the target database, it indicates that the data writing of this data synchronization has been completed, and the data in the target database will no longer be modified, so the read lock can be released, thereby restoring the normal operation of the database in a timely manner.

[0061] According to an embodiment of the present invention, after the data is written to the synchronous memory, the write lock is released, and after the data is written from the synchronous memory to the target database, the read lock is released. This allows the database lock to be released as early as possible while ensuring data correctness and data security, thereby reducing performance degradation and other problems caused by database locking.

[0062] According to an embodiment of the present invention, based on the number of shards of the data table to be synchronized in the source database, an initial value of the shard count value is determined, including: based on the data table to be synchronized, determining the number of shards from the log file of the source database; based on the number of shards, setting the initial value of the shard count value.

[0063] According to an embodiment of the present invention, based on the data table to be synchronized, the number of shards is determined from the log file of the source database, wherein the log file of the source database can be used to record the number of servers used when the data in the source database is distributedly stored, and the number of shards of the log file can be determined based on the number of servers.

[0064] According to an embodiment of the present invention, an initial value of the shard count value is set based on the number of shards, wherein the initial value can be set to the product of the number of shards and a first value. When the first value is 1, the initial value is equal to the number of shards.

[0065] According to an embodiment of the present invention, based on the log file, the number of shards of the data table to be synchronized is determined and the initial value of the shard count value is further set, so that the number of shards can be accurately obtained and a suitable initial value can be set for the shard count value, thereby improving the efficiency of data synchronization.

[0066] Figure 3 A schematic diagram of the synchronization process of the data synchronization method according to an embodiment of the present invention is shown.

[0067] like Figure 3As shown, the data table to be synchronized includes four shards, namely, the first shard 311, the second shard 312, the third shard 313 and the fourth shard 314. When the first value is 1 and the second value is 0, the initial value of the shard count value is set to 4, and the four shards are written into the synchronous memory 320 respectively. After each shard is written, the shard count value is reduced by 1 until the shard count value is 0, indicating that all four shards of the data table to be synchronized have been written into the synchronous memory 320.

[0068] Since the synchronous memory 320 stores all the data in the data table to be synchronized, the data stored in the synchronous memory 320 is fully written into the first target database 331, the second target database 332 and the third target database 333 respectively, until the data is written into all the target databases, and then the data stored in the synchronous memory 320 is cleared.

[0069] According to an embodiment of the present invention, the data synchronization method also includes: in response to a data synchronization pause instruction submitted by a user, pausing data synchronization by using a scheduling policy script and multiple binary synchronization tools, wherein the scheduling policy script is used to control multiple binary synchronization tools, and the multiple binary synchronization tools are used to parse and synchronize the data to be synchronized in the source database, and the data to be synchronized includes multiple data tables to be synchronized; based on the scheduling policy script, determining the synchronized data; based on the data to be synchronized and the synchronized data, determining the unsynchronized data; in response to the data synchronization continue instruction submitted by the user, synchronizing the unsynchronized data to the synchronization memory by using the scheduling policy script and multiple binary synchronization tools.

[0070] According to an embodiment of the present invention, during the data synchronization process, a scheduling strategy script and multiple binary synchronization tools are used to write data from multiple data shards into the synchronization memory, wherein the scheduling strategy script is used to control multiple binary synchronization tools, and the multiple binary synchronization tools are used to parse and synchronize the data to be synchronized in the source database.

[0071] According to an embodiment of the present invention, in response to a data synchronization pause instruction submitted by a user, data synchronization is paused using a scheduling policy script and multiple binary synchronization tools, wherein the data synchronization pause instruction can be used to instruct the suspension of the currently ongoing data synchronization work, and the data to be synchronized includes multiple data tables to be synchronized.

[0072] According to an embodiment of the present invention, the synchronized data is determined based on the scheduling policy script, wherein the synchronized data may include data that has been written into the synchronization memory among the data to be synchronized.

[0073] According to an embodiment of the present invention, unsynchronized data is determined based on the data to be synchronized and the synchronized data, wherein the unsynchronized data includes data in the data to be synchronized that has not yet been written into the synchronization memory.

[0074] According to an embodiment of the present invention, in response to a data synchronization continue instruction submitted by a user, the unsynchronized data is synchronized to the synchronous memory using a scheduling policy script and multiple binary synchronization tools, wherein the data synchronization continue instruction can be used to instruct to continue the data synchronization work that is currently in a paused state.

[0075] According to an embodiment of the present invention, a scheduling policy script and a binary synchronization tool are used to pause and resume data synchronization. The remaining data that needs to be synchronized after data synchronization resumes is determined based on the synchronized data and the unsynchronized data. This allows for pause and resumption of the data synchronization process. After data synchronization resumes, there is no need to re-synchronize the entire data, only the unsynchronized data needs to be synchronized, thereby improving data synchronization efficiency and user experience.

[0076] According to an embodiment of the present invention, the data synchronization method further includes: for each sub-data to be synchronized, after synchronization of the sub-data to be synchronized is completed, creating a synchronization record corresponding to the data tag of the sub-data to be synchronized in the synchronization file of the synchronization memory.

[0077] According to an embodiment of the present invention, for each sub-data to be synchronized, after the synchronization of the sub-data to be synchronized is completed, a synchronization record corresponding to the data tag of the sub-data to be synchronized is created in the synchronization file of the synchronization memory, wherein the data tag of the sub-data to be synchronized can be generated by a group version resource component (Group Version Resource, GVR), and the synchronization record can determine the sub-data to be synchronized that has been synchronized by recording different data tags.

[0078] According to an embodiment of the present invention, after the sub-data to be synchronized is written into the synchronous memory, a synchronization record corresponding to the data tag of the sub-data to be synchronized is created in the synchronization file of the synchronous memory, thereby ensuring that the sub-data to be synchronized corresponding to all the data tags recorded in the synchronization record have been written into the synchronous memory, thereby improving the accuracy of data synchronization.

[0079] According to an embodiment of the present invention, based on the scheduling strategy script, the synchronized data is determined, including: determining the data tags of multiple sub-data to be synchronized; determining one or more data tags that have been synchronized based on one or more synchronization records stored in the synchronization file of the synchronization memory; and determining the synchronized data based on the one or more data tags that have been synchronized.

[0080] According to an embodiment of the present invention, data tags of each of the plurality of sub-data to be synchronized are determined, and one or more data tags of completed synchronization are determined based on one or more synchronization records stored in the synchronization file of the synchronization memory.

[0081] According to an embodiment of the present invention, based on one or more data markers that have been synchronized, sub-data to be synchronized corresponding to the one or more data markers are determined, and synchronized data is determined based on the one or more sub-data to be synchronized.

[0082] According to an embodiment of the present invention, the data mark is determined through the synchronization record in the synchronization file, and the synchronized data is determined based on the data mark. Since the synchronization record is created after the sub-data to be synchronized is written, the synchronized data can be accurately determined based on the synchronization record.

[0083] According to an embodiment of the present invention, based on the data to be synchronized and the synchronized data, the unsynchronized data is determined, including: calculating the difference between the data tags of all the sub-data to be synchronized and the data tags included in the synchronized data to determine the data tags that are not included in the synchronization record; and determining the unsynchronized data based on the data tags that are not included in the synchronization record.

[0084] According to an embodiment of the present invention, since the unsynchronized data can be calculated by the difference between all the sub-data to be synchronized and the synchronized data, the difference between the data tags of all the sub-data to be synchronized and the data tags included in the synchronized data is calculated to determine the data tags not included in the synchronization record, and the data to be synchronized corresponding to the data tags not included in the synchronization record is the unsynchronized data. When the number of data tags not included in the synchronization record is greater than 1, the unsynchronized data is determined based on the sub-data to be synchronized corresponding to each of the multiple data tags not included in the synchronization record.

[0085] According to an embodiment of the present invention, by calculating the difference between the data tags of all sub-data to be synchronized and the data tags of the synchronized data, the data tags of the sub-data to be synchronized included in the unsynchronized data can be accurately determined, thereby determining the unsynchronized data and avoiding data loss and repeated synchronization during the data synchronization process.

[0086] According to an embodiment of the present invention, data synchronization is suspended using a scheduling policy script and multiple binary synchronization tools, including: determining multiple binary synchronization tools currently running; and stopping data synchronization of the scheduling policy script and the multiple binary synchronization tools.

[0087] According to an embodiment of the present invention, a scheduling policy script is used to determine multiple binary synchronization tools currently running, wherein each scheduling policy script is used to execute a data synchronization task, and multiple binary synchronization tools controlled by each scheduling policy script are used to execute the data synchronization tasks corresponding to the scheduling policy script.

[0088] According to an embodiment of the present invention, the scheduling policy script corresponding to the data synchronization task that needs to be paused and the operations of multiple binary synchronization tools controlled by the scheduling policy script are stopped to stop data synchronization.

[0089] According to an embodiment of the present invention, by stopping the scheduling policy script and multiple binary synchronization tools for executing data synchronization tasks, the data synchronization tasks that need to be stopped can be stopped accurately, and the impact on other ongoing data synchronization tasks can be avoided.

[0090] Based on the above data synchronization method, the present invention also provides a data synchronization device. Figure 4 The device is described in detail.

[0091] Figure 4 A structural block diagram of a data synchronization device according to an embodiment of the present invention is shown.

[0092] like Figure 4 As shown, the data synchronization device 400 of this embodiment includes an initial value determination module 410 , a data writing module 420 , a count value updating module 430 and a data synchronization module 440 .

[0093] The initial value determination module 410 is used to determine the initial value of the shard count value based on the number of shards of the data table to be synchronized in the source database, wherein the data table to be synchronized includes one or more shards, and each shard includes sub-data to be synchronized of a first value. In one embodiment, the initial value determination module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0094] The data writing module 420 is used to write the sub-data to be synchronized into the synchronization memory corresponding to the data table to be synchronized for each shard. In one embodiment, the data writing module 420 can be used to perform the operation S220 described above, which will not be described in detail here.

[0095] The count value updating module 430 is used to reduce the slice count value by a first value. In one embodiment, the count value updating module 430 can be used to perform the operation S230 described above, which will not be described in detail here.

[0096] The data synchronization module 440 is used to synchronize data on the target database based on the data stored in the synchronization memory when the shard count value is the second value. In one embodiment, the data synchronization module 440 can be used to perform the operation S240 described above, which will not be described in detail here.

[0097] According to an embodiment of the present invention, the data synchronization device 400 includes a data locking module, a write lock releasing module and a read lock releasing module.

[0098] The data locking module is used to add write locks and read locks to the source database when data synchronization starts.

[0099] The write lock release module is used to release the write lock after the sub-data to be synchronized in each shard is written into the synchronization memory.

[0100] The read lock release module is used to release the read lock after writing the data stored in the synchronous memory to the target database.

[0101] According to an embodiment of the present invention, the initial value determination module 410 includes a quantity determination submodule and an initial value determination submodule.

[0102] The quantity determination submodule is used to determine the number of shards from the log files of the source database based on the data table to be synchronized.

[0103] The initial value determination submodule is used to set the initial value of the shard count value based on the number of shards.

[0104] According to an embodiment of the present invention, the data synchronization device 400 includes a pause synchronization module, a first data determination module, a second data determination module and a resume synchronization module.

[0105] The synchronization pause module is used to respond to the data synchronization pause instruction submitted by the user, and use the scheduling strategy script and multiple binary synchronization tools to pause data synchronization, wherein the scheduling strategy script is used to control multiple binary synchronization tools, and the multiple binary synchronization tools are used to parse and synchronize the data to be synchronized in the source database, and the data to be synchronized includes multiple data tables to be synchronized.

[0106] The first data determination module is used to determine the synchronized data based on the scheduling policy script.

[0107] The second data determination module is used to determine the unsynchronized data based on the data to be synchronized and the synchronized data.

[0108] The continue synchronization module is used to respond to the data synchronization continue instruction submitted by the user, and synchronize the unsynchronized data to the synchronization memory by using the scheduling strategy script and multiple binary synchronization tools.

[0109] According to an embodiment of the present invention, the data synchronization device 400 includes a record creation module.

[0110] The record creation module is used to create a synchronization record corresponding to the data mark of the sub-data to be synchronized in the synchronization file of the synchronization memory after the synchronization of the sub-data to be synchronized is completed.

[0111] According to an embodiment of the present invention, the first data determination module includes a first mark determination submodule, a second mark determination submodule and a first data determination submodule.

[0112] The first tag determination submodule is used to determine the data tags of each of the plurality of sub-data to be synchronized.

[0113] The second mark determination submodule is used to determine one or more data marks that have been synchronized based on one or more synchronization records stored in the synchronization file of the synchronization memory.

[0114] The first data determination submodule is used to determine the synchronized data based on one or more data markers that have been synchronized.

[0115] According to an embodiment of the present invention, the second data determination module includes a mark calculation submodule and a second data determination submodule.

[0116] The tag calculation submodule is used to calculate the difference between the data tags of all sub-data to be synchronized and the data tags included in the synchronized data, and determine the data tags that are not included in the synchronization record.

[0117] The second data determination submodule is used to determine the unsynchronized data based on the data mark not included in the synchronization record.

[0118] According to an embodiment of the present invention, the pause synchronization module includes a tool determination submodule and a pause synchronization submodule.

[0119] The tool determination submodule is used to determine the multiple binary synchronization tools currently running.

[0120] The Pause Synchronization submodule is used to stop the data synchronization of the scheduling policy script and multiple binary synchronization tools.

[0121] According to an embodiment of the present invention, any multiple modules of the initial value determination module 410, the data writing module 420, the count value update module 430 and the data synchronization module 440 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the initial value determination module 410, the data writing module 420, the count value update module 430 and the data synchronization module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the initial value determination module 410 , the data writing module 420 , the count value updating module 430 and the data synchronization module 440 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0122] Figure 5 A block diagram of an electronic device suitable for implementing a data synchronization method according to an embodiment of the present invention is shown.

[0123] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0124] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0125] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage portion 508 as needed.

[0126] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0127] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0128] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0129] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0130] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0131] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0132] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0133] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0135] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data synchronization method, characterized in that: The method comprises: Determine an initial value of the shard count value based on the number of shards of the data table to be synchronized in the source database, wherein the data table to be synchronized includes one or more shards, and each of the shards includes sub-data to be synchronized of a first value; For each of the shards, Writing the sub-data to be synchronized into the synchronization memory corresponding to the data table to be synchronized; Decrease the fragment count value by the first value; When the shard count value is a second value, data synchronization is performed on the target database based on the data stored in the synchronization memory.

2. The method according to claim 1, characterized in that The method further comprises: When data synchronization starts, adding a write lock and a read lock to the source database; After the sub-data to be synchronized in each of the shards is written into the synchronization memory, releasing the write lock; After the data stored in the synchronous memory is written into the target database, the read lock is released.

3. The method according to claim 1, characterized in that The determining the initial value of the shard count value based on the number of shards of the data table to be synchronized in the source database includes: Based on the data table to be synchronized, determine the number of shards from the log file of the source database; Based on the number of fragments, an initial value of the fragment count value is set.

4. The method according to claim 1, characterized in that: The method further comprises: In response to a data synchronization suspension instruction submitted by a user, suspending data synchronization using a scheduling policy script and a plurality of binary synchronization tools, wherein the scheduling policy script is used to control the plurality of binary synchronization tools, and the plurality of binary synchronization tools are used to parse and synchronize the data to be synchronized in the source database, wherein the data to be synchronized includes a plurality of data tables to be synchronized; Based on the scheduling policy script, determining synchronized data; Determining unsynchronized data based on the data to be synchronized and the synchronized data; In response to the data synchronization continuation instruction submitted by the user, the unsynchronized data is synchronized to the synchronization memory using the scheduling policy script and the plurality of binary synchronization tools.

5. The method according to claim 4, characterized in that The method further comprises: For each of the sub-data to be synchronized, after synchronization of the sub-data to be synchronized is completed, a synchronization record corresponding to the data tag of the sub-data to be synchronized is created in the synchronization file of the synchronization memory.

6. The method according to claim 5, characterized in that The step of determining the synchronized data based on the scheduling policy script includes: Determine the data mark of each of the plurality of sub-data to be synchronized; Determine one or more data markers that have been synchronized based on one or more synchronization records stored in the synchronization file of the synchronization memory; The synchronized data is determined based on one or more of the data marks that have been synchronized.

7. The method according to claim 6, characterized in that The determining the unsynchronized data based on the data to be synchronized and the synchronized data includes: Calculate the difference between the data marks of all the sub-data to be synchronized and the data marks included in the synchronized data to determine the data marks not included in the synchronization record; The unsynchronized data is determined based on the data marker not included in the synchronization record.

8. The method according to claim 4, characterized in that The method of using a scheduling policy script and multiple binary synchronization tools to suspend data synchronization includes: Determining the plurality of binary synchronization tools currently running; Stop data synchronization between the scheduling policy script and the plurality of binary synchronization tools.

9. A data synchronization device, characterized in that: The device comprises: An initial value determination module, configured to determine an initial value of a shard count value based on the number of shards of a data table to be synchronized in a source database, wherein the data table to be synchronized includes one or more shards, and each of the shards includes sub-data to be synchronized of a first value; A data writing module is used for each of the shards. Writing the sub-data to be synchronized into the synchronization memory corresponding to the data table to be synchronized; A count value updating module, used for reducing the slice count value by the first value; The data synchronization module is used to synchronize data on the target database based on the data stored in the synchronization memory when the shard count value is a second value.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images