Data synchronization method, device and electronic device

Abstract

The present invention provides a data synchronization method, device and electronic device, which can be applied to the fields of data processing and computer technology. The method comprises: in response to receiving a data comparison instruction, determining an abnormal data table identifier in a database; determining the data table revision number of each abnormal data table identifier in a plurality of data slices; when the plurality of data table revision numbers are not all the same, dividing each candidate data table into a plurality of data blocks according to a preset size; based on the data block revision number of each data block of each candidate data table, determining a standard data block and an error data block from the plurality of candidate data tables, wherein the data block revision number is used to characterize the update status of the data block, the standard data block characterizes that the data stored in the data block is normal, and the error data block characterizes that the data stored in the data block is incorrect; and based on the standard data block, performing data synchronization on the error data block.

Description

Technical Field

[0001] The present invention relates to the field of data processing and computer technology, and in particular to a data synchronization method, device and electronic equipment. Background Art

[0002] The data of a distributed database is physically stored on different nodes of a computer network, but logically belongs to the same entity. This database architecture achieves parallel processing and load balancing of data by distributing and storing data on multiple nodes, thereby improving the overall performance of the database. Distributed database data backup can adopt different strategies and methods, including full backup of data using different data shards on different nodes. When operating data, it is necessary to operate the backup data on all data shards, resulting in data leakage on some data shards, affecting data consistency.

[0003] In the process of implementing the present invention, the inventors found that the prior art has at least the following problems: when synchronizing backup data on multiple data shards, erroneous data tables are determined by data comparison, and the entire data table with errors is synchronized. The comparison process is complicated and the amount of synchronized data is large, resulting in low data synchronization efficiency. Summary of the invention

[0004] In view of the above problems, the present invention provides a data synchronization method, device and electronic device.

[0005] According to a first aspect of the present invention, there is provided a data synchronization method, which is applied to a data synchronization filter, and comprises: in response to receiving a data comparison instruction, determining an abnormal data table identifier in a database, wherein the data comparison instruction is generated by a data synchronization server based on an error log existing in the database; determining data table revision numbers of the abnormal data table identifiers in a plurality of data shards, wherein the database comprises a plurality of data shards, each of which stores a candidate data table corresponding to the abnormal data table identifier and a data table revision number of the candidate data table, wherein the data table revision number is used to characterize the update status of the candidate data table; in the case where the plurality of data table revision numbers are not all the same, dividing each candidate data table into a plurality of data blocks according to a preset size; based on the data block revision number of each data block of each candidate data table, determining a standard data block and an error data block from the plurality of candidate data tables, wherein the data block revision number is used to characterize the update status of the data block, the standard data block characterizing that the data stored in the data block is normal, and the error data block characterizing that the data stored in the data block is incorrect; and synchronizing the error data block based on the standard data block.

[0006] According to an embodiment of the present invention, based on the data block revision number of each data block in each candidate data table, standard data blocks and error data blocks are determined from multiple candidate data tables, including: from multiple data block revision numbers of data blocks in multiple data shards, a data block corresponding to the latest data block revision number representing the data block is determined as a standard data block; data in the standard data block is determined as standard data; data blocks with revision numbers different from the standard data blocks are determined as error data blocks; and data in the error data block is determined as error data.

[0007] According to an embodiment of the present invention, data synchronization is performed on an erroneous data block based on standard data and erroneous data, including: determining the data difference between the erroneous data and the standard data; determining a data block operation statement for data synchronization based on the data difference; and performing data synchronization on the data in the erroneous data block using the data block operation statement.

[0008] According to an embodiment of the present invention, the data synchronization method further includes: determining, for each data shard, the number of operations on data in a data block stored in the data shard; and determining a data block revision number based on the number of operations.

[0009] According to an embodiment of the present invention, the data synchronization method also includes: when the revision numbers of multiple data tables are all the same, determining the data table operation statement used for data synchronization based on the error log; and using the data table operation statement to synchronize the data in the candidate data tables corresponding to the abnormal data table identifier in multiple data shards.

[0010] According to an embodiment of the present invention, a second aspect of the present invention provides a data synchronization method, which is applied to a data synchronization server, including: based on an error log existing in a database, determining an abnormal data table identifier from multiple data tables in the database, wherein the error log indicates that an error occurred when performing data operations on a data table on a data shard in the database, each error log corresponds to a data table, and the abnormal data table identifier is the identifier of the data table where the error log exists; based on the abnormal data table identifier, generating a data comparison instruction, wherein the data comparison instruction is used to instruct the data synchronization filter to divide the candidate data table into multiple data blocks according to a preset size and synchronize data blocks in the multiple data blocks when the data table revision numbers of the candidate data table in multiple data shards are not all the same; the candidate data table is a data table corresponding to the abnormal data table identifier stored in each data shard; and sending the data comparison instruction to the data synchronization filter.

[0011] According to an embodiment of the present invention, based on the error log existing in the database, determining the abnormal data table identifier from multiple data tables in the database includes: determining the error log existing in the database; and determining the abnormal data table identifier from multiple data tables in the database based on the operation record of the error log.

[0012] A third aspect of the present invention provides a data synchronization device, which is applied to a data synchronization filter, and includes: an identification determination module, which is used to determine the abnormal data table identification in the database in response to receiving a data comparison instruction, wherein the data comparison instruction is generated by the data synchronization server based on the error log existing in the database; a revision number determination module, which is used to determine the data table revision numbers of the abnormal data table identification in multiple data slices, wherein the database includes multiple data slices, each of which stores a candidate data table corresponding to the abnormal data table identification and a data table revision number of the candidate data table, and the data table revision number is used to characterize the update status of the candidate data table; a data table partitioning module, which is used to divide each candidate data table into multiple data blocks according to a preset size when the revision numbers of multiple data tables are not all the same; a data determination module, which is used to determine a standard data block and an error data block from multiple candidate data tables based on the data block revision number of each data block of each candidate data table, wherein the data block revision number is used to characterize the update status of the data block, the standard data block indicates that the data stored in the data block is normal, and the error data block indicates that the data stored in the data block is incorrect; and a data synchronization module, which is used to synchronize the error data block based on the standard data block.

[0013] The fourth aspect of the present invention provides a data synchronization device, which is applied to a data synchronization server, including: an exception identification determination module, which is used to determine the exception data table identification from multiple data tables in the database based on the error log existing in the database, wherein the error log indicates that an error occurred when performing data operations on a data table on a data shard in the database, each error log corresponds to a data table, and the exception data table identification is the identification of the data table where the error log exists; an instruction generation module, which is used to generate a data comparison instruction based on the exception data table identification, wherein the data comparison instruction is used to instruct the data synchronization filter to divide the candidate data table into multiple data blocks according to a preset size and synchronize data blocks in the multiple data blocks when the data table revision numbers of the candidate data table in the multiple data shards are not all the same, and the candidate data table is the data table corresponding to the exception data table identification stored in each data shard; and an instruction sending module, which is used to send the data comparison instruction to the data synchronization filter.

[0014] A fifth aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] According to an embodiment of the present invention, in response to a data comparison instruction, an abnormal data table identifier is determined, and multiple candidate data tables in multiple data fragments are determined based on the abnormal data table identifier. According to the data table revision numbers of the multiple candidate data tables, the update status of each of the multiple candidate data tables is determined. The candidate data table can be accurately located through the abnormal data table identifier, without traversing all data tables, reducing the load of data comparison and improving the efficiency of data synchronization. In the case where the revision numbers of multiple data tables are not all the same, the candidate data table is divided into multiple data blocks, and the data block revision numbers of multiple data blocks in multiple data fragments are determined, and the update status of each of the multiple data blocks is determined. The standard data block is used to synchronize the erroneous data block. By dividing the candidate data table into multiple data blocks, the data block where the data to be synchronized is located can be synchronized, reducing the amount of data synchronized. In addition, in the case where there are multiple data to be synchronized in the candidate data table, multiple data blocks can be synchronized at the same time, improving the parallelism of the data synchronization process, thereby further improving the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0017] Figure 1 An application scenario diagram of a data synchronization method, device, and electronic device according to an embodiment of the present invention is shown.

[0018] Figure 2 A flow chart of a data synchronization method according to an embodiment of the present invention is shown.

[0019] Figure 3 A schematic diagram showing the processing of a candidate data table by a data synchronization method according to an embodiment of the present invention is shown.

[0020] Figure 4 A flow chart of a data synchronization method according to another embodiment of the present invention is shown.

[0021] Figure 5 A timing diagram of a data synchronization method according to an embodiment of the present invention is shown.

[0022] Figure 6 A structural block diagram of a data synchronization device according to an embodiment of the present invention is shown.

[0023] Figure 7A structural block diagram of a data synchronization device according to another embodiment of the present invention is shown.

[0024] Figure 8 A block diagram of an electronic device suitable for implementing a data synchronization method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0028] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0029] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, invention and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present invention provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0031] An embodiment of the present invention provides a data synchronization method, which is applied to a data synchronization filter, comprising: in response to receiving a data comparison instruction, determining an abnormal data table identifier in a database, wherein the data comparison instruction is generated by a data synchronization server based on an error log existing in the database; determining data table revision numbers of the abnormal data table identifiers in multiple data shards, wherein the database includes multiple data shards, each of which stores a candidate data table corresponding to the abnormal data table identifier and a data table revision number of the candidate data table, wherein the data table revision number is used to characterize the update status of the candidate data table; in the case where the multiple data table revision numbers are not all the same, dividing each candidate data table into multiple data blocks according to a preset size; based on the data block revision number of each data block of each candidate data table, determining a standard data block and an error data block from the multiple candidate data tables, wherein the data block revision number is used to characterize the update status of the data block, the standard data block characterizes that the data stored in the data block is normal, and the error data block characterizes that the data stored in the data block is incorrect; and based on the standard data block, performing data synchronization on the error data block.

[0032] Figure 1 An application scenario diagram of a data synchronization method, apparatus, device, medium, and program product according to an embodiment of the present invention is shown.

[0033] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a server 105, and a database 106. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0034] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0035] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0036] The server 105 may be a server that provides various services, such as a data synchronization server and a data synchronization filter (only as an example) that provide support for a data synchronization method.

[0037] The database 106 may be a database for storing operations performed and data submitted by users using the terminal devices 101 , 102 , 103 through the network 104 .

[0038] It should be noted that the data synchronization method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data synchronization device provided in the embodiment of the present invention can generally be set in the server 105. The data synchronization method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105 and / or the database 106. Accordingly, the data synchronization device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105 and / or the database 106.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, servers and databases in the embodiment is only for illustration. Any number of terminal devices, networks, servers and databases may be provided as required.

[0040] The following will be based on Figure 1 The scene described by Figure 2~Figure 6 The data synchronization method of the embodiment of the invention is described in detail.

[0041] Figure 2 A flow chart of a data synchronization method according to an embodiment of the present invention is shown.

[0042] like Figure 2 As shown, the data synchronization method of this embodiment includes operations S210 to S250.

[0043] In operation S210 , in response to receiving a data comparison instruction, an abnormal data table identifier in a database is determined.

[0044] In operation S220, respective data table revision numbers of abnormal data table identifiers in a plurality of data shards are determined.

[0045] In operation S230, when the revision numbers of the plurality of data tables are not all the same, each candidate data table is divided into a plurality of data blocks according to a preset size.

[0046] In operation S240, standard data blocks and error data blocks are determined from a plurality of candidate data tables based on a data block revision number of each data block of each candidate data table.

[0047] In operation S250, data synchronization is performed on the error data block based on the standard data block.

[0048] According to an embodiment of the present invention, in response to receiving a data comparison instruction, an abnormal data table identifier in the database is determined, wherein the abnormal data table identifier may include information for identifying the data table, such as the data table number, the data table name, and the data table view number. The data comparison instruction is generated by the data synchronization server based on the error log existing in the database and sent to the data synchronization filter. The data comparison instruction is used to instruct the data synchronization filter to determine the candidate data table that needs to be synchronized based on the abnormal data table identifier, and determine the standard data blocks and error data blocks of the candidate data table from multiple data shards, thereby realizing data synchronization for the candidate data blocks.

[0049] According to an embodiment of the present invention, data table revision numbers of respective abnormal data table identifiers in multiple data shards are determined, wherein a database includes multiple data shards, each data shard stores a candidate data table corresponding to the abnormal data table identifier and a data table revision number of the candidate data table, the data table revision number including a number used to characterize an update status of the candidate data table, such as a system change number (SCN), a count of the number of operations on the data table, and the like.

[0050] According to an embodiment of the present invention, when the revision numbers of multiple data tables are not all the same, each candidate data table is divided into multiple data blocks according to a preset size, wherein the preset size can be 2 16 -1. And determine the data block revision number of each data block of the candidate data table on each data shard.

[0051] According to an embodiment of the present invention, based on the data block revision number of each data block in each candidate data table, a standard data block and an error data block are determined from multiple candidate data tables, and data synchronization is performed on the error data block based on the standard data block. The data block revision number is used to indicate the update status of the data block, the standard data block indicates that the data stored in the data block is normal, and the error data block indicates that the data stored in the data block is incorrect.

[0052] According to an embodiment of the present invention, in response to a data comparison instruction, an abnormal data table identifier is determined, and multiple candidate data tables in multiple data fragments are determined based on the abnormal data table identifier. According to the data table revision numbers of the multiple candidate data tables, the update status of each of the multiple candidate data tables is determined. The candidate data table can be accurately located through the abnormal data table identifier, without traversing all data tables, reducing the load of data comparison and improving the efficiency of data synchronization. In the case where the revision numbers of multiple data tables are not all the same, the candidate data table is divided into multiple data blocks, and the data block revision numbers of multiple data blocks in multiple data fragments are determined, and the update status of each of the multiple data blocks is determined. The standard data block is used to synchronize the erroneous data block. By dividing the candidate data table into multiple data blocks, the data block where the data to be synchronized is located can be synchronized, reducing the amount of data synchronized. In addition, in the case where there are multiple data to be synchronized in the candidate data table, multiple data blocks can be synchronized at the same time, improving the parallelism of the data synchronization process, thereby further improving the efficiency of data synchronization.

[0053] According to an embodiment of the present invention, based on the data block revision number of each data block in each candidate data table, standard data blocks and error data blocks are determined from multiple candidate data tables, including: from multiple data block revision numbers of data blocks in multiple data shards, a data block corresponding to the latest data block revision number representing the data block is determined as a standard data block; data in the standard data block is determined as standard data; data blocks with revision numbers different from the standard data blocks are determined as error data blocks; and data in the error data block is determined as error data.

[0054] According to an embodiment of the present invention, for each data block, a data block revision number representing the latest data block is determined among multiple data block revision numbers of the data block in multiple data shards, and a data block corresponding to the latest data block revision number representing the data block is determined as a standard data block, and data in the standard data block is determined as standard data, wherein the latest data block revision number representing the data block includes a data block revision number representing the most operations on the data block, a data block revision number representing the latest operation performed on the data block, etc.

[0055] According to an embodiment of the present invention, since the data of a data block corresponding to a revision number different from the revision number of a standard data block is data different from the standard data, the data block with a revision number different from the standard data block is determined as an error data block, and the data in the error data block is determined as error data.

[0056] According to an embodiment of the present invention, the latest data block revision number representing the data block is determined from multiple data block revision numbers of the data block, and the data block corresponding to the data block revision number is determined as a standard data block, and the data block corresponding to a data block revision number different from the data block revision number is determined as an error data block. According to the data block revision number, the version of the data stored in the data block can be accurately determined, the data stored in the latest data block can be determined as standard data, and the data stored in the error data block can be determined as error data, so as to facilitate data synchronization.

[0057] According to an embodiment of the present invention, data synchronization is performed on an erroneous data block based on standard data and erroneous data, including: determining the data difference between the erroneous data and the standard data; determining a data block operation statement for data synchronization based on the data difference; and performing data synchronization on the data in the erroneous data block using the data block operation statement.

[0058] According to an embodiment of the present invention, based on the standard data and the error data, the data difference between the error data and the standard data is determined, wherein the types of data difference include different values ​​of data in the standard data and the error data, more rows of standard data than the error data, fewer rows of standard data than the error data, etc., and the content of the data difference includes the number of rows in which the inconsistent data pairs appear in the data block and the respective values ​​of the inconsistent data pairs in the standard data and the error data.

[0059] According to an embodiment of the present invention, based on data differences, data block operation statements for data synchronization are determined, and data in the error data block are synchronized using the data block operation statements, wherein the data block operation statements include data manipulation language (DML). For example, an UPSERT statement is used to synchronize data differences of types such as different values ​​of data in standard data and data in error data and more rows of standard data than error data, and a DELETE statement is used to synchronize data differences of types such as fewer rows of standard data than error data.

[0060] According to an embodiment of the present invention, based on the data difference between the standard data and the error data, a data block operation statement for data synchronization is determined, which can ensure that the data block operation statement is used to synchronize the data with data difference, thereby improving the accuracy of data synchronization. By using the data block operation statement to synchronize the error data block, data synchronization can be performed accurately and quickly.

[0061] Figure 3 A schematic diagram showing the processing of a candidate data table by a data synchronization method according to an embodiment of the present invention is shown.

[0062] like Figure 3 As shown, the data table is stored on four data shards, including data shard 301, data shard 302, data shard 303 and data shard 304, wherein candidate data tables and corresponding data table revision numbers are determined on the four data shards respectively according to the abnormal data table identifier, and when the revision numbers of multiple data tables are not all the same, each candidate data table is divided into five data blocks according to a preset size, wherein the first four data blocks each include 2 16 -1 data, the remaining data is less than 2 16 -1, divided into the fifth data block.

[0063] The data block identifiers of the multiple data blocks in each data shard are determined respectively. Taking SCN as the data block identifier as an example, the SCNs of the first, third, and fifth data blocks in the five data blocks are consistent. For the second data block of each data shard, the SCNs of data block 12, data block 22, data block 32, and data block 42 are 306, 303, 306, and 305 respectively. By comparison, it is found that the SCNs of data block 12 and data block 32 represent the latest data blocks, so data block 12 or data block 32 is determined as a standard data block, and data block 22 and data block 42 are determined as error data blocks. For the fourth data block of each data shard, the SCNs of data block 14, data block 24, data block 34, and data block 44 are 281, 283, 280, and 283, respectively. By comparison, it is found that the SCNs of data block 24 and data block 44 represent the latest data blocks, so data block 24 or data block 44 is determined as a standard data block, and data block 14 and data block 34 are determined as an error data block. Based on the standard data blocks and error data blocks of the second data block and the fourth data block of each data shard, data synchronization can be performed on the error data block in parallel.

[0064] According to an embodiment of the present invention, the data synchronization method further includes: determining, for each data shard, the number of operations on data in a data block stored in the data shard; and determining a data block revision number based on the number of operations.

[0065] According to an embodiment of the present invention, for each data shard, the value of the SCN of the last operation on the data block stored in the data shard may be determined to determine the data block revision number.

[0066] According to an embodiment of the present invention, a data block revision number of each of the multiple data blocks of the candidate data table is determined for each data shard, so that data synchronization is performed based on the data block revision number. By dividing the candidate data table into data blocks, the granularity of data synchronization is reduced, so that when there are multiple errors in the same candidate data table, multiple data blocks can be updated simultaneously, thereby improving the parallelism of data synchronization and thus improving the efficiency of data synchronization.

[0067] According to an embodiment of the present invention, the data synchronization method also includes: when the revision numbers of multiple data tables are all the same, determining the data table operation statement used for data synchronization based on the error log; and using the data table operation statement to synchronize the data in the candidate data tables corresponding to the abnormal data table identifier in multiple data shards.

[0068] According to an embodiment of the present invention, when the data table revision numbers of the candidate data tables in multiple data shards determined based on the abnormal data table identifier are all the same, it indicates that the update situations of the multiple candidate data tables are the same, but since the abnormal data table identifier indicates that there are candidate data tables with erroneous data, it can be determined that there are update errors in the multiple candidate data tables, that is, the data stored in all the candidate data tables are erroneous data.

[0069] According to an embodiment of the present invention, based on an error log, a data table operation statement for data synchronization is determined, and the data in a candidate data table corresponding to an abnormal data table identifier in multiple data shards is synchronized using the data table operation statement, wherein the error log may include each operation on the candidate data table and the standard data of the candidate data table, and by using multiple operations or standard data and erroneous data in the candidate data table, table-level data differences can be determined, and based on the table-level data differences, data table operation statements for data synchronization are determined, and the difference between the data table operation statement and the data block operation statement includes different operation objects.

[0070] According to an embodiment of the present invention, when the revision numbers of multiple data tables are all the same, it is determined that the update conditions of multiple candidate data tables are the same and the stored data is erroneous data, the data table operation statement is determined based on the error log, and the data of the candidate data tables is updated and synchronized. Data synchronization can be completed when the data stored in all candidate data tables are the same, avoiding the special situation of update errors that cause data synchronization failure, and improving the stability of data synchronization.

[0071] According to an embodiment of the present invention, the data synchronization method further includes: determining the data blocks that need to be synchronized and the data differences of the data blocks based on the table-level data differences.

[0072] According to an embodiment of the present invention, data blocks that need to be synchronized and data differences are determined based on table-level data differences, and data block operation statements can be determined based on the data differences and data blocks can be synchronized, thereby avoiding data synchronization of the entire table to reduce the amount of data that needs to be traversed during the data synchronization process, thereby improving the efficiency of data synchronization.

[0073] Figure 4 A flow chart of a data synchronization method according to another embodiment of the present invention is shown.

[0074] like Figure 4 As shown, the data synchronization method of this embodiment includes operations S410 to S430.

[0075] In operation S410, based on the error log existing in the database, an abnormal data table identifier is determined from multiple data tables in the database.

[0076] In operation S420, a data comparison instruction is generated based on the abnormal data table identifier.

[0077] In operation S430 , a data comparison instruction is sent to a data synchronization filter.

[0078] According to an embodiment of the present invention, an error log is stored in the database, wherein the error log indicates that an error occurred when performing data operations on a data table on a data shard in the database, each error log or each record in the error log corresponds to a data table, and the abnormal data table identifier is the identifier of the data table where the error log exists.

[0079] According to an embodiment of the present invention, based on the error log in the database, a data table with errors is determined from multiple data tables in the database, and an abnormal data table identifier is determined based on the data table with errors.

[0080] According to an embodiment of the present invention, a data comparison instruction is generated based on an abnormal data table identifier, wherein the instruction is used to instruct the data synchronization filter to divide the candidate data table into multiple data blocks according to a preset size and synchronize data on the data blocks in the multiple data blocks when the data table revision numbers of the candidate data table in multiple data shards are not all the same, and the candidate data table is a data table corresponding to the abnormal data table identifier stored in each data shard.

[0081] According to an embodiment of the present invention, a data comparison instruction is sent to a data synchronization filter, so that the data synchronization filter performs data comparison and data synchronization based on the data comparison instruction.

[0082] According to an embodiment of the present invention, based on the error log in the database, the abnormal data table identifier is determined, and a data comparison instruction is generated, and the data comparison instruction is sent to the data synchronization filter so that the data synchronization filter performs data comparison and data synchronization based on the data comparison instruction. The data table with errors can be accurately located through the error log, and a data comparison instruction for comparing and synchronizing the data table with errors is generated, so that the data synchronization filter can quickly locate the error data block with data errors based on the data comparison instruction, and synchronize the error data block, thereby improving the synchronization efficiency of data synchronization.

[0083] According to an embodiment of the present invention, based on the error log existing in the database, determining the abnormal data table identifier from multiple data tables in the database includes: determining the error log existing in the database; and determining the abnormal data table identifier from multiple data tables in the database based on the operation record of the error log.

[0084] According to an embodiment of the present invention, an error log existing in a database is determined, wherein the error log includes an operation record of historical operations performed on multiple data tables of the database, and the operation record includes an operation object of the historical operation. According to the operation object, the data table in which the error exists can be determined, and the abnormal data table identifier can be determined.

[0085] According to an embodiment of the present invention, by determining the operation record based on the error log and determining the abnormal data table identifier, the data table with errors can be accurately determined so that the data synchronization filter can perform data comparison and data synchronization based on the data comparison instruction, thereby reducing the amount of data that needs to be traversed during data synchronization, reducing resource consumption, and improving the efficiency of data synchronization.

[0086] Figure 5 A timing diagram of a data synchronization method according to an embodiment of the present invention is shown.

[0087] like Figure 5 As shown, the data synchronization method includes operations S501 to S513.

[0088] In operation S501, the database sends an error log to the data synchronization server. In operation S502, the data synchronization server determines the abnormal data table identifier based on the error log and generates a data comparison instruction. In operation S503, the data synchronization server sends the data comparison instruction to the data synchronization filter. In operation S504, the data synchronization filter determines the abnormal data table identifier based on the data comparison instruction. In operation S505, the data synchronization filter requests the database for the data table revision numbers of the multiple candidate data tables corresponding to the abnormal data table identifier in the multiple data shards. In operation S506, the database sends the multiple data table revision numbers to the data synchronization filter. In operation S507, the data synchronization filter determines the relationship between the multiple data table revision numbers, and divides the candidate data table into multiple data blocks when the multiple data table revision numbers are not all the same. In operation S508, the data synchronization filter requests the database for multiple data block revision numbers corresponding to the multiple data blocks in the multiple data shards. In operation S509, the database sends the multiple data block revision numbers to the data synchronization filter. In operation S510, the data synchronization filter determines a standard data block and an error data block from a plurality of data blocks based on a plurality of data block revision numbers. In operation S511, the data synchronization filter requests standard data in the standard data block from the database. In operation S512, the database sends the standard data to the data synchronization filter. In operation S513, the data synchronization filter performs data synchronization on the error data block based on the standard data.

[0089] Based on the above data synchronization method, the present invention also provides a data synchronization device, which can be applied to a data synchronization filter. Figure 6 The device is described in detail.

[0090] Figure 6 A structural block diagram of a data synchronization device according to an embodiment of the present invention is shown.

[0091] like Figure 6 As shown, the data synchronization device 600 of this embodiment includes an identification determination module 610 , a revision number determination module 620 , a data table division module 630 , a data determination module 640 and a data synchronization module 650 .

[0092] The identification determination module 610 is used to determine the abnormal data table identification in the database in response to receiving the data comparison instruction, wherein the data comparison instruction is generated by the data synchronization server based on the error log existing in the database. In one embodiment, the identification determination module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0093] The revision number determination module 620 is used to determine the data table revision numbers of the abnormal data table identifiers in the multiple data shards, wherein the database includes multiple data shards, each data shard stores a candidate data table corresponding to the abnormal data table identifier and a data table revision number of the candidate data table, and the data table revision number is used to characterize the update status of the candidate data table. In one embodiment, the revision number determination module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0094] The data table partitioning module 630 is used to partition each candidate data table into multiple data blocks according to a preset size when the revision numbers of multiple data tables are not all the same. In one embodiment, the data table partitioning module 630 can be used to perform the operation S230 described above, which will not be described in detail here.

[0095] The data determination module 640 is used to determine a standard data block and an error data block from multiple candidate data tables based on the data block revision number of each data block of each candidate data table, wherein the data block revision number is used to represent the update status of the data block, the standard data block represents that the data stored in the data block is normal, and the error data block represents that the data stored in the data block is incorrect. In one embodiment, the data determination module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0096] The data synchronization module 650 is used to perform data synchronization on the error data block based on the standard data block. In one embodiment, the data synchronization module 650 can be used to perform the operation S250 described above, which will not be described in detail here.

[0097] According to an embodiment of the present invention, the data determination module 640 includes a first data block determination submodule, a first data determination submodule, a second data determination submodule and a second data determination submodule.

[0098] The first data block determination submodule is used to determine, from a plurality of data block revision numbers of the data blocks in a plurality of data slices, a data block corresponding to a latest data block revision number representing the data block as a standard data block.

[0099] The first data determination submodule is used to determine the data in the standard data block as standard data.

[0100] The second data block determination submodule is used to determine a data block having a revision number different from that of a standard data block as an erroneous data block.

[0101] The second data determination submodule is used to determine the data in the error data block as error data.

[0102] According to an embodiment of the present invention, the data synchronization module 650 includes a difference determination submodule, a statement determination submodule and a data synchronization submodule.

[0103] The difference determination submodule is used to determine the data difference between the error data and the standard data.

[0104] The statement determination submodule is used to determine the data block operation statement used for data synchronization based on the data difference.

[0105] The data synchronization submodule is used to synchronize the data in the error data block using the data block operation statement.

[0106] According to an embodiment of the present invention, the data synchronization device 600 further includes a times determination module and a number determination module.

[0107] The number determination module is used to determine the number of operations on the data in the data block stored in the data shard for each data shard.

[0108] The number determination module is used to determine the data block revision number based on the number of operations.

[0109] According to an embodiment of the present invention, the data synchronization device 600 further includes a statement determination module and a table data synchronization module.

[0110] The statement determination module is used to determine the data table operation statement for data synchronization based on the error log when the revision numbers of multiple data tables are all the same.

[0111] The table data synchronization module is used to synchronize the data in the candidate data tables corresponding to the abnormal data table identifiers in the multiple data shards by using the data table operation statements.

[0112] Based on the above data synchronization method, the present invention also provides a data synchronization device, which can be applied to a data synchronization server. Figure 7 The device is described in detail.

[0113] Figure 7 A structural block diagram of a data synchronization device according to another embodiment of the present invention is shown.

[0114] like Figure 7 As shown, the data synchronization device 700 of this embodiment includes an abnormal identification determination module 710, an instruction generation module 720 and an instruction sending module 730.

[0115] The abnormal identification determination module 710 is used to determine the abnormal data table identification from multiple data tables in the database based on the error logs existing in the database, wherein the error log indicates that an error occurred when performing data operations on a data table on a data shard in the database, each error log corresponds to a data table, and the abnormal data table identification is the identification of the data table where the error log exists. In one embodiment, the abnormal identification determination module 710 can be used to perform the operation S410 described above, which will not be repeated here.

[0116] The instruction generation module 720 is used to generate a data comparison instruction based on the abnormal data table identifier, wherein the data comparison instruction is used to instruct the data synchronization filter to divide the candidate data table into multiple data blocks according to a preset size and perform data synchronization on the data blocks in the multiple data blocks when the data table revision numbers of the candidate data table in the multiple data shards are not all the same, and the candidate data table is a data table corresponding to the abnormal data table identifier stored in each data shard. In one embodiment, the instruction generation module 720 can be used to perform the operation S420 described above, which will not be repeated here.

[0117] The instruction sending module 730 is used to send the data comparison instruction to the data synchronization filter. In one embodiment, the instruction sending module 730 can be used to perform the operation S430 described above, which will not be described in detail here.

[0118] According to an embodiment of the present invention, the abnormality identification determination module 710 includes a log determination submodule and an abnormality identification determination submodule.

[0119] The log determination submodule is used to determine the error logs existing in the database.

[0120] The abnormal identification determination submodule is used to determine the abnormal data table identification from multiple data tables in the database based on the operation record of the error log.

[0121] According to an embodiment of the present invention, any multiple modules among the identification determination module 610, the revision number determination module 620, the data table division module 630, the data determination module 640 and the data synchronization module 650, or the abnormal identification determination module 710, the instruction generation module 720 and the instruction sending module 730 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the identification determination module 610, the revision number determination module 620, the data table partition module 630, the data determination module 640, the data synchronization module 650, the abnormal identification determination module 710, the instruction generation module 720 and the instruction sending module 730 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the identification determination module 610, the revision number determination module 620, the data table partition module 630, the data determination module 640, the data synchronization module 650, the abnormal identification determination module 710, the instruction generation module 720 and the instruction sending module 730 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be executed.

[0122] Figure 8 A block diagram of an electronic device suitable for implementing a data synchronization method according to an embodiment of the present invention is schematically shown.

[0123] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0124] In RAM 803, various programs and data required for the operation of electronic device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other via bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0125] According to an embodiment of the present invention, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the input / output (I / O) interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read therefrom is installed into the storage portion 808 as needed.

[0126] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0127] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0128] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0129] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when the processor 801 executes the computer program. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0130] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0131] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0132] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0133] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] It will be appreciated by those skilled in the art that the various embodiments of the present invention and the scope of the present invention are defined by the attached claims and their equivalents, and the features described in the claims may be combined in various combinations and / or combinations, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the various embodiments of the present invention and the scope of the present invention are defined by the attached claims and their equivalents, and the features described in the claims may be combined in various combinations and / or combinations. All of these combinations and / or combinations fall within the scope of the present invention.

[0135] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data synchronization method, applied to a data synchronization filter, characterized in that: The method comprises: In response to receiving a data comparison instruction, determining an abnormal data table identifier in the database, wherein the data comparison instruction is generated by the data synchronization server based on an error log existing in the database; Determine the data table revision numbers of the abnormal data table identifiers in the multiple data shards, wherein the database includes the multiple data shards, each of the data shards stores a candidate data table corresponding to the abnormal data table identifier and the data table revision number of the candidate data table, and the data table revision number is used to characterize the update status of the candidate data table; In the case that the revision numbers of the plurality of data tables are not all the same, dividing each of the candidate data tables into a plurality of data blocks according to a preset size; Based on the data block revision number of each data block of each candidate data table, determining a standard data block and an error data block from the plurality of candidate data tables, wherein the data block revision number is used to characterize the update status of the data block, the standard data block characterizes that the data stored in the data block is normal, and the error data block characterizes that the data stored in the data block is incorrect; and Based on the standard data block, data synchronization is performed on the error data block.

2. The method according to claim 1, characterized in that The step of determining a standard data block and an error data block from a plurality of the candidate data tables based on the data block revision number of each data block in each candidate data table comprises: From the plurality of data block revision numbers of the data block in the plurality of data fragments, determine the data block corresponding to the latest data block revision number representing the data block as the standard data block; Determining the data in the standard data block as standard data; determining the data block having a data block revision number different from that of the standard data block as the error data block; and The data in the error data block is determined as error data.

3. The method according to claim 2, characterized in that The step of performing data synchronization on the error data block based on the standard data block includes: determining a data difference between the error data and the standard data; Based on the data difference, determining a data block operation statement for data synchronization; and The data in the erroneous data block is synchronized using the data block operation statement.

4. The method according to claim 1, characterized in that: The method further comprises: For each of the data slices, determining the number of operations on the data in the data block stored in the data slice; and Based on the number of operations, the data block revision number is determined.

5. The method according to claim 1, characterized in that The method further comprises: In a case where the revision numbers of the plurality of data tables are all the same, determining a data table operation statement for data synchronization based on the error log; and The data table operation statement is used to synchronize the data in the candidate data table corresponding to the abnormal data table identifier in the multiple data shards.

6. A data synchronization method, applied to a data synchronization server, characterized in that: The method comprises: Based on the error log existing in the database, determining the abnormal data table identifier from multiple data tables of the database, wherein the error log indicates that an error occurred when performing data operation on the data table on a data shard in the database, each error log corresponds to one data table, and the abnormal data table identifier is the identifier of the data table where the error log exists; Based on the abnormal data table identifier, a data comparison instruction is generated, wherein the data comparison instruction is used to instruct the data synchronization filter to divide each candidate data table into multiple data blocks according to a preset size when the data table revision numbers of the candidate data tables in multiple data shards are not all the same, and based on the data block revision number of each data block of each candidate data table, a standard data block and an error data block are determined from the multiple candidate data tables, wherein the data block revision number is used to characterize the update status of the data block, the standard data block characterizes that the data stored in the data block is normal, and the error data block characterizes that the data stored in the data block is incorrect; and based on the standard data block, data synchronization is performed on the error data block, the candidate data table is a data table corresponding to the abnormal data table identifier stored in each data shard; and The data comparison instruction is sent to the data synchronization filter.

7. The method according to claim 6, characterized in that The step of determining an abnormal data table identifier from a plurality of data tables in the database based on an error log existing in the database includes: Determining the error log existing in the database; and Based on the operation record of the error log, the abnormal data table identifier is determined from multiple data tables in the database.

8. A data synchronization device, applied to a data synchronization filter, characterized in that: The device comprises: An identification determination module, configured to determine an abnormal data table identification in a database in response to receiving a data comparison instruction, wherein the data comparison instruction is generated by a data synchronization server based on an error log existing in the database; A revision number determination module, used to determine the data table revision numbers of the abnormal data table identifier in each of the multiple data shards, wherein the database includes the multiple data shards, each of the data shards stores a candidate data table corresponding to the abnormal data table identifier and the data table revision number of the candidate data table, and the data table revision number is used to characterize the update status of the candidate data table; A data table partitioning module, configured to partition each of the candidate data tables into a plurality of data blocks according to a preset size when the revision numbers of the plurality of data tables are not all the same; a data determination module, configured to determine a standard data block and an error data block from a plurality of the candidate data tables based on a data block revision number of each of the data blocks in each of the candidate data tables, wherein the data block revision number is used to characterize the update status of the data block, the standard data block characterizes that the data stored in the data block is normal, and the error data block characterizes that the data stored in the data block is incorrect; and A data synchronization module is used to synchronize the erroneous data block based on the standard data block.

9. A data synchronization device, applied to a data synchronization server, characterized in that: The device comprises: An abnormal identification determination module is used to determine an abnormal data table identification from multiple data tables in the database based on an error log existing in the database, wherein the error log indicates that an error occurred when performing data operation on the data table on a data shard in the database, each error log corresponds to one data table, and the abnormal data table identification is the identification of the data table where the error log exists; An instruction generation module is used to generate a data comparison instruction based on the abnormal data table identifier, wherein the data comparison instruction is used to instruct the data synchronization filter to divide each candidate data table into multiple data blocks according to a preset size when the data table revision numbers of the candidate data tables in multiple data shards are not all the same, and determine a standard data block and an error data block from the multiple candidate data tables based on the data block revision number of each data block of each candidate data table, wherein the data block revision number is used to characterize the update status of the data block, the standard data block characterizes that the data stored in the data block is normal, and the error data block characterizes that the data stored in the data block is incorrect; and based on the standard data block, perform data synchronization on the error data block, the candidate data table is a data table corresponding to the abnormal data table identifier stored in each data shard; and The instruction sending module is used to send the data comparison instruction to the data synchronization filter.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

Images