Data synchronization method and device, computer device, and storage medium

Abstract

An embodiment of the specification provides a data synchronization method and device, computer device and storage medium, wherein the method comprises: obtaining a plurality of batches of to-be-synchronized data from a source database in a batch data obtaining manner, and setting a batch number for each batch of to-be-synchronized data; for each batch, setting a unique data number for each to-be-synchronized data in the batch according to the batch number of the batch and the data obtaining sequence of each to-be-synchronized data in the batch; for each to-be-synchronized data, determining a target synchronization partition of the to-be-synchronized data in a message middleware corresponding to a target database according to primary key information of the to-be-synchronized data; grouping all to-be-synchronized data in each batch according to the target synchronization partition corresponding to each to-be-synchronized data; and sending each to-be-synchronized data in the same group to the corresponding target synchronization partition according to the unique data number. Through the embodiment, the data order in the data synchronization process can be ensured.

Description

TECHNICAL FIELD

[0001] The present specification relates to the field of data processing, and in particular, to a data synchronization method and device, a computer device, and a storage medium. BACKGROUND

[0002] At present, when synchronizing data in a database, the data to be synchronized is usually sent to a message middleware first, and then the message middleware sends the data to be synchronized to a target database. The data order in the data synchronization process has always been a concern in the industry. When the same field has multiple data changes in a short period of time, if the multiple data changes are not sent to the message middleware in the corresponding change order, it may cause errors in the data storage result.

[0003] For example, a user first sets the value of field A to 1, and then modifies the value of field A to 2 in a short period of time. If the two data changes are not sent to the message middleware in the corresponding change order, for example, the latter data change is sent to the message middleware first, and then the former data change is sent to the message middleware, it may cause the value of field A stored in the target database to be 1, resulting in an error in the data storage result. SUMMARY

[0004] An embodiment of the present specification aims to provide a data synchronization method, device, computer device, and storage medium to ensure the data order in the data synchronization process.

[0005] To achieve the above technical solutions, an embodiment of the present specification is implemented as follows:

[0006] In a first aspect, an embodiment of the present specification provides a data synchronization method, comprising:

[0007] According to a batch data acquisition manner, a plurality of batches of data to be synchronized are acquired from a source database, and a batch number is set for each batch of data to be synchronized;

[0008] For each batch, a unique data number is set for each piece of data to be synchronized in the batch according to the batch number of the batch and the data acquisition order of each piece of data to be synchronized in the batch;

[0009] For each piece of data to be synchronized in each batch, the target synchronization partition of the data to be synchronized in the message middleware corresponding to the target database is determined according to the primary key information of the data to be synchronized;

[0010] According to the target synchronization partitions corresponding to all the data to be synchronized included in each batch, all the data to be synchronized are grouped;

[0011] The data synchronization device sends each piece of the to-be-synchronized data located in the same group to the corresponding target synchronization partition according to the unique data number.

[0012] In a second aspect, an embodiment of the present specification provides a data synchronization device, comprising:

[0013] The first number setting unit is configured to obtain a plurality of batches of to-be-synchronized data from the source database in a manner of obtaining data by batches, and set a batch number for each batch of to-be-synchronized data.

[0014] The second number setting unit is configured to set a unique data number for each piece of to-be-synchronized data in each batch according to the batch number of the batch and the data obtaining sequence of each piece of to-be-synchronized data in the batch.

[0015] The target partition determination unit is configured to determine, for each piece of to-be-synchronized data in each batch, a target synchronization partition of the to-be-synchronized data in the message-oriented middleware corresponding to the target database according to the primary key information of the to-be-synchronized data.

[0016] The data grouping unit is configured to group all to-be-synchronized data contained in each batch according to the target synchronization partition corresponding to the to-be-synchronized data.

[0017] The data synchronization unit is configured to send each piece of the to-be-synchronized data located in the same group to the corresponding target synchronization partition according to the unique data number.

[0018] In a third aspect, an embodiment of the present specification provides a computer device, comprising: a processor; and a memory arranged to store computer executable instructions configured to be executed by the processor, the computer executable instructions being configured to perform the steps in the method of the first aspect.

[0019] In a fourth aspect, an embodiment of the present specification provides a storage medium for storing computer executable instructions, the computer executable instructions causing a computer to perform the steps in the method of the first aspect.

[0020] It can be seen that, in the embodiment, the plurality of batches of to-be-synchronized data are acquired from the source database in a batch-by-batch manner, and a batch number is set for each batch of to-be-synchronized data. For each batch, a unique data number is set for each piece of to-be-synchronized data in the batch according to the batch number of the batch and the data acquisition order of each piece of to-be-synchronized data in the batch. For each piece of to-be-synchronized data in each batch, a target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database is determined according to the primary key information of the to-be-synchronized data. According to the target synchronization partitions corresponding to all the to-be-synchronized data included in each batch, all the to-be-synchronized data are grouped, and each piece of to-be-synchronized data in the same group is sent to the corresponding target synchronization partition according to the unique data number. Through the embodiment, the unique data number can reflect the data acquisition order of each piece of to-be-synchronized data in the batch. The batch number, the unique data number set according to the batch number, the partition determination, the data grouping, and the data sending process can achieve the effect of sending the plurality of pieces of to-be-synchronized data that need to be sent to the same partition to the corresponding partition according to the data acquisition order, thereby ensuring the orderliness of data synchronization in the same partition of the message middleware and avoiding the error of data storage results. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only illustrate some embodiments in the present specification, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0022] Figure 1 A scene diagram of data synchronization provided by an embodiment of the present specification;

[0023] Figure 2 A flowchart of a data synchronization method provided by an embodiment of the present specification;

[0024] Figure 3 A scene diagram of data synchronization provided by another embodiment of the present specification;

[0025] Figure 4 A structure diagram of a data synchronization device provided by an embodiment of the present specification;

[0026] Figure 5 A structure diagram of a computer device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0027] In the interests of facilitating a better understanding of the technical solutions in the one or more solutions by those skilled in the art, the technical solutions in the solutions will be described clearly and completely below in conjunction with the accompanying drawings in the solutions. Obviously, the described solutions are only a part of the solutions and not all solutions. Based on the solutions in the one or more solutions, all other solutions obtained by those skilled in the art without creative work should fall within the protection scope of the solutions.

[0028] It should be noted that the one or more solutions and the features in the solutions can be combined with each other in the case of no conflict. The solutions will be described in detail below in conjunction with the accompanying drawings and the solutions.

[0029] In order to ensure the data order in the data synchronization process and avoid data storage result errors, the one or more solutions provide a data synchronization method. In the method, a plurality of batches of to-be-synchronized data are acquired from a source database in a batch data acquisition manner, and a batch number is set for each batch of to-be-synchronized data. For each batch, a unique data number is set for each piece of to-be-synchronized data in the batch according to the batch number of the batch and the data acquisition order of each piece of to-be-synchronized data in the batch. For each piece of to-be-synchronized data in each batch, a target synchronization partition of the to-be-synchronized data in a message middleware corresponding to a target database is determined according to the primary key information of the to-be-synchronized data. All to-be-synchronized data are grouped according to the target synchronization partitions corresponding to all to-be-synchronized data in each batch. Each piece of to-be-synchronized data in the same group is sent to the corresponding target synchronization partition according to the unique data number. On the one hand, through the unique data number, the data acquisition order of each piece of to-be-synchronized data in the batch can be reflected. Through the process of setting the batch number, setting the unique data number according to the batch number, determining the partition, grouping the data, and sending the data, the effect that a plurality of pieces of to-be-synchronized data needing to be sent to the same partition are sent to the corresponding partition according to the data acquisition order can be achieved, so that the order of data synchronization in the same partition of the message middleware is ensured, and the case of data storage result errors is avoided.

[0030] In another aspect, the data synchronization method in the embodiment sets the batch numbers of each batch first, and then sets the unique data numbers of each piece of data to be synchronized in each batch in parallel. Compared with the way of setting the unique data numbers of each piece of data to be synchronized in different batches in series, the efficiency of setting the unique data numbers is effectively improved. The way of setting the unique data numbers of each piece of data to be synchronized in each batch in parallel means that the unique data numbers of each piece of data to be synchronized in different batches are set in parallel, for example, the unique data numbers of each piece of data to be synchronized in the first batch and the second batch are set in parallel. The way of setting the unique data numbers of each piece of data to be synchronized in different batches in series means that the unique data numbers of each piece of data to be synchronized in the first batch are set first, and then the unique data numbers of each piece of data to be synchronized in the second batch are set, and so on.

[0031] Figure 1 A scene diagram of data synchronization provided by an embodiment of the present specification is shown in FIG. 1. As shown in FIG. 1, the scene includes a source database, a data synchronization device, a message middleware and a target database. The data synchronization device can execute the data synchronization method in each embodiment of the present specification to synchronize the data in the source database to the message middleware, and the message middleware synchronizes the received data to the target database. In one embodiment, the source database can be a MySQL master database in a MySQL cluster. When data change occurs in the source database, the data change is recorded in the data change log in the form of data change record. The data change log includes a plurality of data change records, and each data change record is used to record a data change. Figure 1

[0032] In one embodiment, a log analysis program for the data change log, such as a Canal program, can be run in the data synchronization device. The Canal program can connect the MySQL master database through the mysql protocol to analyze the data change log in the MySQL master database. The data change log in the MySQL master database can be specifically a MySQL Binlog (MySQL binary log). In one embodiment, the message middleware can be a Kafka message middleware. The target database is not limited to any form of database.

[0033] In Figure 1 ​In the illustrated scenario, the data synchronization device can perform the data synchronization method in various embodiments of the present specification, obtain multiple batches of to-be-synchronized data from the source database in a batch data acquisition manner, set a batch number for each batch of to-be-synchronized data, for each batch, set a unique data number for each piece of to-be-synchronized data in the batch according to the batch number of the batch and the data acquisition order of each piece of to-be-synchronized data in the batch, for each piece of to-be-synchronized data in each batch, determine a target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database according to the primary key information of the to-be-synchronized data, group all to-be-synchronized data according to the target synchronization partitions corresponding to all to-be-synchronized data included in each batch, and send each piece of to-be-synchronized data in the same group to the corresponding target synchronization partition according to the unique data number. Through this embodiment, the unique data number can reflect the data acquisition order of each piece of to-be-synchronized data in the batch, and the process of setting the batch number, setting the unique data number according to the batch number, determining the partition, grouping the data, and sending the data can achieve the effect of sending multiple pieces of to-be-synchronized data that need to be sent to the same partition to the corresponding partition according to the data acquisition order, thereby ensuring the orderliness of data synchronization in the same partition of the message middleware and avoiding errors in data storage results.

[0034] Figure 2 The flowchart of the data synchronization method provided by an embodiment of the present specification is shown in FIG. 2, which can be performed by the data synchronization device described above. The method includes the following steps: Figure 2

[0035] In step S202, multiple batches of to-be-synchronized data are obtained from the source database in a batch data acquisition manner, and a batch number is set for each batch of to-be-synchronized data.

[0036] In step S204, for each batch, a unique data number is set for each piece of to-be-synchronized data in the batch according to the batch number of the batch and the data acquisition order of each piece of to-be-synchronized data in the batch.

[0037] In step S206, for each piece of to-be-synchronized data in each batch, a target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database is determined according to the primary key information of the to-be-synchronized data.

[0038] In step S208, all to-be-synchronized data is grouped according to the target synchronization partitions corresponding to all to-be-synchronized data included in each batch.

[0039] In step S210, each piece of to-be-synchronized data in the same group is sent to the corresponding target synchronization partition according to the unique data number.

[0040] ​It can be seen that, in the embodiment, the plurality of batches of to-be-synchronized data are obtained from the source database in a batch data obtaining manner, and a batch number is set for each batch of to-be-synchronized data. For each batch, a unique data number is set for each piece of to-be-synchronized data in the batch according to the batch number of the batch and the data obtaining sequence of each piece of to-be-synchronized data in the batch. For each piece of to-be-synchronized data in each batch, the target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database is determined according to the primary key information of the to-be-synchronized data. According to the target synchronization partitions corresponding to all the to-be-synchronized data included in each batch, all the to-be-synchronized data are grouped, and each piece of to-be-synchronized data in the same group is sent to the corresponding target synchronization partition according to the unique data number. Through the embodiment, the data obtaining sequence of each piece of to-be-synchronized data in the batch can be reflected through the unique data number. The process of setting the batch number, setting the unique data number according to the batch number, determining the partition, grouping the data, and sending the data can achieve the effect that the plurality of pieces of to-be-synchronized data that need to be sent to the same partition are sent to the corresponding partition according to the data obtaining sequence, thereby ensuring the order of data synchronization in the same partition of the message middleware and avoiding the error of data storage result.

[0041] In step S202, the data synchronization device obtains a plurality of batches of to-be-synchronized data from the source database in a batch data obtaining manner. Specifically, each batch of to-be-synchronized data includes a plurality of pieces of to-be-synchronized data. The log analysis program in the data synchronization device analyzes each data change record in the data change log in the source database to obtain the to-be-synchronized data corresponding to each data change record. For example, the log analysis program in the data synchronization device analyzes a data change record that records that user A modifies the nickname from “Xiaowang” to “Xiaoli” at a certain time to obtain the to-be-synchronized data corresponding to the data change record, which indicates that the nickname of user A is modified from “Xiaowang” to “Xiaoli”. Correspondingly, after the message middleware synchronizes the to-be-synchronized data to the target database, the target database can modify the nickname of user A stored in the target database from “Xiaowang” to “Xiaoli” according to the to-be-synchronized data.

[0042] The data synchronization device obtains the to-be-synchronized data by analyzing the data change records in the data change log of the source database. The data synchronization device obtains a plurality of batches of to-be-synchronized data from the source database in a batch data obtaining manner, and each batch of to-be-synchronized data includes a plurality of pieces of to-be-synchronized data. Then, the data synchronization device sets a batch number for each batch of to-be-synchronized data.

[0043] The data synchronization device can serially obtain a plurality of batches of to-be-synchronized data from the source database, that is, serially parse each data change record. Then, the data synchronization device serially sets a batch number for each batch of to-be-synchronized data. The batch number can represent the order of obtaining between each batch of to-be-synchronized data. When the data synchronization device obtains each batch of to-be-synchronized data, the obtained to-be-synchronized data is parsed according to the order of the data change records recorded in the data change log. The recording order of the data change records is consistent with the data change order in the source database. Therefore, the order of serially obtaining each batch of to-be-synchronized data by the data synchronization device is consistent with the data change order in the source database.

[0044] It can be understood that for any batch of to-be-synchronized data, the batch numbers corresponding to the to-be-synchronized data are the same. In an embodiment, the data synchronization device sets a batch number for each batch of to-be-synchronized data, specifically:

[0045] (a1) for the to-be-synchronized data obtained in the first batch, determining the obtaining time information of the to-be-synchronized data in the first batch, and determining the batch number of the to-be-synchronized data in the first batch according to the obtaining time information;

[0046] (a2) for the to-be-synchronized data obtained in the Nth batch, processing the batch number of the to-be-synchronized data in the (N-1)th batch to obtain the batch number of the to-be-synchronized data in the Nth batch;

[0047] Wherein, N is a positive integer greater than or equal to 2.

[0048] Specifically, first, for the to-be-synchronized data obtained in the first batch, the obtaining time information of the to-be-synchronized data in the first batch is determined. The obtaining time information can be the timestamp of obtaining the to-be-synchronized data in the first batch. Then, the batch number of the to-be-synchronized data in the first batch is determined according to the obtaining time information. In an embodiment, the batch number of the to-be-synchronized data in the first batch can be determined according to the obtaining time information, that is, the obtaining time information is calculated according to a first calculation rule, and the calculation result is taken as the batch number of the to-be-synchronized data in the first batch. For example, the obtaining time information can be the timestamp of obtaining the to-be-synchronized data in the first batch. The timestamp is multiplied by 10 to obtain a product, and the product is taken as the batch number of the to-be-synchronized data in the first batch. Or, the timestamp is multiplied by 10 and then added by 1 to obtain a value, and the value is taken as the batch number of the to-be-synchronized data in the first batch. The first calculation rule can be set according to requirements. Here, only appropriate examples are given.

[0049] Then, based on the time sequence of obtaining the to-be-synchronized data of each batch, for the to-be-synchronized data obtained in the Nth batch, the batch number of the to-be-synchronized data in the (N-1) th batch is processed to obtain the batch number of the to-be-synchronized data in the Nth batch. In an embodiment, the batch number of the to-be-synchronized data in the (N-1) th batch is processed to obtain the batch number of the to-be-synchronized data in the Nth batch, specifically: a first value is added to the batch number of the to-be-synchronized data in the (N-1) th batch to obtain the batch number of the to-be-synchronized data in the Nth batch.

[0050] Specifically, based on the time sequence of obtaining the to-be-synchronized data of each batch, a first value is added to the batch number of the to-be-synchronized data in the (N-1) th batch, for example, 1 is added, to obtain the batch number of the to-be-synchronized data in the Nth batch. That is, the batch number of the to-be-synchronized data obtained in the second batch is equal to the batch number of the to-be-synchronized data obtained in the first batch + 1, the batch number of the to-be-synchronized data obtained in the third batch is equal to the batch number of the to-be-synchronized data obtained in the second batch + 1, and the batch number of the to-be-synchronized data obtained in the Nth batch is equal to the batch number of the to-be-synchronized data obtained in the (N-1) th batch + 1. Wherein, the first value can be 1, and can also be 2, 3 or other values, which are not limited here.

[0051] It can be seen that through the above process, the batch number of the to-be-synchronized data in the first batch can be determined according to the time information of obtaining the to-be-synchronized data in the first batch, and based on the time sequence of obtaining the to-be-synchronized data of each batch, a first value is added to the batch number of the to-be-synchronized data in the (N-1) th batch to obtain the batch number of the to-be-synchronized data in the Nth batch. Through this batch number setting method, the values of the batch numbers can be gradually increased, and the size order of the batch numbers can reflect the obtaining order of the to-be-synchronized data of each batch, that is, the data change order of the source database, so as to ensure that the size order of the batch numbers is consistent with the data change order of the source database. Wherein, the larger the batch number, the later the data change time, and the smaller the batch number, the earlier the data change time.

[0052] In the above step S204, for each batch, a unique data number is set for each piece of to-be-synchronized data in the batch according to the batch number of the batch and the data obtaining sequence of each piece of to-be-synchronized data in the batch. Wherein, since the to-be-synchronized data is obtained by parsing the data change records recorded in the data change log in sequence, the obtaining sequence of the to-be-synchronized data is consistent with the recording sequence of the data change records, and the recording sequence of the data change records is consistent with the data change order in the source database, therefore, the obtaining sequence of the to-be-synchronized data is consistent with the data change order in the source database.

[0053] Taking a batch of to-be-synchronized data as an example, according to the batch number of the batch and the data acquisition sequence of each piece of to-be-synchronized data in the batch, a unique data number is set for each piece of to-be-synchronized data in the batch, specifically as follows:

[0054] (b1) According to the batch number of the batch, the unique data number of the first acquired to-be-synchronized data in the batch is determined;

[0055] (b2) For the Mth acquired to-be-synchronized data in the batch, the unique data number of the (M-1) th acquired to-be-synchronized data in the batch is processed to obtain the unique data number of the Mth acquired to-be-synchronized data in the batch;

[0056] Wherein, M is a positive integer greater than or equal to 2.

[0057] First, for each batch of to-be-synchronized data, according to the batch number of the batch, the unique data number of the first acquired to-be-synchronized data in the batch is determined. In an embodiment, according to the batch number of the batch, the unique data number of the first acquired to-be-synchronized data in the batch can be determined as follows: the batch number is calculated according to the second calculation rule, and the calculation result is taken as the unique data number of the first acquired to-be-synchronized data in the batch. For example, the batch number is multiplied by 10000 to be taken as the unique data number of the first acquired to-be-synchronized data in the batch, or the batch number is multiplied by 10000 and then added by 1 to be taken as the unique data number of the first acquired to-be-synchronized data in the batch. The second calculation rule can be set according to the demand, which is only an appropriate example here.

[0058] Then, based on the acquisition sequence of each piece of to-be-synchronized data in the batch, for the Mth acquired to-be-synchronized data in the batch, the unique data number of the (M-1) th acquired to-be-synchronized data in the batch is processed to obtain the unique data number of the Mth acquired to-be-synchronized data in the batch. In an embodiment, the unique data number of the (M-1) th acquired to-be-synchronized data in the batch is processed to obtain the unique data number of the Mth acquired to-be-synchronized data in the batch, which can be: adding a second value, such as 1, to the unique data number of the (M-1) th acquired to-be-synchronized data in the batch to obtain the unique data number of the Mth acquired to-be-synchronized data in the batch.

[0059] That is, the unique data number of the second acquired to-be-synchronized data in the same batch is equal to the unique data number of the first acquired to-be-synchronized data in the batch + 1, the unique data number of the third acquired to-be-synchronized data is equal to the unique data number of the second acquired to-be-synchronized data in the batch + 1, and the unique data number of the Mth acquired to-be-synchronized data is equal to the unique data number of the (M-1)th acquired to-be-synchronized data in the batch + 1. The second value can be 1, and can also be 2, 3 or other values, which are not limited here.

[0060] It can be seen that in the embodiment, the unique data number of the first acquired to-be-synchronized data in the batch is determined according to the batch number of the batch, and the unique data number of the Mth acquired to-be-synchronized data in the batch is obtained by adding the second value to the unique data number of the (M-1)th acquired to-be-synchronized data in the batch based on the acquisition order of each to-be-synchronized data in the batch. Therefore, within the same batch, the unique data number is a value that gradually increases, and the size order of the unique data number is consistent with the acquisition order of each to-be-synchronized data, and thus is consistent with the data change order in the source database. Therefore, the size order of the unique data number can reflect the data change order in the source database. The data with a larger unique data number has a later data change time, and the data with a smaller unique data number has an earlier data change time.

[0061] As can be seen from the above batch number setting mode and unique data number setting mode, by setting reasonable first calculation rule, second calculation rule, first value and second value, for example, setting the first calculation rule as multiplying the time information by 10, setting the second calculation rule as multiplying the batch number by 10000, setting the first value as 1, and setting the second value as 1, the unique data number of the last acquired to-be-synchronized data in the to-be-synchronized data acquired in a previous batch and the unique data number of the first acquired to-be-synchronized data in the to-be-synchronized data acquired in a subsequent batch can be a value that is not adjacent to each other, and the value of the unique data number of the last acquired to-be-synchronized data in the to-be-synchronized data acquired in the previous batch is smaller than the value of the unique data number of the first acquired to-be-synchronized data in the to-be-synchronized data acquired in the subsequent batch. The acquisition order between the two to-be-synchronized data, that is, the data change order, can still be reflected by the size order between the two numbers.

[0062] In another aspect, based on the above batch number setting manner and the unique data number setting manner, the batch numbers of the batches can be set in series, and the unique data numbers in each batch can be set in parallel, so as to improve the setting efficiency of the unique data numbers. The setting of the batch numbers of the batches in series means that the batch number of the first batch is set first, and then the batch number of the second batch is set after the setting of the first batch is completed, and so on. The setting of the unique data numbers in each batch in parallel means that the unique data numbers of the pieces of to-be-synchronized data in different batches are set in parallel and synchronously, for example, the unique data numbers of the pieces of to-be-synchronized data in the first batch and the second batch are set synchronously.

[0063] In the step S206, for each piece of to-be-synchronized data in each batch, the target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database is determined according to the primary key information of the to-be-synchronized data. Specifically, for any piece of to-be-synchronized data in any batch, the target synchronization partition of the to-be-synchronized data in the message middleware is determined according to the primary key information of the to-be-synchronized data. The message middleware is a message middleware that synchronizes data to the target database.

[0064] In one embodiment, for each piece of to-be-synchronized data in each batch, the target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database is determined according to the primary key information of the to-be-synchronized data, specifically as follows.

[0065] (c1) calculating a hash value of the primary key information of the to-be-synchronized data;

[0066] (c2) determining the target synchronization partition of the to-be-synchronized data in the message middleware according to the hash value and the number of partitions in the message middleware corresponding to the target database.

[0067] Taking any piece of to-be-synchronized data in any batch as an example, the primary key information of the to-be-synchronized data is hashed to obtain a hash value of the primary key information of the to-be-synchronized data, and then the target synchronization partition of the to-be-synchronized data in the message middleware is determined according to the hash value and the number of partitions in the message middleware corresponding to the target database.

[0068] In one embodiment, the target synchronization partition of the to-be-synchronized data in the message middleware is determined according to the hash value and the number of partitions in the message middleware corresponding to the target database, specifically as follows.

[0069] (c21) numbering each partition in the message middleware, and calculating the remainder of the hash value relative to the number of partitions in the message middleware;

[0070] (c22) If there is a target number equal to the remainder in the value of the number corresponding to the partition of the message middleware, the partition corresponding to the target number in the message middleware is taken as the target synchronization partition of the to-be-synchronized data in the message middleware.

[0071] First, each partition in the message middleware is numbered, and the number can be a positive integer starting from 0 and increasing by 1. Then, the hash value of the primary key information of the to-be-synchronized data is divided by the number of partitions in the message middleware to obtain a remainder. Finally, if there is a target number equal to the remainder in the value of the number corresponding to the partition of the message middleware, the partition corresponding to the target number in the message middleware is taken as the target synchronization partition of the to-be-synchronized data in the message middleware. That is, the value of the number of the target synchronization partition is equal to the remainder.

[0072] For example, if the hash value is divided by the number of partitions in the message middleware to obtain a remainder of 3, then in the message middleware, the partition with the number 3 is taken as the target synchronization partition of the to-be-synchronized data in the message middleware.

[0073] It can be seen that in the embodiment, the target synchronization partition of the to-be-synchronized data in the message middleware can be determined by dividing and obtaining the remainder, and the implementation process is simple and efficient.

[0074] In one embodiment, the to-be-synchronized data includes multiple fields, and accordingly, before step S206, the primary key information of the to-be-synchronized data can be determined in the following manner:

[0075] (d1) For each to-be-synchronized data in each batch, at least one target field used to represent the message entity to which the to-be-synchronized data belongs is determined in each field included in the to-be-synchronized data;

[0076] (d2) In the case where the number of target fields is multiple, for each to-be-synchronized data in each batch, the field values of the target fields of the to-be-synchronized data are spliced to obtain the primary key information of the to-be-synchronized data;

[0077] (d3) In the case where the number of target fields is one, for each to-be-synchronized data in each batch, the field value of the target field of the to-be-synchronized data is taken as the primary key information of the to-be-synchronized data.

[0078] Specifically, first, for any one of the to-be-synchronized data in any batch, in each field contained in the to-be-synchronized data, a target field used to represent a message entity to which the to-be-synchronized data belongs is determined. The message entity to which the to-be-synchronized data belongs is a message entity of data to be added, modified or deleted. For example, when the to-be-synchronized data is used to represent modification of information of a user, the message entity to which the to-be-synchronized data belongs is the user, and when the to-be-synchronized data is used to represent modification of information of a product, the message entity to which the to-be-synchronized data belongs is the product. Here, the target field used to represent the message entity to which the to-be-synchronized data belongs is determined in each field contained in the to-be-synchronized data, for example, when the message entity is a user, the target field includes but is not limited to a real name, a mobile phone number, etc., and when the message entity is a product, the target field includes but is not limited to a product name, a product number, etc.

[0079] Next, in the case where the number of target fields is multiple, for any one of the to-be-synchronized data in any batch, the field values of the target fields of the to-be-synchronized data are spliced as primary key information of the to-be-synchronized data. Alternatively, in the case where the number of target fields is one, for each piece of to-be-synchronized data of each batch, the field value of the target field of the to-be-synchronized data is used as the primary key information of the to-be-synchronized data.

[0080] Of course, in other embodiments, the number of target fields of different to-be-synchronized data in different batches can be different, for any one of the to-be-synchronized data in any batch, if the number of target fields thereof is one, the field value of the target field of the to-be-synchronized data is used as the primary key information of the to-be-synchronized data, and if the number of target fields thereof is multiple, the field values of the target fields of the to-be-synchronized data are spliced as the primary key information of the to-be-synchronized data.

[0081] It can be seen that, through the primary key information, the message entity to which the to-be-synchronized data belongs can be reflected, therefore, for different to-be-synchronized data of the same message entity, the primary key information of these to-be-synchronized data is the same, so that when the primary key information is used for hash calculation and determination of a target synchronization partition of the to-be-synchronized data in the message middleware, it can be ensured that different to-be-synchronized data of the same message entity correspond to the same target synchronization partition, that is, different to-be-synchronized data of the same message entity are sent to the same partition.

[0082] For example, a user changes a nickname twice in a short time, and the two changes correspond to two pieces of to-be-synchronized data. Since the same user has two data changes, the message entities of the two pieces of to-be-synchronized data are the same, both being the user, therefore, the primary key information of the two pieces of to-be-synchronized data is the same, and through the above process, it can be determined that the two pieces of to-be-synchronized data correspond to the same target synchronization partition, that is, the two pieces of to-be-synchronized data are sent to the same partition in the message middleware.

[0083] In step S208, all the to-be-synchronized data is grouped according to the target synchronization partition corresponding to all the to-be-synchronized data contained in each batch, and the to-be-synchronized data corresponding to the same target synchronization partition is grouped into the same group in all the to-be-synchronized data contained in each batch. It can be understood that the to-be-synchronized data in the same group can come from different batches.

[0084] In step S210, each piece of to-be-synchronized data in the same group is sent to the corresponding target synchronization partition according to the unique data number. Since the unique data number can reflect the order of obtaining the to-be-synchronized data, it can also reflect the order of data change. Therefore, sending each piece of to-be-synchronized data in the same group to the corresponding target synchronization partition according to the unique data number, i.e., according to the order of data change, can ensure that the order of data synchronization is consistent with the order of data change, and ensure the orderliness of data change.

[0085] In an embodiment, in each group, the to-be-synchronized data can be sorted according to the relationship from small to large of the unique data number, and the to-be-synchronized data is sent to the target synchronization partition according to the sorting.

[0086] In particular, according to the foregoing, the to-be-synchronized data of the same message entity can correspond to the same target synchronization partition, and therefore, when sending the to-be-synchronized data of the same message entity to the corresponding target synchronization partition, the to-be-synchronized data can be sent according to the order of data change, thereby ensuring the orderliness of data change.

[0087] For example, a user changes the nickname twice in a short period of time, and the two changes correspond to two pieces of to-be-synchronized data. Since the user has changed data twice, the message entity of the two pieces of to-be-synchronized data is the same, i.e., the user, and the primary key information of the two pieces of to-be-synchronized data is the same. Therefore, through the above process, it can be determined that the two pieces of to-be-synchronized data correspond to the same target synchronization partition, i.e., the two pieces of to-be-synchronized data are sent to the same partition in the message middleware. As described above, the larger the unique data number, the later the data change time, and the smaller the unique data number, the earlier the data change time. Therefore, when sending the two pieces of to-be-synchronized data to the corresponding partition, the two pieces of to-be-synchronized data can be sent according to the order from small to large of the unique data number, i.e., according to the order from early to late of the data change, so that the synchronization order of the two pieces of to-be-synchronized data is consistent with the order of data change.

[0088] In one embodiment, step S210, sending each piece of data to be synchronized within the same group to the corresponding target synchronization partition according to its unique data number, can be achieved by using the same thread to send each piece of data to be synchronized within the same group to the corresponding target synchronization partition according to the data acquisition order indicated by the unique data number. As mentioned above, the acquisition order of the data to be synchronized is consistent with the order in which the data was changed.

[0089] Sending each piece of data to be synchronized within the same group to the corresponding target synchronization partition via the same thread can improve the sending efficiency of each piece of data to be synchronized within the same group, and ensure that the data is sent in the order of acquisition represented by the unique data number. This avoids the problems of low sending efficiency and difficulty in ensuring the data is sent in the order of acquisition represented by the unique data number when sending via different threads.

[0090] Figure 3 This is a schematic diagram illustrating a data synchronization scenario provided in another embodiment of this specification, such as... Figure 3 As shown, in this scenario, the data synchronization device includes a data acquisition module, a numbering module, a grouping module, a sorting module, and multiple threads. The data acquisition module can acquire multiple batches of data to be synchronized from the source database in batches. The numbering module can set batch numbers for each batch of data to be synchronized. For each batch, a unique data number is assigned to each piece of data to be synchronized within the batch, based on the batch number and the data acquisition order of each piece of data within the batch. The grouping module can determine the target synchronization partition in the message middleware corresponding to the target database for each piece of data to be synchronized in each batch, based on the primary key information of the data. The module then groups the data to be synchronized in each batch according to the target synchronization partitions corresponding to each piece of data in each batch. The sorting module can sort the data to be synchronized within each group according to the unique data number in ascending order. Each thread can send the data to be synchronized within a group to the corresponding target synchronization partition in ascending order of the unique data number.

[0091] In summary, this embodiment achieves two main goals. First, by setting batch numbers, assigning unique data numbers based on these batch numbers, determining partitions, grouping data, and sending data, multiple data items to be synchronized within the same partition can be sent to their respective partitions in the order of data acquisition. This ensures the orderly synchronization of data within the same partition of the message middleware and avoids data storage errors. Second, for multiple data items to be synchronized for the same message entity, they can be sent to the same partition of the message middleware according to the order of data changes, thus ensuring the orderly changes to the same message entity and guaranteeing the accuracy of data storage for the same message entity.

[0092] Figure 4 A structural schematic diagram of a data synchronization device provided by an embodiment of the present specification is shown in the figure. The data synchronization device can be located in the data synchronization device described above. As shown in the figure, the device comprises: Figure 4

[0093] A first number setting unit 41 is configured to obtain a plurality of batches of to-be-synchronized data from a source database in a manner of batch obtaining data, and set a batch number for each batch of to-be-synchronized data;

[0094] A second number setting unit 42 is configured to set a unique data number for each piece of to-be-synchronized data in each batch according to the batch number of the batch and the data obtaining sequence of each piece of to-be-synchronized data in the batch;

[0095] A target partition determining unit 43 is configured to determine a target synchronization partition of each piece of to-be-synchronized data in each batch in a message-oriented middleware corresponding to a target database according to the primary key information of the to-be-synchronized data;

[0096] A data grouping unit 44 is configured to group all to-be-synchronized data contained in each batch according to the target synchronization partition corresponding to the to-be-synchronized data;

[0097] A data synchronization unit 45 is configured to send each piece of to-be-synchronized data in the same group to the corresponding target synchronization partition according to the unique data number.

[0098] Optionally, the first number setting unit 41 is specifically configured to:

[0099] For the to-be-synchronized data obtained in the first batch, determine the obtaining time information of the to-be-synchronized data in the first batch, and determine the batch number of the to-be-synchronized data in the first batch according to the obtaining time information;

[0100] For the to-be-synchronized data obtained in the Nth batch, process the batch number of the to-be-synchronized data in the (N-1)th batch to obtain the batch number of the to-be-synchronized data in the Nth batch;

[0101] Wherein, N is a positive integer greater than or equal to 2.

[0102] Optionally, the first number setting unit 41 is further specifically configured to:

[0103] According to the first calculation rule, the obtaining time information is calculated, and the calculation result is taken as the batch number of the to-be-synchronized data in the first batch.

[0104] Optionally, the second number setting unit 42 is specifically configured to:​

[0105] determining, according to the batch number of the batch, a unique data number of the to-be-synchronized data of the first acquisition in the batch;

[0106] processing, for the to-be-synchronized data of the Mth acquisition in the batch, the unique data number of the to-be-synchronized data of the (M-1)th acquisition in the batch, to obtain the unique data number of the to-be-synchronized data of the Mth acquisition in the batch;

[0107] wherein the M is a positive integer greater than or equal to 2.

[0108] Optionally, the second number setting unit 42 is further specifically configured to:

[0109] calculating, according to a second calculation rule, the batch number of the batch, and taking the calculation result as the unique data number of the to-be-synchronized data of the first acquisition in the batch.

[0110] Optionally, the target partition determining unit 43 is specifically configured to:

[0111] calculating a hash value of the primary key information of the to-be-synchronized data;

[0112] determining, according to the hash value and the number of partitions in the message middleware corresponding to the target database, a target synchronization partition of the to-be-synchronized data in the message middleware.

[0113] Optionally, the target partition determining unit 43 is further specifically configured to:

[0114] numbering each partition in the message middleware, and calculating a remainder of the hash value relative to the number of partitions in the message middleware;

[0115] if there is a target number equal to the remainder in the value of the number corresponding to the partition of the message middleware, taking the partition corresponding to the target number in the message middleware as the target synchronization partition of the to-be-synchronized data in the message middleware.

[0116] Optionally, the to-be-synchronized data includes a plurality of fields, such as Figure 4 As shown in the figure, the system further includes a primary key determining unit 46 configured to:

[0117] before determining, for each batch of to-be-synchronized data, the target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database according to the primary key information of the to-be-synchronized data, determining, for each batch of to-be-synchronized data, at least one target field in each field included in the to-be-synchronized data for indicating a message entity to which the to-be-synchronized data belongs.

[0118] In the case where the number of the target fields is multiple, for each piece of the to-be-synchronized data of each batch, the field values of the target fields of the to-be-synchronized data are spliced as the primary key information of the to-be-synchronized data.

[0119] In the case where the number of the target fields is one, for each piece of the to-be-synchronized data of each batch, the field value of the target field of the to-be-synchronized data is taken as the primary key information of the to-be-synchronized data.

[0120] Optionally, the data synchronization unit 45 is specifically configured to:

[0121] In the case where the number of the target fields is one, for each piece of the to-be-synchronized data of each batch, the field value of the target field of the to-be-synchronized data is taken as the primary key information of the to-be-synchronized data.

[0122] It should be noted that the data synchronization apparatus in this embodiment can implement each process of the foregoing data synchronization method embodiments and achieve the same effects and functions, which will not be repeated here.

[0123] An embodiment of the present specification also provides a computer device, which can be the data synchronization device described above, and is used to execute the data synchronization method described above, Figure 5 The structural schematic diagram of the computer device provided by an embodiment of the present specification is shown in Figure 5 The computer device can have great differences due to different configurations or performances, and can include one or more processors 1001 and memories 1002, and the memories 1002 can store one or more storage application programs or data. The memory 1002 can be temporary storage or persistent storage. The application programs stored in the memory 1002 can include one or more modules (not shown in the figure), and each module can include a series of computer executable instructions in the computer device. Furthermore, the processor 1001 can be arranged to communicate with the memory 1002 and execute a series of computer executable instructions in the memory 1002 on the computer device. The computer device can also include one or more power supplies 1003, one or more wired or wireless network interfaces 1004, one or more input and output interfaces 1005, one or more keyboards 1006, and the like.

[0124] In one specific embodiment, the computer device includes a processor and a memory arranged to store computer executable instructions configured to be executed by the processor to implement the following processes:

[0125] According to the batch data acquisition mode, the plurality of batches of to-be-synchronized data are acquired from the source database, and batch numbers are set for the to-be-synchronized data in each batch respectively;

[0126] For each batch, unique data numbers are set for each piece of to-be-synchronized data in the batch according to the batch number of the batch and the data acquisition sequence of each piece of to-be-synchronized data in the batch;

[0127] For each piece of to-be-synchronized data in each batch, a target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database is determined according to the primary key information of the to-be-synchronized data;

[0128] According to the target synchronization partitions corresponding to all to-be-synchronized data contained in each batch, the all to-be-synchronized data are grouped;

[0129] Each piece of to-be-synchronized data located in the same group is sent to the corresponding target synchronization partition according to the unique data number.

[0130] It should be noted that the computer device in the embodiment can realize each process of the foregoing data synchronization method embodiment and achieve the same effects and functions, which will not be repeated here.

[0131] The embodiment of the present specification also provides a storage medium for storing computer executable instructions.

[0132] In a specific embodiment, the storage medium can be a U disk, an optical disk, a hard disk, etc. The computer executable instructions stored in the storage medium can realize the following processes when executed by a processor:

[0133] According to the batch data acquisition mode, the plurality of batches of to-be-synchronized data are acquired from the source database, and batch numbers are set for the to-be-synchronized data in each batch respectively;

[0134] For each batch, unique data numbers are set for each piece of to-be-synchronized data in the batch according to the batch number of the batch and the data acquisition sequence of each piece of to-be-synchronized data in the batch;

[0135] For each piece of to-be-synchronized data in each batch, a target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database is determined according to the primary key information of the to-be-synchronized data;

[0136] According to the target synchronization partitions corresponding to all to-be-synchronized data contained in each batch, the all to-be-synchronized data are grouped;

[0137] Each piece of to-be-synchronized data located in the same group is sent to the corresponding target synchronization partition according to the unique data number.

[0138] It should be noted that the storage medium in this embodiment can realize each process of the foregoing data synchronization method embodiment and achieve the same effects and functions, which will not be repeated here.

[0139] The above describes particular embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in an order other than that in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0140] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, system or computer program product. Therefore, the embodiments of the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-readable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing computer usable program code.

[0141] The present specification is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present specification. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0142] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0143] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0144] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0145] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the computer stores the information. The memory is an example of computer readable media.

[0146] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic disks storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "comprising", "comprises", "including", "includes" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article or apparatus that includes the recited element.

[0148] ​​Embodiments of the present specification can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. One or more embodiments of the present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0149] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiments.

[0150] The above only describes the embodiments of the present specification and is not intended to limit the present specification. The present specification can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A data synchronization method, characterized by, The method comprises the following steps: Obtaining a plurality of batches of to-be-synchronized data from a source database in a batch manner, and setting a batch number for each batch of to-be-synchronized data; For each batch, setting a unique data number for each piece of to-be-synchronized data in the batch according to the batch number of the batch and the data acquisition sequence of each piece of to-be-synchronized data in the batch; For each piece of to-be-synchronized data in each batch, determining a target synchronization partition of the to-be-synchronized data in a message middleware corresponding to a target database according to the primary key information of the to-be-synchronized data; Grouping all to-be-synchronized data contained in each batch according to the target synchronization partition corresponding to the to-be-synchronized data; Sending each piece of to-be-synchronized data in the same group to the corresponding target synchronization partition according to the data acquisition sequence represented by the unique data number through the same thread.

2. The method of claim 1, wherein, The step of setting a batch number for each batch of to-be-synchronized data comprises the following steps: For the to-be-synchronized data obtained in the first batch, determining the acquisition time information of the to-be-synchronized data in the first batch, and determining the batch number of the to-be-synchronized data in the first batch according to the acquisition time information; For the to-be-synchronized data obtained in the Nth batch, processing the batch number of the to-be-synchronized data in the (N-1)th batch to obtain the batch number of the to-be-synchronized data in the Nth batch; Wherein, N is a positive integer greater than or equal to 2.

3. The method of claim 2, wherein, The step of determining the batch number of the to-be-synchronized data in the first batch according to the acquisition time information comprises the following steps: According to a first calculation rule, the acquisition time information is calculated, and the calculation result is taken as the batch number of the to-be-synchronized data in the first batch.

4. The method of claim 1, wherein, The step of setting a unique data number for each piece of to-be-synchronized data in the batch according to the batch number of the batch and the data acquisition sequence of each piece of to-be-synchronized data in the batch comprises the following steps: According to the batch number of the batch, the unique data number of the first obtained to-be-synchronized data in the batch is determined; For the Mth obtained to-be-synchronized data in the batch, the unique data number of the (M-1)th obtained to-be-synchronized data in the batch is processed to obtain the unique data number of the Mth obtained to-be-synchronized data in the batch; Wherein, M is a positive integer greater than or equal to 2.

5. The method of claim 4, wherein, The step of determining the unique data number of the first obtained to-be-synchronized data in the batch according to the batch number of the batch comprises the following steps: According to a second calculation rule, the batch number of the batch is calculated, and the calculation result is taken as the unique data number of the first obtained to-be-synchronized data in the batch.

6. The method of claim 1, wherein, The step of determining a target synchronization partition of each piece of to-be-synchronized data in each batch in a message middleware corresponding to a target database according to the primary key information of the to-be-synchronized data comprises the following steps: Calculating the hash value of the primary key information of the to-be-synchronized data; According to the hash value and the number of partitions in the message middleware corresponding to the target database, the target synchronization partition of the to-be-synchronized data in the message middleware is determined.

7. The method of claim 6, wherein, The determining the target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database according to the hash value and the number of partitions in the message middleware includes: numbering each partition in the message middleware, and calculating the remainder of the hash value relative to the number of partitions in the message middleware; if there is a target number equal to the remainder in the value of the number corresponding to the partition of the message middleware, the partition corresponding to the target number in the message middleware is taken as the target synchronization partition of the to-be-synchronized data in the message middleware.

8. The method of claim 1, wherein, The to-be-synchronized data includes multiple fields, and before the determining the target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database according to the primary key information of the to-be-synchronized data for each piece of to-be-synchronized data in each batch, it further includes: For each piece of to-be-synchronized data in each batch, determining at least one target field in each field included in the to-be-synchronized data for representing the message entity to which the to-be-synchronized data belongs; In the case where the number of target fields is multiple, for each piece of to-be-synchronized data in each batch, the field values of the target fields of the to-be-synchronized data are spliced as the primary key information of the to-be-synchronized data; In the case where the number of target fields is one, for each piece of to-be-synchronized data in each batch, the field value of the target field of the to-be-synchronized data is taken as the primary key information of the to-be-synchronized data.

9. A data synchronization apparatus, characterized by comprising: It includes: The first numbering setting unit is configured to acquire a plurality of batches of to-be-synchronized data from the source database in a batch data acquisition manner, and set batch numbers for the to-be-synchronized data in each batch respectively; The second numbering setting unit is configured to set unique data numbers for each piece of to-be-synchronized data in each batch according to the batch number of the batch and the data acquisition order of the to-be-synchronized data in the batch; The target partition determining unit is configured to determine the target synchronization partition of the to-be-synchronized data in the message middleware corresponding to the target database according to the primary key information of the to-be-synchronized data for each piece of to-be-synchronized data in each batch; The data grouping unit is configured to group all to-be-synchronized data included in each batch according to the target synchronization partition corresponding to the to-be-synchronized data; The data synchronization unit is configured to send each piece of to-be-synchronized data in the same group to the corresponding target synchronization partition according to the data acquisition order represented by the unique data number through the same thread.

10. A computer device, comprising: The device includes: a processor; and a memory arranged to store computer-executable instructions configured to be executed by the processor, the computer-executable instructions being used to perform the steps in the method of any one of claims 1-8.

11. A storage medium, characterized by The storage medium is used to store computer-executable instructions, which make the computer execute the steps in the method of any one of claims 1-8.

Images