Method, apparatus and device for processing synchronization data, and storage medium

Abstract

The application relates to the field of big data, and discloses a processing method and device of synchronous data, equipment and a storage medium, which are used for guaranteeing the data consistency of two synchronization parties. The method comprises the following steps: receiving incremental data synchronized in an incremental synchronization mode by a relational database; acquiring a table operation log of the relational database, reading modification records of a primary key value of a relational database table each time the primary key value is modified within a preset time length from the table operation log; acquiring the primary key value before modification from the modification records, and acquiring a last modification value corresponding to each primary key value before modification, wherein the last modification value represents a modification value after the last modification of each primary key value before modification within the preset time length; determining redundant modification records in the table operation log according to the last modification value corresponding to each primary key value before modification and the primary key value before modification; deleting the redundant modification records, and deleting incremental data corresponding to the redundant modification records from the received incremental data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of incremental synchronization, in particular to a method and device for processing synchronization data, and a storage medium. BACKGROUND

[0002] In a big data platform system, data synchronization is a very key step, and when the data volume grows with the business volume, full synchronization of data from the database is not suitable for this scenario, and incremental synchronization is a good solution.

[0003] At present, in the process of synchronizing data in a relational database to a big database in an incremental synchronization manner, the updated time field (such as the updated_date field) of a table is generally used as a condition, the open source tool sqoop (mainly used for data transmission between a big database and a traditional relational database) is used to query the data of the last day and synchronize the data to a temporary table of the big database, and then a data merging operation is performed, that is, the incremental data generated yesterday is merged with the full data before yesterday, and only the row with the latest update time is retained for the same primary key value (referring to a combination of one or more columns of a database table, which can uniquely identify each row in the table, and through which the entity integrity of the table can be forced) in the merging process.

[0004] However, due to business needs or some non-standard database operations, the primary key value of the table in the relational database may be modified, and the data before the modification of the primary key value belongs to the historical state in the data operation process. These redundant historical state data are abnormal data in the relational database table, and when the abnormal data are synchronized to the big database, the data inconsistency between the big database and the relational database will occur when the data are merged, thereby affecting the backtracking and checking of the downstream processed data. SUMMARY

[0005] The main purpose of the present application is to provide a method and device for processing synchronization data, which can ensure the data consistency of both sides during the process of synchronizing data in a relational database to a big database.

[0006] The first aspect of the present application provides a method for processing synchronization data, which comprises the following steps:

[0007] Receiving incremental data synchronized from a relational database in an incremental synchronization manner, wherein the incremental data is the data saved in the relational database within a preset time period;

[0008] Obtaining a table operation log of the relational database, and reading the modification record of the primary key value of the relational database table each time the primary key value is modified within the preset time period from the table operation log;

[0009] obtaining a pre-modification primary key value from the modification record, and obtaining a last modification value corresponding to each pre-modification primary key value, the last modification value representing a modification value after each pre-modification primary key value is lastly modified within the preset time length;

[0010] determining a redundant modification record in the table operation log according to the last modification value corresponding to each pre-modification primary key value and the pre-modification primary key value;

[0011] deleting the redundant modification record, and deleting the incremental data corresponding to the redundant modification record from the received incremental data.

[0012] Optionally, in the first implementation manner of the first aspect, the step of obtaining a pre-modification primary key value from the modification record, and obtaining a last modification value corresponding to each pre-modification primary key value, the last modification value representing a modification value after each pre-modification primary key value is lastly modified within the preset time length, comprises:

[0013] obtaining a record of two fields of the pre-modification primary key value and the post-modification primary key value from the modification record;

[0014] saving the obtained record of two fields of the pre-modification primary key value and the post-modification primary key value to a preset first temporary table;

[0015] querying, in the first temporary table, a last modification value corresponding to each pre-modification primary key value, the last modification value representing a modification value after each pre-modification primary key value is lastly modified within the preset time length;

[0016] the step of determining a redundant modification record in the table operation log according to the last modification value corresponding to each pre-modification primary key value and the pre-modification primary key value, comprises:

[0017] generating a second temporary table according to the queried last modification value corresponding to each pre-modification primary key value;

[0018] determining a redundant modification record in the table operation log according to the second temporary table.

[0019] Optionally, in the second implementation manner of the first aspect, the step of obtaining a record of two fields of the pre-modification primary key value and the post-modification primary key value from the modification record, comprises:

[0020] screening the modification record according to a screening condition that the pre-modification primary key value and the post-modification primary key value are not empty, to obtain screened modification record;

[0021] obtaining a record of two fields of the pre-modification primary key value and the post-modification primary key value from the screened modification record.

[0022] Optionally, in a third implementation form of the first aspect of the present application, the step of querying, in the first temporary table, a last modified value corresponding to each pre-modification primary key value, the last modified value representing a modified value after the pre-modification primary key value is last modified within the preset time length, comprises:

[0023] For any pre-modification primary key value in the first temporary table, obtaining a primary key value after the pre-modification primary key value is first modified;

[0024] determining whether the primary key value after the pre-modification primary key value is first modified is continuously modified within the preset time length;

[0025] when the primary key value after the pre-modification primary key value is first modified is continuously modified, obtaining a modified value after the primary key value is last modified, and determining the obtained modified value as the last modified value corresponding to the pre-modification primary key value;

[0026] when the primary key value after the pre-modification primary key value is first modified is not continuously modified, determining the primary key value after the pre-modification primary key value is first modified as the last modified value corresponding to the pre-modification primary key value.

[0027] Optionally, in a fourth implementation form of the first aspect of the present application, the step of generating a second temporary table according to the queried last modified value corresponding to each pre-modification primary key value comprises:

[0028] replacing the modified primary key value in the first temporary table with the queried last modified value corresponding to each pre-modification primary key value to obtain the second temporary table.

[0029] Optionally, in a fifth implementation form of the first aspect of the present application, the step of determining the redundant modification record in the table operation log according to the second temporary table comprises:

[0030] querying the modification record in the table operation log with the pre-modification primary key value in the second temporary table as an index to obtain a to-be-merged modification record;

[0031] replacing the modified primary key value in the to-be-merged modification record with the last modified value in the second temporary table;

[0032] for a plurality of to-be-merged modification records with the same last modified value, obtaining a modification time of each modification record, determining a modification record with a closest modification time as a reserved modification record, and determining modification records other than the reserved modification record in the plurality of to-be-merged modification records as the redundant modification record in the table operation log.

[0033] Optionally, in a sixth implementation form of the first aspect of the application, the step of obtaining the table operation log of the relational database, and reading, from the table operation log, a modification record of a primary key value of a relational database table each time the primary key value is modified within the preset time length, comprises:

[0034] reading a table operation log written to the big data platform by a pre-configured data incremental synchronization tool, and taking the read table operation log as the table operation log of the relational database, wherein the data incremental synchronization tool is configured to synchronize data increments in the relational database to a big database in the big data platform;

[0035] reading, from the table operation log, a modification record of a primary key value of a relational database table each time the primary key value is modified within the preset time length.

[0036] The second aspect of the application provides a processing device for synchronizing data, comprising:

[0037] a receiving module configured to receive incremental data synchronized in an incremental synchronization manner from a relational database, the incremental data being data saved by the relational database within a preset time length;

[0038] a first obtaining module configured to obtain a table operation log of the relational database, and read, from the table operation log, a modification record of a primary key value of a relational database table each time the primary key value is modified within the preset time length;

[0039] a second obtaining module configured to obtain, from the modification record, a pre-modification primary key value, and obtain a last modification value corresponding to each pre-modification primary key value, the last modification value representing a modification value after each pre-modification primary key value is modified last time within the preset time length;

[0040] a determining module configured to determine a redundant modification record in the table operation log according to the last modification value corresponding to each pre-modification primary key value and the pre-modification primary key value;

[0041] a deleting module configured to delete the redundant modification record, and delete, from the received incremental data, incremental data corresponding to the redundant modification record.

[0042] Optionally, in a first implementation form of the second aspect of the application, the second obtaining module is further configured to:

[0043] obtain a record of two fields of a pre-modification primary key value and a post-modification primary key value from the modification record;

[0044] save the obtained record of two fields of the pre-modification primary key value and the post-modification primary key value to a preset first temporary table;

[0045] In the first temporary table, a last modification value corresponding to each pre-modification primary key value is queried, the last modification value representing a modification value after each pre-modification primary key value is lastly modified within the preset time length;

[0046] The determining module is further configured to:

[0047] generate a second temporary table according to the queried last modification value corresponding to each pre-modification primary key value;

[0048] determine the redundant modification record in the table operation log according to the second temporary table.

[0049] Optionally, in a second implementation manner of the second aspect of the present application, the second obtaining module is further configured to:

[0050] screen the modification record according to a screening condition that the pre-modification primary key value and the post-modification primary key value are both not empty, to obtain a screened modification record;

[0051] obtain a record of the pre-modification primary key value and the post-modification primary key value from the screened modification record.

[0052] Optionally, in a third implementation manner of the second aspect of the present application, the second obtaining module is further configured to:

[0053] for any pre-modification primary key value in the first temporary table, obtain a post-modification primary key value after the pre-modification primary key value is first modified;

[0054] determine whether the post-modification primary key value after the pre-modification primary key value is first modified is continuously modified within the preset time length;

[0055] when the post-modification primary key value after the pre-modification primary key value is first modified is continuously modified, obtain a modification value after the post-modification primary key value is lastly modified, and determine the obtained modification value as a last modification value corresponding to the pre-modification primary key value;

[0056] when the post-modification primary key value after the pre-modification primary key value is first modified is not continuously modified, determine the post-modification primary key value after the pre-modification primary key value is first modified as the last modification value corresponding to the pre-modification primary key value.

[0057] Optionally, in a fourth implementation manner of the second aspect of the present application, the determining module is further configured to:

[0058] replace the post-modification primary key value in the first temporary table with the queried last modification value corresponding to each pre-modification primary key value, to obtain a second temporary table.

[0059] Optionally, in a fifth implementation manner of the second aspect of the present application, the determining module is further configured to:

[0060] In the table operation log, the modified record is queried by the pre-modification primary key value in the second temporary table, and the to-be-merged modified record is obtained;

[0061] The post-modification primary key value in the to-be-merged modified record is replaced by the last modification value in the second temporary table;

[0062] For a plurality of to-be-merged modified records with the same last modification value, the modification time of each modified record is obtained, the modified record with the closest modification time to the current time is determined as a reserved modified record, and the modified records in the plurality of to-be-merged modified records except the reserved modified record are determined as redundant modified records in the table operation log.

[0063] Optionally, in a sixth implementation manner of the second aspect of the present application, the first obtaining module is further configured to:

[0064] read the table operation log written by the pre-configured data incremental synchronization tool to the big data platform, and take the read table operation log as the table operation log of the relational database, wherein the data incremental synchronization tool is used to synchronize the data increment in the relational database to a big database in the big data platform.

[0065] read the modified record of the primary key value of the relational database table each time the primary key value is modified within the preset time length from the table operation log.

[0066] The third aspect of the present application provides a processing device for synchronizing data, the processing device for synchronizing data comprising a memory and at least one processor, the memory storing instructions; the at least one processor invoking the instructions in the memory to enable the processing device for synchronizing data to perform the processing method for synchronizing data described above.

[0067] The fourth aspect of the present application provides a storage medium, the storage medium storing instructions, the instructions being executed by a processor to implement the processing method for synchronizing data described above.

[0068] The application receives incremental data synchronized in an incremental synchronization manner from a relational database, the incremental data being data saved by the relational database in a preset time length; a table operation log of the relational database is acquired, and a modification record of a primary key value of a relational database table being modified each time in the preset time length is read from the table operation log; a primary key value before modification is acquired from the modification record, and a last modification value corresponding to each primary key value before modification is acquired, the last modification value representing a modification value after the primary key value before modification is modified last time in the preset time length; redundant modification records in the table operation log are determined according to the last modification value corresponding to each primary key value before modification and the primary key value before modification; the redundant modification records are deleted, and incremental data corresponding to the redundant modification records is deleted from the received incremental data. In this way, the redundant modification records of the primary key value of the relational database table and the corresponding incremental data are deleted in the process of synchronizing data in the relational database to a large database, so that the data consistency of both sides of synchronization is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 A flowchart of an embodiment of the data synchronization processing method of the application;

[0070] Figure 2 A schematic diagram of a first temporary table in the embodiment of the application;

[0071] Figure 3 A schematic diagram of a second temporary table in the embodiment of the application;

[0072] Figure 4 A module schematic diagram of an embodiment of the data synchronization processing device of the application;

[0073] Figure 5 A structure schematic diagram of the data synchronization processing device provided by the embodiment of the application. DETAILED DESCRIPTION

[0074] The embodiment of the application provides a data synchronization processing method, device, equipment and storage medium, which realizes the data consistency of both sides of synchronization in the process of synchronizing data in a relational database to a large database.

[0075] The terms "first", "second", "third", "fourth" and the like in the description and claims of the present application and above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed herein can interchange depending upon the context in which it is used and the intention is to recite a single feature having two or more names and not a limitation on the features themselves, as such. Also, the use of "including", "comprising", "having" and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items not specifically listed. Furthermore, the terms "a" or "an", as used herein, are defined as one or more, unless explicitly stated otherwise.

[0076] For the convenience of understanding, the specific flow of the embodiment of the method for processing synchronization data of the present application is described below.

[0077] Referring to Figure 1 , Figure 1 The flowchart of one embodiment of the method for processing synchronization data of the present application is shown in the figure, which comprises:

[0078] Step 101, receiving the incremental data synchronized by the relational database in an incremental synchronization manner, the incremental data being the data saved by the relational database within a preset time length;

[0079] In the embodiment, the method for processing synchronization data is applied to a server, and the server is equipped with a big data platform. The embodiment is applied to the scene of synchronizing the data in the relational database to the big database of the big data platform. The relational database can be MySQL, Oracle, etc., and the big database can be Hive, Hbase, etc., which are not limited here.

[0080] Firstly, the server receives the incremental data synchronized by the relational database in an incremental synchronization manner, and the incremental data is the data saved by the relational database within a preset time length. The preset time length can be flexibly set according to business requirements, such as the last day, the last 2 hours, etc. In the specific implementation, the data updated in the last day can be queried from the relational database to the temporary table of the big database by taking the updated_date field of the table as the condition, and the data synchronized is the incremental data.

[0081] Step 102, obtaining the table operation log of the relational database, and reading the modification record of the primary key value of the relational database table each time being modified within the preset time length from the table operation log;

[0082] After receiving the incremental data from the relational database synchronization, the server obtains a table operation log of the relational database, and the table operation log records modification records based on a primary key value of a table of the relational database.

[0083] Specifically, the step 102 can include: reading a table operation log written by a pre-configured data incremental synchronization tool to a big data platform, taking the read table operation log as a table operation log of a relational database, wherein the data incremental synchronization tool is used to synchronize data increments in the relational database to a big database in the big data platform; and reading, from the table operation log, modification records of a primary key value of a table of the relational database each time the primary key value is modified within a preset time length.

[0084] In the embodiment, data incremental synchronization between the relational database and the big database can be implemented by using a data incremental synchronization tool OGG (oracle golden gate, referred to as OGG). The OGG is a log-based structured data replication software, which can realize real-time capture, transformation and delivery of a large amount of transaction data, and realize data synchronization between a source database and a target database, and keep sub-second data delay. After the OGG synchronizes data increments in the relational database to the big database of the big data platform, the OGG writes a table operation log of the relational database to the big data platform, and the server obtains the table operation log written by the OGG to the big data platform.

[0085] Then, the server reads, from the table operation log, modification records of a primary key value of a table of the relational database each time the primary key value is modified within a preset time length, and the modification records include operation time, operation type (such as insertion, update, etc.), a primary key value before modification and a primary key value after modification. The primary key value refers to a combination of one column or multiple columns of a database table, and the value can uniquely identify each row in the table, and the entity integrity of the table can be forced through the primary key value.

[0086] In step 103, the primary key value before modification is obtained from the modification records, and a last modification value corresponding to each primary key value before modification is obtained. The last modification value represents a modification value after each primary key value before modification is modified for the last time within a preset time length.

[0087] In this step, the server obtains the primary key value before modification from the modification record and obtains the last modification value corresponding to each primary key value before modification, wherein the last modification value represents the modification value after the last modification of each primary key value before modification within a preset time length. Within the preset time length, any primary key value before modification (i.e. the original primary key value not subjected to modification) can be modified multiple times. For example, the primary key value before modification is 10001, 10001 is modified to 10002 in a modification record, then 10002 is modified to 10003 in another modification record, and then 10003 is modified to 10005 in another modification record, and the last modification value corresponding to the primary key value before modification 10001 is 10005.

[0088] In step 104, the redundant modification record in the table operation log is determined according to the last modification value corresponding to each primary key value before modification and the primary key value before modification.

[0089] In this step, the server determines the redundant modification record in the table operation log of the relational database according to the last modification value corresponding to each primary key value before modification and the primary key value before modification. Specifically, for a primary key value before modification, the server can only retain the modification record when it is modified last time, and the modification record when it is modified before is determined as a redundant modification record. Taking the primary key value 10001 as an example, assuming that it is modified to 10005 last time, 10001 is modified to 10002 and 10002 is modified to 10003 are all redundant modification records of the primary key value.

[0090] In step 105, the redundant modification record is deleted, and the incremental data corresponding to the redundant modification record is deleted from the received incremental data.

[0091] In this step, the server deletes the redundant modification record from the table operation log and deletes the incremental data corresponding to the redundant modification record from the received incremental data, so as to guarantee the data consistency of the two synchronization parties, improve the data quality, and make it more convenient to trace back and check the downstream processing data.

[0092] In the embodiment, the server receives the incremental data synchronized in an incremental synchronization manner of the relational database, the incremental data is the data saved by the relational database within a preset time length, obtains the table operation log of the relational database, reads the modification record of the primary key value of the relational database table each time being modified within the preset time length from the table operation log, obtains the primary key value before modification from the modification record, and obtains the last modification value corresponding to each primary key value before modification, the last modification value representing the modification value after the last modification of each primary key value before modification within the preset time length, determines the redundant modification record in the table operation log according to the last modification value corresponding to each primary key value before modification and the primary key value before modification, deletes the redundant modification record, and deletes the incremental data corresponding to the redundant modification record from the received incremental data. In this way, the redundant modification record of the primary key value of the relational database table and the deletion of the corresponding incremental data are realized in the process of synchronizing the data in the relational database to the large database, so that the data consistency of both sides is ensured.

[0093] Further, based on the first embodiment of the method for processing the synchronized data, the second embodiment of the method for processing the synchronized data is provided.

[0094] In the embodiment, the step 103 can include: obtaining the record of the two fields of the primary key value before modification and the primary key value after modification from the modification record; saving the obtained record of the two fields of the primary key value before modification and the primary key value after modification to the preset first temporary table; and querying the last modification value corresponding to each primary key value before modification in the first temporary table, the last modification value representing the modification value after the last modification of each primary key value before modification within the preset time length.

[0095] In the embodiment, after the server reads the modification record of the primary key value of the relational database table each time being modified within the preset time length from the table operation log, the record of the two fields of the primary key value before modification and the primary key value after modification is obtained from the modification record and saved to the preset first temporary table, and the table has the primary key change history of all the primary key change operation records.

[0096] Further, the step of obtaining the record of the two fields of the primary key value before modification and the primary key value after modification from the modification record can include: screening the modification record according to the screening condition that the primary key value before modification and the primary key value after modification are not empty, to obtain the screened modification record; and obtaining the record of the two fields of the primary key value before modification and the primary key value after modification from the screened modification record.

[0097] In the embodiment, for some data insertion operations, a new primary key value is created, and in this case, the primary key value before modification in the corresponding modification record is empty, and only data update operation causes modification of the primary key value, and in this case, the primary key value before modification and the primary key value after modification in the corresponding modification record are not empty. To exclude interference of operations other than the primary key value change, the server can screen the modification record according to the screening condition that the primary key value before modification and the primary key value after modification are not empty, obtain the screened modification record, and then obtain the record of the primary key value before modification and the primary key value after modification from the screened modification record, so as to ensure that the obtained record is all the primary key value change record.

[0098] After the server saves the record of the primary key value before modification and the primary key value after modification into the first temporary table, the server queries the last modification value corresponding to each primary key value before modification in the first temporary table. Referring to Figure 2 , Figure 2 is a schematic diagram of the first temporary table in the embodiment. In the first temporary table, the primary key change history in a preset time length is recorded, and the server can obtain the last modification value corresponding to the primary key value before modification 10001, 10002, 10003 and 30001 as 10005, 10005, 10005 and 30002 respectively through a query operation.

[0099] Correspondingly, the step 104 can specifically include: generating a second temporary table according to the last modification value corresponding to each primary key value before modification obtained by querying; and determining the redundant modification record in the table operation log according to the second temporary table.

[0100] After the server queries the last modification value corresponding to each primary key value before modification in the first temporary table, the server generates a second temporary table according to the last modification value corresponding to each primary key value before modification obtained by querying, and the second temporary table is a mapping table of the primary key value change result.

[0101] Further, the step of generating the second temporary table according to the last modification value corresponding to each primary key value before modification obtained by querying can include: replacing the primary key value after modification in the first temporary table with the last modification value corresponding to each primary key value before modification obtained by querying to obtain the second temporary table.

[0102] Referring to Figure 3 , Figure 3 is a schematic diagram of the second temporary table in the embodiment. The Figure 2The "modified primary key value" in the first temporary table is replaced with the "last modified value," thus creating a second temporary table. In this second temporary table, the last modified values ​​corresponding to the original primary key values ​​10001, 10002, 10003, and 30001 are 10005, 10005, 10005, and 30002, respectively. The server then uses this second temporary table to identify redundant modification records in the table operation log.

[0103] In this embodiment, the primary key change process of all primary key change operations is recorded in the first temporary table, and the primary key value change result is recorded in the second temporary table. The operation is simple and convenient, and provides a prerequisite guarantee for the subsequent determination of redundant modification records in the table operation log.

[0104] Furthermore, based on the second embodiment of the synchronous data processing method of the present invention, a third embodiment of the synchronous data processing method of the present invention is proposed.

[0105] In this embodiment, the step of querying the last modified value corresponding to each primary key value before modification in the first temporary table, where the last modified value represents the modified value after the last modification of each primary key value before modification within a preset time period, may further include: for any primary key value before modification in the first temporary table, obtaining the primary key value after the first modification of any primary key value before modification; determining whether the primary key value after the first modification is further modified within the preset time period; when the primary key value after the first modification is further modified, obtaining the last modified value after the first modification of the primary key value after modification, and determining the obtained modified value as the last modified value corresponding to any primary key value before modification; when the primary key value after the first modification is not further modified, determining the primary key value after the first modification as the last modified value corresponding to any primary key value before modification.

[0106] by Figure 2 For example, for the primary key value 10001 in the first temporary table before modification, the server first obtains its primary key value after the first modification, which is 10002. Then, it determines whether the primary key value 10002 has been modified again within a preset time period. Here, 10002 has been modified again, and the last modified value is 10005. Therefore, 10005 is determined as the last modified value corresponding to 10001. Similarly, the last modified values ​​of primary key values ​​10002 and 10003 can also be obtained as 10005. For the primary key value 30001, the server first obtains its primary key value after the first modification, which is 30002. Within the preset time period, 30002 has not been modified again. Therefore, 30002 is determined as the last modified value corresponding to 30001.

[0107] Using the above method, the last modified value corresponding to each primary key value before modification can be accurately obtained.

[0108] Further, based on the second embodiment of the method for processing synchronization data, the fourth embodiment of the method for processing synchronization data is provided.

[0109] In the embodiment, the step of determining the redundant modification records in the table operation log according to the second temporary table can further include: querying the modification records in the table operation log with the primary key value before modification in the second temporary table as an index to obtain the modification records to be merged; replacing the primary key value after modification in the modification records to be merged with the last modification value in the second temporary table; for the multiple modification records to be merged with the same last modification value, obtaining the modification time of each modification record, determining the modification record closest to the current time as the reserved modification record, and determining the modification records other than the reserved modification record in the multiple modification records to be merged as the redundant modification records in the table operation log.

[0110] Specifically, the server first queries the modification records in which the primary key value of the relational database table is modified within a preset time length each time with the primary key value before modification in the second temporary table as an index, and determines the modification records containing the primary key value before modification as the modification records to be merged, so as to Figure 3 For example, the modification records containing 10001, 10002, 10003, 10004 and 30001 can be determined as the modification records to be merged.

[0111] Then, for each modification record to be merged, the server replaces the primary key value after modification with the last modification value in the second temporary table, such as replacing the primary key value after modification 10002 in the original record of the primary key value 10001 with the last modification value 10005, and replacing the primary key value after modification 10003 in the original record of the primary key value 10002 with the last modification value 10005.

[0112] For the multiple modification records to be merged with the same last modification value, the server obtains the modification time of each modification record, i.e. the time when the modification operation is generated, and then determines the modification record closest to the current time as the reserved modification record, and determines the modification records other than the reserved modification record as the redundant modification records.

[0113] After that, the server deletes the redundant modification records, and deletes the incremental data corresponding to the redundant modification records from the received incremental data, such as retaining only the modification record closest to the current time for the three modification records with the same last modification value 10005, and determining the other two modification records as the redundant modification records to be deleted.

[0114] By the above manner, the redundant modification record in the table operation log is accurately determined, the redundant modification record is deleted from the table operation log, and the incremental data corresponding to the redundant modification record is deleted from the received incremental data, so that the data consistency of the two synchronization parties is guaranteed, the data quality is improved, and the data backtracking and checking of downstream processing data are more convenient.

[0115] The embodiment of the present application also provides a processing device for synchronizing data.

[0116] With reference to Figure 4 , Figure 4 The figure is a module schematic diagram of one embodiment of the processing device for synchronizing data. In the embodiment, the processing device for synchronizing data comprises:

[0117] The receiving module 401 is configured to receive incremental data synchronized by a relational database in an incremental synchronization manner, wherein the incremental data is data saved by the relational database within a preset time length;

[0118] The first obtaining module 402 is configured to obtain a table operation log of the relational database, and read a modification record of a primary key value of a relational database table each time being modified within the preset time length from the table operation log;

[0119] The second obtaining module 403 is configured to obtain a primary key value before modification from the modification record, and obtain a last modification value corresponding to each primary key value before modification, wherein the last modification value represents a modification value after each primary key value before modification is lastly modified within the preset time length;

[0120] The determining module 404 is configured to determine a redundant modification record in the table operation log according to the last modification value corresponding to each primary key value before modification and the primary key value before modification;

[0121] The deleting module 405 is configured to delete the redundant modification record, and delete incremental data corresponding to the redundant modification record from the received incremental data.

[0122] Optionally, the second obtaining module 403 is further configured to:

[0123] obtain a record of two fields of the primary key value before modification and the primary key value after modification from the modification record;

[0124] save the obtained record of two fields of the primary key value before modification and the primary key value after modification to a preset first temporary table;

[0125] In the first temporary table, query a last modification value corresponding to each primary key value before modification, wherein the last modification value represents a modification value after each primary key value before modification is lastly modified within the preset time length;

[0126] The determining module 404 is further configured to:

[0127] generate a second temporary table according to the last modification value corresponding to each modification-before primary key value queried;

[0128] determine the redundant modification record in the table operation log according to the second temporary table.

[0129] Optionally, the second obtaining module 403 is further configured to:

[0130] filter the modification record according to the filtering condition that the modification-before primary key value and the modification-after primary key value are both not empty, to obtain the filtered modification record;

[0131] obtain the record of the modification-before primary key value and the modification-after primary key value from the filtered modification record.

[0132] Optionally, the second obtaining module 403 is further configured to:

[0133] for any modification-before primary key value in the first temporary table, obtain the primary key value after the modification-before primary key value is modified for the first time;

[0134] determine whether the primary key value after the modification-before primary key value is modified for the first time is continuously modified within the preset time length;

[0135] when the primary key value after the modification-before primary key value is modified for the first time is continuously modified, obtain the modification value after the primary key value after the modification-before primary key value is modified for the first time is modified for the last time, and determine the obtained modification value as the last modification value corresponding to the any modification-before primary key value;

[0136] when the primary key value after the modification-before primary key value is modified for the first time is not continuously modified, determine the primary key value after the modification-before primary key value is modified for the first time as the last modification value corresponding to the any modification-before primary key value.

[0137] Optionally, the determining module 404 is further configured to:

[0138] replace the modification-after primary key value in the first temporary table with the last modification value corresponding to each modification-before primary key value queried, to obtain a second temporary table.

[0139] Optionally, in the fifth implementation manner of the second aspect of the present application, the determining module 404 is further configured to:

[0140] query the modification record in the table operation log with the modification-before primary key value in the second temporary table as an index, to obtain a modification record to be merged;

[0141] replace the modification-after primary key value in the modification record to be merged with the last modification value in the second temporary table.

[0142] For the plurality of modification records to be merged with the same last modification value, the modification time of each modification record is acquired, the modification record with the closest modification time to the current time is determined as a reserved modification record, and the modification records in the plurality of modification records to be merged, except for the reserved modification record, are determined as redundant modification records in the table operation log.

[0143] Optionally, the first acquisition module 402 is further configured to:

[0144] read the table operation log written by the pre-configured data incremental synchronization tool to the big data platform, and take the read table operation log as the table operation log of the relational database, wherein the data incremental synchronization tool is used to synchronize the data increment in the relational database to the big database in the big data platform.

[0145] read the modification record of the primary key value of the relational database table each time the primary key value is modified within the preset time length from the table operation log.

[0146] The functions and advantages of the modules in the above data synchronization processing apparatus correspond to the steps in the above data synchronization processing method, and thus will not be repeated here.

[0147] The above data synchronization processing apparatus in the embodiment of the application is described in detail from the perspective of modular functional entities, and the data synchronization processing device in the embodiment of the application is described in detail from the perspective of hardware processing.

[0148] Referring to Figure 5 , Figure 5 A structural schematic diagram of the data synchronization processing device provided by the embodiment of the application is shown in FIG. 5. The data synchronization processing device 500 can have great differences due to different configurations or performances, and can include one or more processors (central processing units, CPUs) 510 (for example, one or more processors) and a memory 520, one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. The memory 520 and the storage media 530 can be temporary storage or persistent storage. The programs stored in the storage media 530 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the data synchronization processing device 500. Furthermore, the processor 510 can be configured to communicate with the storage media 530, execute the series of instruction operations in the storage media 530 on the data synchronization processing device 500, so as to implement the steps of the above data synchronization processing method.

[0149] The processing device for synchronous data 500 can also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that, Figure 5 The illustrated processing device for synchronous data structure does not constitute a limitation on the processing device for synchronous data, and can include more or fewer components than illustrated, or combine certain components, or arrange the components differently.

[0150] The present application also provides a storage medium, which can be a non-volatile storage medium or a volatile storage medium, and the storage medium stores instructions, which are executed by a processor to implement the steps of the processing method for synchronous data.

[0151] The method implemented by the instructions running on the processor and the advantages thereof can refer to the embodiments of the processing method for synchronous data, which will not be described here.

[0152] Those skilled in the art can understand that the integrated modules or units described above, if realized in the form of software function units and sold or used as independent products, can be stored in a storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0153] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of processing data in synchronism, characterized in that, The processing method of the synchronization data comprises the following steps: Receiving incremental data synchronized by a relational database in an incremental synchronization manner, wherein the incremental data is data saved by the relational database within a preset time length; Obtaining a table operation log of the relational database, reading modification records of a primary key value of a relational database table each time the primary key value is modified within the preset time length from the table operation log, wherein the modification records comprise an operation time, an operation type, a primary key value before modification and a primary key value after modification, wherein the primary key value refers to a combination of one column or multiple columns of the relational database table, and the value can uniquely identify each row in the table; Obtaining the primary key value before modification from the modification records, and obtaining a last modification value corresponding to each primary key value before modification, wherein the last modification value represents a modification value after each primary key value before modification is lastly modified within the preset time length; Determining redundant modification records in the table operation log according to the last modification value corresponding to each primary key value before modification and the primary key value before modification; Deleting the redundant modification records, and deleting incremental data corresponding to the redundant modification records from the received incremental data; The step of obtaining the primary key value before modification and the last modification value corresponding to each primary key value before modification from the modification records comprises the following steps:

2. The method of claim 1, wherein the synchronization data is processed by: Saving the obtained records of the two fields of the primary key value before modification and the primary key value after modification into a preset first temporary table; For any primary key value before modification in the first temporary table, obtaining a primary key value after the primary key value before modification is first modified; Determining whether the primary key value after the primary key value before modification is first modified is continuously modified within the preset time length; 3. The method of claim 2, wherein the step of synchronizing the data comprises the step of: When the primary key value after the primary key value before modification is first modified is continuously modified, obtaining a modification value after the primary key value after the primary key value before modification is first modified is lastly modified, and determining the obtained modification value as the last modification value corresponding to the any primary key value before modification; When the primary key value after the primary key value before modification is first modified is not continuously modified, determining the primary key value after the primary key value before modification is first modified as the last modification value corresponding to the any primary key value before modification. The step of determining the redundant modification records in the table operation log according to the last modification value corresponding to each primary key value before modification and the primary key value before modification comprises the following steps: Generating a second temporary table according to the last modification value corresponding to each primary key value before modification; Determining the redundant modification records in the table operation log according to the second temporary table. The step of obtaining the records of the two fields of the primary key value before modification and the primary key value after modification from the modification records comprises the following steps: Screening the modification records according to a screening condition that the primary key value before modification and the primary key value after modification are not empty, to obtain screened modification records; Obtaining the records of the two fields of the primary key value before modification and the primary key value after modification from the screened modification records.

4. The method of claim 2, wherein the synchronization data is processed by: The step of generating the second temporary table according to the last modification value corresponding to each pre-modification primary key value queried comprises: The pre-modification primary key value in the first temporary table is replaced by the last modification value corresponding to each pre-modification primary key value queried to obtain the second temporary table.

5. The method of claim 2, wherein the step of synchronizing the data comprises the step of: The step of determining the redundant modification record in the table operation log according to the second temporary table comprises: ​ The modification record to be merged is obtained by querying the modification record in the table operation log with the pre-modification primary key value in the second temporary table as an index; The post-modification primary key value in the modification record to be merged is replaced by the last modification value in the second temporary table; For a plurality of modification records to be merged with the same last modification value, the modification time of each modification record is obtained, the modification record closest to the current time is determined as a reserved modification record, and the modification records in the plurality of modification records to be merged except the reserved modification record are determined as the redundant modification record in the table operation log.

6. The method of processing synchronized data according to any one of claims 1 to 5, wherein, The step of obtaining the table operation log of the relational database and reading the modification record of the primary key value of the relational database table each time the primary key value is modified within the preset time length from the table operation log comprises: The table operation log written by a pre-configured data incremental synchronization tool to a big data platform is read, and the read table operation log is taken as the table operation log of the relational database, wherein the data incremental synchronization tool is used to synchronize the data increment in the relational database to a large database in the big data platform; The modification record of the primary key value of the relational database table each time the primary key value is modified within the preset time length is read from the table operation log.

7. A processing apparatus for synchronizing data, characterized by, The processing device for synchronizing data comprises: A receiving module is configured to receive incremental data synchronized from a relational database in an incremental synchronization manner, wherein the incremental data is data saved by the relational database within a preset time length; A first obtaining module is configured to obtain a table operation log of the relational database and read a modification record of a primary key value of a relational database table each time the primary key value is modified within the preset time length from the table operation log, wherein the modification record comprises an operation time, an operation type, a pre-modification primary key value and a post-modification primary key value, and the primary key value refers to a combination of one column or multiple columns of the relational database table, and the value can uniquely identify each row in the table; A second obtaining module is configured to obtain the pre-modification primary key value from the modification record and obtain a last modification value corresponding to each pre-modification primary key value, wherein the last modification value represents a modification value after the last modification of each pre-modification primary key value within the preset time length; A determining module is configured to determine a redundant modification record in the table operation log according to the last modification value corresponding to each pre-modification primary key value and the pre-modification primary key value; A deleting module is configured to delete the redundant modification record and delete incremental data corresponding to the redundant modification record from the received incremental data. The second obtaining module is specifically configured to: obtain records of two fields of the primary key value before modification and the primary key value after modification from the modification record; save the obtained records of the two fields of the primary key value before modification and the primary key value after modification into a preset first temporary table; for any primary key value before modification in the first temporary table, obtain a primary key value after the any primary key value before modification is modified for the first time; determine whether the primary key value after the any primary key value before modification is modified for the first time is continuously modified within the preset time length; when the primary key value after the any primary key value before modification is modified for the first time is continuously modified, obtain a modification value after the primary key value after the any primary key value before modification is modified for the first time is modified for the last time, and determine the obtained modification value as a last modification value corresponding to the any primary key value before modification; when the primary key value after the any primary key value before modification is modified for the first time is not continuously modified, determine the primary key value after the any primary key value before modification is modified for the first time as the last modification value corresponding to the any primary key value before modification.

8. A processing device for synchronizing data, characterized by The processing device of the synchronization data comprises a memory and at least one processor, and the memory stores instructions; The at least one processor invokes the instructions in the memory, so that the processing device of the synchronization data executes the processing method of the synchronization data according to any one of claims 1-6.

9. A storage medium having stored thereon instructions, the instructions comprising: The instructions are executed by the processor to implement the processing method of the synchronization data according to any one of claims 1-6.

Images