A data processing method and device

Abstract

The present invention discloses a data processing method and device, and relates to the field of computer technology. A specific implementation of the method includes: parsing a log file in a source library to obtain first incremental data, obtaining first incremental information from a message header of the first incremental data; determining second incremental data in a target library according to a primary key identifier and a table name in the first incremental information; comparing whether the first incremental information is greater than the second incremental information of the second incremental data, and then replacing the second incremental data in the target library with the first incremental data or not according to the comparison result. This implementation method filters historical data through incremental information, ensures the sequential execution of data synchronization under high throughput, reduces unnecessary data update operations, and avoids the situation where historical data overwrites new data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] Currently, when applied to master-slave replication of databases or incremental recovery of data, sequential or out-of-order message methods are usually used to process Binlog parsing messages. Binlog parsing messages support multiple routing methods: single topic single partition, single topic multiple partitions, multiple topics single partition, and multiple topics multiple partitions. Since the performance of each partition and the number of writes per second are limited, the performance of multiple partitions is better than that of a single partition.

[0003] In the process of implementing the present invention, the inventors found that the prior art has at least the following problems:

[0004] 1. The sequential message method has strict requirements on the processing order, which may cause the problem of delayed consumption of Binlog parsing messages;

[0005] 2. Although the processing speed of the out-of-order message method is better than that of the sequential message method, when data operations are frequent, it is easy to cause historical data to overwrite new data, resulting in data synchronization errors. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a data processing method and device, which can at least solve the problem of slow Binlog parsing message consumption or data synchronization errors in the prior art.

[0007] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0008] Parse the log file in the source library to obtain first incremental data, and obtain first incremental information from a message header of the first incremental data;

[0009] Determine the second incremental data in the target database according to the primary key identifier and the table name in the first incremental information; wherein the primary key identifier indicates the position of the first incremental information in the log file;

[0010] Compare the first incremental information to see whether it is greater than the second incremental information of the second incremental data, and then replace the second incremental data in the target library with the first incremental data or not according to the comparison result.

[0011] Optionally, the first incremental information includes a first execution time in the source library;

[0012] The comparing whether the first incremental information is greater than the second incremental information of the second incremental data, and then replacing the second incremental data in the target library with the first incremental data or not replacing it according to the comparison result, includes:

[0013] Comparing whether the first execution time is greater than the second execution time of the second incremental data;

[0014] If it is greater, determining that the first incremental data is the latest data, and replacing the second incremental data in the target database with the first incremental data;

[0015] If it is less, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0016] Optionally, the first incremental information further includes a first file sequence number, and the first file sequence number is obtained by splitting the file name of the log file;

[0017] The comparing whether the first incremental information is greater than the second incremental information of the second incremental data, and then replacing the second incremental data in the target library with the first incremental data or not replacing it according to the comparison result, includes:

[0018] When the first execution time and the second execution time are the same, comparing whether the first file sequence number is greater than the second file sequence number of the second incremental data;

[0019] If it is greater, determining that the first incremental data is the latest data, and replacing the second incremental data in the target database with the first incremental data;

[0020] If it is less, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0021] Optionally, the incremental information further includes a first file offset, where the first file offset indicates a storage order of the first incremental data in the log file;

[0022] The comparing whether the first incremental information is greater than the second incremental information of the second incremental data, and then replacing the second incremental data in the target library with the first incremental data or not replacing it according to the comparison result, includes:

[0023] When the first execution time and the second execution time are the same, and the first file sequence number and the second file sequence number are the same, comparing whether the first file offset is greater than the second file offset of the second incremental data;

[0024] If it is greater, determining that the first incremental data is the latest data, and replacing the second incremental data in the target database with the first incremental data;

[0025] If it is less, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0026] Optionally, the method further includes: setting a key name of a distributed lock to the primary key identifier and the table name, so as to use the distributed lock to perform locking processing on the first incremental data; and

[0027] After determining whether the first incremental data is the latest data or not, releasing the distributed lock.

[0028] To achieve the above object, according to another aspect of an embodiment of the present invention, a data processing device is provided, comprising:

[0029] A parsing module, configured to parse the log file in the source library to obtain first incremental data, and obtain first incremental information from a message header of the first incremental data;

[0030] A determination module, configured to determine the second incremental data in the target database according to the primary key identifier and the table name in the first incremental information; wherein the primary key identifier indicates the position of the first incremental information in the log file;

[0031] A comparison module is used to compare whether the first incremental information is greater than the second incremental information of the second incremental data, and then replace the second incremental data in the target library with the first incremental data or not replace it according to the comparison result.

[0032] Optionally, the first incremental information includes a first execution time in the source library;

[0033] The comparison module is used for:

[0034] Comparing whether the first execution time is greater than the second execution time of the second incremental data;

[0035] If it is greater, determining that the first incremental data is the latest data, and replacing the second incremental data in the target database with the first incremental data;

[0036] If it is less, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0037] Optionally, the first incremental information further includes a first file sequence number, and the first file sequence number is obtained by splitting the file name of the log file;

[0038] The comparison module is used for:

[0039] When the first execution time and the second execution time are the same, comparing whether the first file sequence number is greater than the second file sequence number of the second incremental data;

[0040] If it is greater, determining that the first incremental data is the latest data, and replacing the second incremental data in the target database with the first incremental data;

[0041] If it is less, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0042] Optionally, the incremental information further includes a first file offset, where the first file offset indicates a storage order of the first incremental data in the log file;

[0043] The comparison module is used for:

[0044] When the first execution time and the second execution time are the same, and the first file sequence number and the second file sequence number are the same, comparing whether the first file offset is greater than the second file offset of the second incremental data;

[0045] If it is greater, determining that the first incremental data is the latest data, and replacing the second incremental data in the target database with the first incremental data;

[0046] If it is less, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0047] Optionally, a distributed lock module is also included for:

[0048] Setting the key name of the distributed lock to the primary key identifier and the table name, so as to use the distributed lock to perform locking processing on the first incremental data; and

[0049] After determining whether the first incremental data is the latest data or not, releasing the distributed lock.

[0050] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data processing electronic device is provided.

[0051] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned data processing methods.

[0052] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above data processing methods is implemented.

[0053] According to the solution provided by the present invention, an embodiment of the above invention has the following advantages or beneficial effects: filtering historical data by Binlog parsing the SQL source library execution time, Binlog file and corresponding file offset in the first incremental data message header, ensuring the sequential execution of data synchronization under high throughput, reducing unnecessary data update operations, and avoiding the situation where historical data overwrites new data.

[0054] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0056] Figure 1 is a main flow chart of a data processing method according to an embodiment of the present invention;

[0057] Figure 2 is a flowchart of an optional data processing method according to an embodiment of the present invention;

[0058] Figure 3 is a flowchart of another optional data processing method according to an embodiment of the present invention;

[0059] Figure 4 is a flowchart of another optional data processing method according to an embodiment of the present invention;

[0060] Figure 5 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0061] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0062] Figure 7 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0064] See also Figure 1, which shows a main flow chart of a data processing method provided by an embodiment of the present invention, comprising the following steps:

[0065] S101: parsing a log file in a source database to obtain first incremental data, and obtaining first incremental information from a message header of the first incremental data;

[0066] S102: Determine the second incremental data in the target database according to the primary key identifier and the table name in the first incremental information; wherein the primary key identifier indicates the position of the first incremental information in the log file;

[0067] S103: Compare the first incremental information to see whether it is greater than the second incremental information of the second incremental data, and then replace the second incremental data in the target library with the first incremental data or not according to the comparison result.

[0068] In the above implementation, for step S101, Binlog is a log file in binary format, which is used to record the SQL (Structured Query Language) statement information of the user's addition, deletion and modification of the database. Binlog is usually used for master-slave synchronization of databases and incremental recovery of data, so this solution can be regarded as a master-slave synchronization of incremental data.

[0069] A Binlog file includes Binlog records generated by multiple SQL executions. A Binlog message is a message sent through MQ (message queue) by parsing the Binlog records in the Binlog file. A Binlog message corresponds to a Binlog record generated by one SQL execution.

[0070] Parse the Binlog file to obtain multiple Binlog records. For a single Binlog record (i.e., the first incremental data), parse its message header to obtain the SQL execution time execTime in the MySQL source database, the file name log_name of the file where the Binlog record line is located, and the file offset fileOffset. Then, separate the file number fileNum from log_name, for example, separate 002067 from mysql-bin.002067.log.

[0071] For step S102, the primary key identifier is similar to the row number in Excel, corresponding to a row of data in Excel, and is used to indicate the position of the Binlog record in the Binlog file in the MySQL database. There may be multiple Binlog files in the MySQL database, and the primary key identifiers in the same Binlog file will not be repeated, but the primary key identifiers of different Binlog files may be repeated.

[0072] In response to this situation, the primary key identifier and table name in the first incremental information are used to determine the corresponding Binlog parsing message (i.e., the second incremental data) execution information in Redis (or other cache-type databases), and the second incremental information is obtained: the execution time of the SQL in the source database, the file number and file offset of the Binlog record line, in the format of execTime-fileNum-fileOffset.

[0073] It should be noted that the incremental data in Redis is stored in key-value format, where the key is the primary key identifier and the table name, such as com:xx:online:order:100000000, and the value is the Binlog record, including incremental information such as execTime, fileNum, and fileOffset.

[0074] For step S103, if there is no such record in Redis, that is, the second incremental data is empty, it proves that the first incremental data has not been processed and is directly stored to complete the business logic processing.

[0075] However, if it exists, you need to compare execTime, fileNum, and fileOffset one by one to determine whether the second incremental data in Redis needs to be replaced with the first incremental data or not executed. For specific operations, see the subsequent Figure 2 to Figure 4 The description shown is not repeated here.

[0076] In addition, to ensure that the synchronization of incremental data is not interfered with by other data synchronization processes, after parsing the first incremental data, it is locked using a distributed lock, and the lock is released immediately after the judgment process is completed to execute normal business logic. When locking, the lock key will be marked as the primary key identifier + table name, and this information will be deleted directly after the lock is released.

[0077] The method provided in the above embodiment filters historical data by parsing execTime, fileNum and fileOffset in the first incremental data message header through Binlog, thereby ensuring sequential execution of data synchronization under high throughput, reducing unnecessary data update operations, and avoiding the situation where historical data overwrites new data.

[0078] See also Figure 2 , shows a schematic flow chart of an optional data processing method according to an embodiment of the present invention, comprising the following steps:

[0079] S201: Parse the log file in the source database to obtain first incremental data, and obtain first incremental information from the message header of the first incremental data; wherein the first incremental information includes a primary key identifier, a table name, and a first execution time in the source database;

[0080] S202: Determine the second incremental data in the target database according to the primary key identifier and the table name in the first incremental information; wherein the primary key identifier indicates the position of the first incremental information in the log file;

[0081] S203: Compare whether the first execution time is greater than the second execution time of the second incremental data;

[0082] S204: If it is greater than, determining that the first incremental data is the latest data, and replacing the second incremental data in the target library with the first incremental data;

[0083] S205: If it is less than, it is determined that the first incremental data is not the latest data, and the replacement operation is not performed.

[0084] In the above embodiment, for steps S201 and S202, see Figure 1 The description of steps S101 and S102 is not repeated here.

[0085] In the above implementation, for steps S203 to S205, if the second incremental data corresponding to the primary key identifier and table name of the first incremental data is recorded in Redis, the SQL execution time execTime is first compared:

[0086] 1) If the second execTime is greater than the first execTime, it means that the first incremental data is not the latest and is skipped without processing;

[0087] 2) The second execTime is less than the first execTime, which indicates that the first incremental data is the latest and the business logic can be processed normally, that is, the second incremental data in Redis is replaced by the first incremental data.

[0088] It should be noted that for the same primary key identifier, Redis only stores the key information of the latest record in the Binlog parsed message whose primary key identifier has been processed.

[0089] The following is a specific example of this implementation method:

[0090] For the same data in the MySQL source database, one SQL statement (abbreviated as A) is executed at 6:00 and one SQL statement (abbreviated as B) is executed at 6:01, resulting in a total of 2 Binlog messages. The execution time of A is 6:00, the file number is 101, and the file offset is 100000. The execution time of B is 6:01, the file number is 101 (or 102, no limit here), and the file offset is 200000.

[0091] This solution uses a disordered message method, so the order of A and B received by Redis is uncertain. Assuming that B arrives at Redis first, it means that B is currently stored in Redis. After parsing the Binlog file to obtain A, since the execution time of the latest record of B in Redis, 6:01, is greater than 6:00, it means that A is not the latest data and will not be processed. On the contrary, if A arrives at Redis first, A in Redis needs to be replaced with B.

[0092] The method provided in the above embodiment is applicable to the situation where the incremental data has different execution times. Only by comparing the execution time of the first incremental data and the second incremental data, it is possible to determine whether the parsed first incremental data is the latest, thereby achieving the purpose of data synchronization and avoiding data synchronization errors.

[0093] See also Figure 3 , shows a schematic flow chart of another optional data processing method according to an embodiment of the present invention, comprising the following steps:

[0094] S301: Parse the log file in the source database to obtain first incremental data, and obtain first incremental information from the message header of the first incremental data; wherein the first incremental information includes a primary key identifier, a table name, a first execution time in the source database, and a first file sequence number; the first file sequence number is obtained by splitting the file name of the log file;

[0095] S302: Determine the second incremental data in the target database according to the primary key identifier and the table name in the first incremental information; wherein the primary key identifier indicates the position of the first incremental information in the log file;

[0096] S303: Compare whether the first execution time is greater than the second execution time of the second incremental data;

[0097] S304: If it is greater than, determining that the first incremental data is the latest data, and replacing the second incremental data in the target library with the first incremental data;

[0098] S305: If it is less than, determining that the first incremental data is not the latest data, and not performing the replacement operation;

[0099] S306: If they are equal, comparing whether the first file sequence number is greater than the second file sequence number of the second incremental data;

[0100] S307: If it is greater than, determining that the first incremental data is the latest data, and replacing the second incremental data in the target library with the first incremental data;

[0101] S308: If it is less than, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0102] In the above embodiment, for steps S301 and S302, see Figure 1 For the description of steps S101 and S102, steps S303 to S305 can be found in Figure 2 The description of steps S203 to S205 is not repeated here.

[0103] In the above implementation, for steps S306 to S308, the Binlog file number increases continuously as the data grows, and the amount of data that a Binlog file can carry is limited. When the upper limit is reached, a new Binlog file will be created, and the file number in the new Binlog file is greater than the file number of the current Binlog file.

[0104] If the second execTime of the second incremental data in Redis is equal to the first execTime of the first incremental data, it proves that the two incremental data SQL statements are executed almost simultaneously (in seconds) in the source database. At this time, compare the file numbers fileNum of the two:

[0105] 1) If the second fileNum is greater than the first fileNum, it proves that the first incremental data is historical data and the processing is skipped;

[0106] 2) If the second fileNum is smaller than the first fileNum, it proves that the first incremental data is the latest. The second incremental data in Redis is replaced with the first incremental data to complete the normal business logic.

[0107] The following is a specific example of this implementation method:

[0108] For the same data in the MySQL source database, if one SQL statement (A for short) is executed at 6:00 and another SQL statement (B for short) is executed at 6:00, A is executed first and two Binlog messages will be received.

[0109] In the same Binlog file, the file offset increases with the recording time. Assume that after A is executed, it is recorded in Binlog file mysql-bin.101. After the recording is completed, the file storage limit is reached, so A is at the end of the file and its file offset is the largest in the file. Then a new Binlog file mysql-bin.102 is created and B is recorded in it. Therefore, B is at the front of the file sort and has a smaller file offset.

[0110] Assume that A and B are generated, the execution time of A is 6:00, the file number is 101, and the file offset is 100000; the execution time of B is 6:00, the file number is 102, and the file offset is 1000. Compared with A, the file number is smaller than B, but the file offset is larger than B.

[0111] Binlog messages are also processed in a disordered manner. If A arrives at Redis first, A is recorded in Redis. After B arrives, since the file number 102 (B)>101 (A), it means that B is the latest and A needs to be replaced by B. However, if B arrives before A, A is discarded and not processed.

[0112] The method provided in the above embodiment is mainly applied to the situation of the same execution time in seconds but different file numbers. By comparing the file numbers, it is determined whether the first incremental data currently parsed is the latest, thereby avoiding data synchronization errors and achieving the purpose of data synchronization.

[0113] See also Figure 4 , shows a schematic flow chart of another optional data processing method according to an embodiment of the present invention, comprising the following steps:

[0114] S401: Parse the log file in the source database to obtain first incremental data, and obtain first incremental information from the message header of the first incremental data; wherein the first incremental information includes a primary key identifier, a table name, a first execution time in the source database, a first file sequence number, and a first file offset; the first file sequence number is obtained by splitting the file name of the log file, and the first file offset indicates the storage order of the first incremental data in the log file;

[0115] S402: Determine the second incremental data in the target database according to the primary key identifier and the table name in the first incremental information; wherein the primary key identifier indicates the position of the first incremental information in the log file;

[0116] S403: Compare whether the first execution time is greater than the second execution time of the second incremental data;

[0117] S404: If it is greater than, determining that the first incremental data is the latest data, and replacing the second incremental data in the target library with the first incremental data;

[0118] S405: If it is less than, determining that the first incremental data is not the latest data, and not performing the replacement operation;

[0119] S406: If they are equal, comparing whether the first file sequence number is greater than the second file sequence number of the second incremental data;

[0120] S407: If it is greater than, determining that the first incremental data is the latest data, and replacing the second incremental data in the target library with the first incremental data;

[0121] S408: If it is less than, determining that the first incremental data is not the latest data, and not performing the replacement operation;

[0122] S409: If they are equal, comparing whether the first file offset is greater than the second file offset of the second incremental data;

[0123] S410: If it is greater than, determining that the first incremental data is the latest data, and replacing the second incremental data in the target library with the first incremental data;

[0124] S411: If it is less than, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0125] In the above embodiment, for steps S401 and S402, see Figure 1 For the description of steps S101 and S102, steps S403 to S405 can be found in Figure 2 For the description of steps S203 to S205, see Figure 3 The description of steps S306 to S308 is not repeated here.

[0126] In the above implementation, for steps S409 to S411, when the second fileNum of the second incremental data in Redis is equal to the first fileNum of the first incremental data in the MySQL source database, it indicates that the two SQLs are executed at a similar time.

[0127] In the same Binlog file, the file offset will increase as data records are inserted, so the file offset of the new record data in the same file must be greater than the historical file offset. Therefore, when the second execTime is equal to the first execTime, and the second fileNum is equal to the first fileNum, compare the file offsets fileOffset of the two:

[0128] 1) The first fileOffset is greater than the second fileOffset, indicating that the first incremental data is the latest, and the business logic can be processed normally, and the second incremental data is replaced by the first incremental data;

[0129] 2) The first fileOffset is smaller than the second fileOffset, indicating that the first incremental data is not the latest and can be skipped.

[0130] The following is a specific example of this implementation method:

[0131] For the same data in the MySQL source database, if one SQL statement (A for short) and one SQL statement (B for short) are executed at 6:00, A is executed first and two Binlog messages are received: the execution time of A is 6:00, the file number is 101, and the file offset is 100000; the execution time of B is 6:00, the file number is 101, and the file offset is 100100.

[0132] If A arrives at Redis first, A will be recorded in Redis. If B arrives later and it is judged that 100100(B)>100000(A), A in Redis will be replaced by B. However, if B arrives after A, A will be discarded and not processed.

[0133] The method provided in the above embodiment is mainly applied to the situation of the same second-level execution time and the same file sequence number but different file offsets. By comparing the file offsets, the latest incremental data in the same Binlog file can be determined, thereby achieving the purpose of data synchronization.

[0134] The method provided in the embodiment of the present invention uses Binlog out-of-order messages and ensures message consumption performance, and compares the SQL execution time, file sequence number, and file offset of the parsed first incremental data with the second incremental data in Redis to determine whether the first incremental data is the latest. Compared with the prior art, the method has at least the following beneficial effects:

[0135] 1) Reduce unnecessary Binlog parsing message processing and improve Binlog parsing message consumption rate; for example, if the same data is frequently changed from A->B->C in seconds, Redis receives A->C->B. This solution determines that B can be discarded and not processed, thereby reducing one data processing and ensuring that Redis stores the latest data;

[0136] 2) When the amount of data to be processed is large, it reduces the waste of corresponding server resources and reduces the use of server CPU, memory and other resources that are not needed to process messages;

[0137] 3) Since Binlog parsed messages are all full data rows, when a single data row changes frequently within seconds, it can achieve the same consumption purpose as sequential messages and reduce the situation where historical data overwrites new data under high data concurrency operations.

[0138] See also Figure 5 , which shows a schematic diagram of main modules of a data processing device 500 provided in an embodiment of the present invention, including:

[0139] A parsing module 501 is configured to parse a log file in a source repository to obtain first incremental data, and obtain first incremental information from a message header of the first incremental data;

[0140] Determination module 502, used to determine the second incremental data in the target database according to the primary key identifier and the table name in the first incremental information; wherein the primary key identifier indicates the position of the first incremental information in the log file;

[0141] The comparison module 503 is used to compare whether the first incremental information is greater than the second incremental information of the second incremental data, and then replace the second incremental data in the target library with the first incremental data or not according to the comparison result.

[0142] In the implementation device of the present invention, the first incremental information includes a first execution time in the source library;

[0143] The comparison module 503 is used to:

[0144] Comparing whether the first execution time is greater than the second execution time of the second incremental data;

[0145] If it is greater, determining that the first incremental data is the latest data, and replacing the second incremental data in the target database with the first incremental data;

[0146] If it is less, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0147] In the implementation device of the present invention, the first incremental information also includes a first file serial number, and the first file serial number is obtained by splitting the file name of the log file;

[0148] The comparison module 503 is used to:

[0149] When the first execution time and the second execution time are the same, comparing whether the first file sequence number is greater than the second file sequence number of the second incremental data;

[0150] If it is greater, determining that the first incremental data is the latest data, and replacing the second incremental data in the target database with the first incremental data;

[0151] If it is less, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0152] In the implementation device of the present invention, the incremental information also includes a first file offset, and the first file offset represents the storage order of the first incremental data in the log file;

[0153] The comparison module 503 is used to:

[0154] When the first execution time and the second execution time are the same, and the first file sequence number and the second file sequence number are the same, comparing whether the first file offset is greater than the second file offset of the second incremental data;

[0155] If it is greater, determining that the first incremental data is the latest data, and replacing the second incremental data in the target database with the first incremental data;

[0156] If it is less, it is determined that the first incremental data is not the latest data and the replacement operation is not performed.

[0157] The implementation device of the present invention further includes a distributed lock module 504 (not shown in the figure), which is used to:

[0158] Setting the key name of the distributed lock to the primary key identifier and the table name, so as to use the distributed lock to perform locking processing on the first incremental data; and

[0159] After determining whether the first incremental data is the latest data or not, releasing the distributed lock.

[0160] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0161] Figure 6 An exemplary system architecture 600 is shown in which embodiments of the present invention may be applied.

[0162] like Figure 6 As shown, the system architecture 600 may include terminal devices 601, 602, 603, a network 604 and a server 605 (just an example). The network 604 is used to provide a medium for communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0163] The user can use the terminal devices 601, 602, 603 to interact with the server 605 through the network 604 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 601, 602, 603.

[0164] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0165] The server 605 may be a server that provides various services, such as a background management server that provides support for shopping websites browsed by users using the terminal devices 601 , 602 , and 603 (only as an example).

[0166] It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the device is generally set in the server 605 .

[0167] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0168] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0169] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0170] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0171] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0172] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0173] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a parsing module, a determination module, and a comparison module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the comparison module may also be described as an "incremental information comparison module".

[0175] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0176] Parse the log file in the source library to obtain first incremental data, and obtain first incremental information from a message header of the first incremental data;

[0177] Determine the second incremental data in the target database according to the primary key identifier and the table name in the first incremental information; wherein the primary key identifier indicates the position of the first incremental information in the log file;

[0178] Compare the first incremental information to see whether it is greater than the second incremental information of the second incremental data, and then replace the second incremental data in the target library with the first incremental data or not according to the comparison result.

[0179] According to the technical solution of an embodiment of the present invention, the SQL source library execution time, Binlog file and corresponding file offset in the first incremental data message header are parsed by Binlog to filter historical data, ensure the sequential execution of data synchronization under high throughput, reduce unnecessary data update operations, and avoid the situation where historical data overwrites new data.

[0180] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, using a random order mode, characterized in that: include: Parse the log file in the source library to obtain first incremental data, and obtain first incremental information from a message header of the first incremental data; Determine the second incremental data in the target database according to the primary key identifier and the table name in the first incremental information; wherein the primary key identifier indicates the position of the first incremental information in the log file; Compare the first incremental information to see if it is greater than the second incremental information of the second incremental data, and then determine whether to replace the second incremental data in the target library with the first incremental data based on the comparison result, including: if the first incremental information is greater than the second incremental information, determine that the first incremental data is the latest data, and replace the second incremental data in the target library with the first incremental data; if the first incremental information is less than the second incremental information, determine that the first incremental data is not the latest data, and do not perform the replacement operation.

2. The method according to claim 1, characterized in that The first incremental information includes a first execution time in the source library; The comparing whether the first incremental information is greater than the second incremental information of the second incremental data includes: Compare and check whether the first execution time is greater than the second execution time of the second incremental data.

3. The method according to claim 1, characterized in that The first incremental information includes a first execution time and a first file sequence number in the source library; The comparing whether the first incremental information is greater than the second incremental information of the second incremental data includes: When the first execution time and the second execution time of the second incremental data are the same, compare whether the first file serial number is greater than the second file serial number of the second incremental data.

4. The method according to claim 1, characterized in that: The incremental information includes a first execution time, a first file sequence number, and a first file offset in the source library; The comparing whether the first incremental information is greater than the second incremental information of the second incremental data includes: When the first execution time and the second execution time of the second incremental data are the same, and the first file serial number and the second file serial number of the second incremental data are the same, compare whether the first file offset is greater than the second file offset of the second incremental data.

5. The method according to any one of claims 1 to 4, characterized in that Also includes: Setting the key name of the distributed lock to the primary key identifier and the table name, so as to use the distributed lock to perform locking processing on the first incremental data; as well as After determining whether the first incremental data is the latest data or not, releasing the distributed lock.

6. A data processing device, using a random order mode, characterized in that: include: A parsing module, configured to parse the log file in the source library to obtain first incremental data, and obtain first incremental information from a message header of the first incremental data; A determination module, configured to determine the second incremental data in the target database according to the primary key identifier and the table name in the first incremental information; wherein the primary key identifier indicates the position of the first incremental information in the log file; A comparison module is used to compare whether the first incremental information is greater than the second incremental information of the second incremental data, and then determine whether to replace the second incremental data in the target library with the first incremental data based on the comparison result, including: if the first incremental information is greater than the second incremental information, it is determined that the first incremental data is the latest data, and the second incremental data in the target library is replaced with the first incremental data; if the first incremental information is less than the second incremental information, it is determined that the first incremental data is not the latest data, and the replacement operation is not performed.

7. The device according to claim 6, characterized in that The first incremental information includes a first execution time in the source library; The comparison module is used for: Compare and check whether the first execution time is greater than the second execution time of the second incremental data.

8. The device according to claim 7, characterized in that The first incremental information includes a first execution time and a first file sequence number in the source library; The comparison module is used for: When the first execution time and the second execution time of the second incremental data are the same, compare whether the first file serial number is greater than the second file serial number of the second incremental data.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images