Data synchronization method and apparatus, electronic device, and storage medium

Abstract

This application proposes a data synchronization method, apparatus, electronic device, and storage medium. The method includes: determining data to be processed in a source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period; determining duplicate rows of data in the at least one row of data, wherein the duplicate rows of data exist in the historical synchronization data of the source table; deleting the duplicate rows of data in the data to be processed to obtain the data to be synchronized; and synchronizing the data to be synchronized to a destination table, thereby solving the problem of data duplication during data synchronization and improving data synchronization efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data synchronization method, apparatus, electronic device and storage medium. Background Technology

[0002] In related technologies, when synchronizing data between a source table and a destination table, the time field in the source table is used as a query condition for incremental data. Incremental data is then retrieved from the source table based on this condition, and the synchronization of data in the destination table is achieved based on this incremental data. However, this approach may result in data duplication, leading to inconsistencies between the source and destination tables after synchronization, and consequently, poor data synchronization efficiency. Summary of the Invention

[0003] The purpose of this application is to at least partially solve one of the aforementioned technical problems.

[0004] To address this issue, this application proposes a data synchronization method, apparatus, electronic device, and storage medium. By determining the data to be processed in the source table, identifying duplicate rows in at least one row of data, deleting the duplicate rows in the data to be processed, obtaining the data to be synchronized, and synchronizing the data to be synchronized to the destination table, this solution resolves the problem of data duplication during data synchronization in existing technologies and improves data synchronization efficiency.

[0005] The first aspect of this application proposes a data synchronization method, including:

[0006] Determine the data to be processed in the source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period;

[0007] Identify duplicate rows in the at least one row of data, wherein the duplicate rows exist in the historical synchronized data of the source table;

[0008] Delete the duplicate rows of data in the data to be processed to obtain the data to be synchronized;

[0009] Synchronize the data to be synchronized to the destination table.

[0010] Optionally, determining the data to be processed in the source table includes:

[0011] Determine the maximum historical synchronization timestamp and the maximum timestamp of the source table;

[0012] The synchronization time period is determined by using the maximum historical synchronization timestamp as the start timestamp and the maximum timestamp as the end timestamp;

[0013] The row data whose timestamps are located within the synchronization time period in the source table are extracted to obtain the data to be processed.

[0014] Optionally, determining the maximum historical synchronization timestamp and the maximum timestamp of the source table includes:

[0015] Determine the maximum timestamp of the source table;

[0016] Obtain the stored maximum historical synchronization timestamp; or, if the maximum historical synchronization timestamp is not stored, determine the user-configured synchronization timestamp or the preset synchronization timestamp as the maximum historical synchronization timestamp.

[0017] Optionally, determining duplicate row data in the at least one row of data includes:

[0018] Determine candidate row data from the at least one row of data, wherein the timestamp of the candidate row data is consistent with the start timestamp of the synchronization time period;

[0019] Determine a reference row of data in the historical synchronization data, wherein the timestamp of the reference row of data is consistent with the start timestamp;

[0020] The candidate row data and the reference row data are compared to determine the duplicate row data in the candidate row data.

[0021] Optionally, the source table's row data includes a primary key field. The process of comparing the candidate row data and the reference row data to determine the duplicate row data within the candidate row data includes:

[0022] The primary key fields of the candidate row data and the primary key fields of the reference row data are compared to determine the primary key fields that are the same in the candidate row data and the reference row data;

[0023] The candidate row data corresponding to the same primary key field are identified as the duplicate row data.

[0024] Optionally, the source table does not have a primary key field in its row data. The step of comparing the candidate row data and the reference row data to determine the duplicate row data in the candidate row data includes:

[0025] Extract the summary information of the candidate row data and the summary information of the reference row data;

[0026] The summary information of the candidate row data and the summary information of the reference row data are compared to determine the same summary information and the number of the same summary information.

[0027] The number of candidate rows with the same summary information are identified as the duplicate rows.

[0028] Optionally, after synchronizing the data to be synchronized to the destination table, the method further includes:

[0029] The maximum historical synchronization timestamp stored can be updated using the termination timestamp within the synchronization time period; or, if no maximum historical synchronization timestamp is stored, the termination timestamp can be stored as the maximum historical synchronization timestamp.

[0030] The data synchronization method of this application embodiment determines the data to be processed in the source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period; determines duplicate row data in the at least one row of data, wherein the duplicate row data exists in the historical synchronization data of the source table; deletes the duplicate row data in the data to be processed to obtain the data to be synchronized; and synchronizes the data to be synchronized to the destination table, thereby solving the problem of data duplication during data synchronization and improving data synchronization efficiency.

[0031] A second aspect of this application provides a data synchronization device, comprising:

[0032] The first determining module is used to determine the data to be processed in the source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period;

[0033] The second determining module is used to determine duplicate row data in the at least one row of data, wherein the duplicate row data exists in the historical synchronized data of the source table;

[0034] The processing module is used to delete the duplicate rows of data in the data to be processed to obtain the data to be synchronized;

[0035] The synchronization module is used to synchronize the data to be synchronized to the destination table.

[0036] Optionally, the first determining module is specifically used for,

[0037] Determine the maximum historical synchronization timestamp and the maximum timestamp of the source table;

[0038] The synchronization time period is determined by using the maximum historical synchronization timestamp as the start timestamp and the maximum timestamp as the end timestamp;

[0039] The row data whose timestamps are located within the synchronization time period in the source table are extracted to obtain the data to be processed.

[0040] Optionally, the first determining module is specifically used for,

[0041] Determine the maximum timestamp of the source table;

[0042] Obtain the stored maximum historical synchronization timestamp; or, if the maximum historical synchronization timestamp is not stored, determine the user-configured synchronization timestamp or the preset synchronization timestamp as the maximum historical synchronization timestamp.

[0043] Optionally, the second determining module is specifically used for,

[0044] Determine candidate row data from the at least one row of data, wherein the timestamp of the candidate row data is consistent with the start timestamp of the synchronization time period;

[0045] Determine a reference row of data in the historical synchronization data, wherein the timestamp of the reference row of data is consistent with the start timestamp;

[0046] The candidate row data and the reference row data are compared to determine the duplicate row data in the candidate row data.

[0047] Optionally, the source table's row data includes a primary key field, and the second determining module is specifically used for...

[0048] The primary key fields of the candidate row data and the primary key fields of the reference row data are compared to determine the primary key fields that are the same in the candidate row data and the reference row data;

[0049] The candidate row data corresponding to the same primary key field are identified as the duplicate row data.

[0050] Optionally, the source table's row data does not have a primary key field, and the second determining module is specifically used for:

[0051] Extract the summary information of the candidate row data and the summary information of the reference row data;

[0052] The summary information of the candidate row data and the summary information of the reference row data are compared to determine the same summary information and the number of the same summary information.

[0053] The number of candidate rows with the same summary information are identified as the duplicate rows.

[0054] Optionally, the device further includes: an update module;

[0055] The update module is used to update the stored maximum historical synchronization timestamp using the termination timestamp in the synchronization time period; or, when no maximum historical synchronization timestamp is stored, the termination timestamp is stored as the maximum historical synchronization timestamp.

[0056] The data synchronization device of this application embodiment determines the data to be processed in the source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period; determines duplicate row data in the at least one row of data, wherein the duplicate row data exists in the historical synchronization data of the source table; deletes the duplicate row data in the data to be processed to obtain the data to be synchronized; and synchronizes the data to be synchronized to the destination table, thereby solving the problem of data duplication during data synchronization and improving data synchronization efficiency.

[0057] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data synchronization method as described in the first aspect.

[0058] The fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the data synchronization method as described in the first aspect.

[0059] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0060] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0061] Figure 1 A flowchart illustrating a data synchronization method provided in an embodiment of this application;

[0062] Figure 2 A flowchart illustrating another data synchronization method provided in an embodiment of this application;

[0063] Figure 3 This is a schematic diagram of the structure of a data synchronization device provided in an embodiment of this application;

[0064] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0066] The data synchronization method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings.

[0067] The following is combined Figure 1 This application provides a detailed description of the data synchronization method provided.

[0068] Figure 1 This is a flowchart illustrating a data synchronization method provided in an embodiment of this application.

[0069] The execution subject of this application embodiment is the data synchronization device provided in this application. The data synchronization device can be configured in an electronic device so that the electronic device can perform data synchronization function; or, the data synchronization device can be configured in an application of the electronic device so that the application can perform data synchronization function.

[0070] The electronic device can be any device with computing capabilities, and the device or the application within the device can communicate with multiple data applications to achieve data synchronization between the multiple data applications. The device with computing capabilities can be, for example, a personal computer (PC), a mobile terminal, or a server. The mobile terminal can be, for example, in-vehicle equipment, mobile phones, tablets, personal digital assistants, wearable devices, or other hardware devices with various operating systems, touchscreens, and / or displays.

[0071] Furthermore, electronic devices can also be clusters of multiple devices, such as cloud platforms. Cloud platforms, or specific cloud services within them, can perform data synchronization functions.

[0072] like Figure 1 As shown, this data synchronization method includes the following steps:

[0073] Step 11: Determine the data to be processed in the source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period.

[0074] In this embodiment, the source table is the table whose data is to be synchronized to other tables; the other tables are the destination tables. One source table can correspond to one or more destination tables; one destination table can also correspond to one or more source tables, which can be configured according to actual needs.

[0075] In this embodiment, for example, the source table and the destination table may belong to different data applications, and data synchronization from the source table to the destination table can achieve data synchronization between different data applications. As another example, the destination table can be a backup table of the source table within the same data application, used to back up the source table. Furthermore, the source table and the destination table may reside on different servers in a cluster, enabling data synchronization between different servers. The source table and destination table can be selected according to actual needs, and no limitation is made here.

[0076] In this embodiment, the source table and the destination table can be of the same or different types. Processing of the source and destination tables can be achieved using drivers corresponding to different table types. For a specific table, the corresponding driver can be selected based on the table type, and the driver, along with structured query language, can be used to perform table queries, data retrieval, and data updates.

[0077] In this embodiment, the source table can be, for example, a log file of a data application. The data synchronization device can parse the log file of the data application, obtain incremental data, and synchronize it to the destination table. If the log file switch is on, incremental data can be directly obtained from the log file and synchronized to the destination table; if the log file switch is off, it is difficult to directly obtain incremental data from the log file. The data synchronization method in this embodiment can be used only when the log file switch is off; or it can be used for both cases where the log file switch is on and off, and can be set according to actual needs.

[0078] In this embodiment, data in the source table can be periodically synchronized to the destination table, for example, every day, every 10 hours, every 3 hours, etc. Alternatively, data in the source table can also be synchronized to the destination table non-periodically. The start timestamp of the synchronization period can be the previous synchronization time point, and the end timestamp of the synchronization period can be the current time point or the maximum timestamp of the source table, etc. It should be noted that the source table in this application must be a table with timestamps. If the source table does not have timestamps, it is necessary to determine the timestamps of each row of data in the source table and extend the timestamp field in the source table.

[0079] In this embodiment of the application, the process of the data synchronization device performing step 11 may be as follows: determine the maximum historical synchronization timestamp and the maximum timestamp of the source table; determine the synchronization time period with the maximum historical synchronization timestamp as the start timestamp and the maximum timestamp as the end timestamp; extract the row data in the source table whose corresponding timestamp is within the synchronization time period to obtain the data to be processed.

[0080] The maximum historical synchronization timestamp can be determined as follows: when no maximum historical synchronization timestamp is stored (i.e., during the initial synchronization of data in the source table), the user-configured synchronization timestamp or a preset synchronization timestamp is used as the maximum historical synchronization timestamp; when a maximum historical synchronization timestamp is stored (i.e., during non-initial synchronization of data in the source table), the stored maximum historical synchronization timestamp is retrieved. The preset synchronization timestamp could be, for example, 1970-01-01 08:00:00. The maximum historical synchronization timestamp can be denoted as `lastRunMaxTime`, and the maximum timestamp can be denoted as `maxTime`.

[0081] In this embodiment, the method for extracting row data from the source table whose timestamps are within the synchronization time period to obtain the data to be processed can be, for example, by constructing query conditions based on the synchronization time period and extracting the data to be processed from the source table based on the query conditions. Taking the start timestamp of the synchronization time period as the maximum historical synchronization timestamp and the end timestamp as the maximum timestamp as an example, the constructed query conditions can specifically be: timestamp field value >= maximum historical synchronization timestamp lastRunMaxTime, and timestamp field value <= maximum timestamp maxTime, where the data to be processed extracted from the source table can be denoted as rows.

[0082] Furthermore, to ensure the accuracy of the stored maximum historical synchronization timestamp, it needs to be updated promptly. Correspondingly, after synchronizing the data to be synchronized to the destination table, if a maximum historical synchronization timestamp exists (i.e., when the data in the source table is not being synchronized for the first time), the maximum historical synchronization timestamp is updated using the end timestamp of the synchronization period; alternatively, if a maximum historical synchronization timestamp is not stored (i.e., when the data in the source table is being synchronized for the first time), the end timestamp is stored as the maximum historical synchronization timestamp. The maximum historical synchronization timestamp can be stored on disk or in the database for easy retrieval during subsequent data synchronizations.

[0083] Step 12: Identify at least one duplicate row in the data, wherein the duplicate row exists in the historical synchronized data of the source table.

[0084] In this embodiment, the start timestamp of the synchronization time period is the maximum historical synchronization timestamp, and the end timestamp is the maximum timestamp of the source table. At least one row of data within the synchronization time period has timestamps that satisfy the constructed query conditions, i.e., timestamp field value >= maximum historical synchronization timestamp lastRunMaxTime, and timestamp field value <= maximum timestamp maxTime. The start timestamp of this synchronization time period is the end timestamp of an adjacent historical synchronization time period; therefore, this synchronization time period intersects with adjacent historical synchronization time periods, i.e., the start timestamp of this synchronization time period is the start timestamp of the previous synchronization time period. This results in at least one row of data within an adjacent historical synchronization time period having duplicate rows of data with at least one row of data within this synchronization time period, and the timestamp of these duplicate rows is equal to the start timestamp of the previous synchronization time period.

[0085] As an example, if the maximum timestamp `maxTime` from the last data synchronization was, for example, 2021-05-31 00:00:00, and the maximum historical synchronization timestamp `lastRunMaxTime` was, for example, 2021-05-30 00:00:00, then the query condition could be "timeColumn >= '2021-05-30 00:00:00' and timeColumn <= '2021-05-31 00:00:00'". Here, `timeColumn` represents the timestamp. Assuming that two rows of data had a `timeColumn` of `maxTime` during the last data synchronization, these two rows would have been synchronized to the destination table during that synchronization. During this data synchronization, the maximum historical synchronization timestamp lastRunMaxTime becomes 2021-05-31 00:00:00. When extracting row data within the synchronization period, these two rows will also be extracted and synchronized to the destination table, causing data duplication. Therefore, in this step, it is necessary to identify the duplicate rows in the historical synchronization data of the source table in the data to be processed.

[0086] Step 13: Delete duplicate rows in the data to be processed to obtain the data to be synchronized.

[0087] In this embodiment of the application, the data synchronization device can store the data to be processed in a new table, determine the corresponding driver according to the type of the new table, and combine the driver with the structured query language to realize the deletion of some data in the table.

[0088] Step 14: Synchronize the data to be synchronized to the destination table.

[0089] In this embodiment, the data synchronization device can determine the corresponding driver based on the type of the target table, and then insert the data to be synchronized into the table in combination with the driver and the structured query language.

[0090] The data synchronization method provided in this application embodiment determines the data to be processed in the source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period; determines duplicate row data in the at least one row of data, wherein the duplicate row data exists in the historical synchronization data of the source table; deletes the duplicate row data in the data to be processed to obtain the data to be synchronized; and synchronizes the data to be synchronized to the destination table, thereby solving the problem of data duplication during data synchronization and improving data synchronization efficiency.

[0091] As one possible implementation of this application, in step 12 above, since the data in the source table may have been updated after the last data synchronization, especially the row data corresponding to the start timestamp of the synchronization period in the source table may have been updated, for example, other row data with that start timestamp may have been added. Therefore, it is necessary to combine the content of historically synchronized data to determine the duplicate row data in the data to be processed. Figure 2 The data synchronization method provided in this application will be further explained.

[0092] Figure 2 This is a flowchart illustrating another data synchronization method provided in an embodiment of this application. Figure 2 As shown, step 12 above may include the following steps:

[0093] Step 21: Determine candidate row data from at least one row of data, wherein the timestamp of the candidate row data is consistent with the start timestamp of the synchronization time period.

[0094] In this embodiment of the application, the method for determining candidate row data can be, for example, constructing a query condition for candidate row data, namely "timeColumn = lastRunMaxTime"; and then extracting candidate row data from at least one row of data based on this query condition. The number of candidate row data can be one or more, and the candidate row data can be denoted as handledMaxTimeRows.

[0095] Step 22: Determine the reference row data in the historical synchronization data, wherein the timestamp of the reference row data is consistent with the start timestamp.

[0096] In this embodiment of the application, the reference row data can be determined by, for example, constructing a query condition for the reference row data, namely "timeColumn = lastRunMaxTime"; and then extracting the reference row data from the historical synchronization data based on this query condition. The number of reference row data can be one or more.

[0097] It should be noted that after the last data synchronization, the data in the source table may have been updated, especially the rows corresponding to the start timestamp of the synchronization period. For example, other rows with that start timestamp may have been added. Therefore, the content of the reference row data may exist in the candidate row data, but the content of the candidate row data may not necessarily exist in the reference row data. The reference row data can be denoted as lastRunHandledRows.

[0098] Step 23: Compare the candidate row data and the reference row data to identify duplicate row data in the candidate row data.

[0099] In this embodiment of the application, as an example, the row data in the source table may have a primary key field. The primary key field can be represented by a single field or by a combination of multiple fields. The primary key field can uniquely identify the row data. That is, different rows of data may have the same timestamp, but they will definitely have different primary key fields.

[0100] Therefore, if the source table's row data contains a primary key field, the data synchronization device can perform step 23 as follows: compare the primary key fields of the candidate row data and the reference row data to determine the common primary key fields in the candidate and reference row data; and identify the candidate row data corresponding to the common primary key fields as duplicate row data. The row data corresponding to different primary key fields are the row data newly added to the source table after the last data synchronization, containing the start timestamp of the synchronization period.

[0101] For example, if the source table t1 has three fields c1, c2, and c3, and the source table t1 has a primary key field with c1 as the primary key field, then the value of c1 can uniquely identify the row data. Assuming that only one row data was synchronized during the last data synchronization: c1 = a, c2 = b, c3 = c; then when identifying duplicate row data during this data synchronization, it is only necessary to compare the row data in the candidate row data where the value of the c1 field is 'a', and directly select the row data with the value of the c1 field as the duplicate row data in the candidate row data.

[0102] As another example, the source table's row data does not have a primary key field. That is, none of the row data fields uniquely identify the row. For two rows, the value of any field could be the same. Therefore, it's necessary to determine if the two rows are identical based on all their contents. Since the row data may contain a lot of information, to reduce computation, we can perform message digest processing on all the row data to determine its MD5 value. We then compare the MD5 values ​​of the two rows to determine if they are identical. The MD5 value uniquely identifies the row. If the contents of two rows are different, their corresponding MD5 values ​​will definitely be different; only when all the contents of two rows are completely identical will their corresponding MD5 values ​​be the same. The message digest algorithm can be replaced with a hash algorithm or other hashing algorithm.

[0103] Furthermore, it's possible for two rows of data in the source table to have completely identical content. For example, consider a row whose timestamp is the start timestamp of the synchronization period. During the last data synchronization, suppose there were two rows in the source table with the same start timestamp and identical content. Before local data synchronization, a new row with the same start timestamp was added to the source table, and its content is completely identical to the two rows mentioned above. When determining duplicate rows, it's necessary to consider the number of rows with the same MD5 value in the reference rows and the number of rows with that MD5 value in the candidate rows. This avoids classifying all three candidate rows with the same start timestamp and identical content as duplicates, because in this case, there should only be two duplicate rows with that start timestamp and content.

[0104] Therefore, if the source table does not have a primary key field in its row data, the data synchronization device can perform step 23 as follows: extract the summary information of the candidate row data and the summary information of the reference row data; compare the summary information of the candidate row data and the summary information of the reference row data to determine the same summary information and the number of the same summary information; and identify the number of candidate row data with the same summary information as duplicate row data.

[0105] In other words, if the source table does not have a primary key field, the MD5 value of the entire row of data and the number of rows with that MD5 value are recorded. The number of rows with that MD5 value in the candidate row data is compared with the number of rows with that MD5 value in the reference row data. If they match, all rows with that MD5 value in the candidate row data are considered duplicate rows. If they do not match, the number of rows with that MD5 value in the reference row data is used as the first value, and the first value of rows with that MD5 value in the candidate row data are considered duplicate rows.

[0106] For example, if source table t2 has three fields c4, c5, and c6, and no primary key field is set in source table t2, then source table t2 can have completely identical data rows. Suppose that during the last data synchronization, two rows with identical content were synchronized: c4 = d, c5 = e, and c6 = f. Then these two rows have the same MD5 value, let's say both are 5a2fd2941d88de3710ecaf01193880ed, and the number of rows with this MD5 value is 2. During this data synchronization, if the number of candidate rows with this MD5 value is also 2, then these two rows are considered duplicate rows. If the number of candidate rows with this MD5 value is greater than 2, then two of the rows with this MD5 value will be considered duplicate rows.

[0107] The data synchronization method of this application embodiment determines candidate row data from at least one row of data, wherein the timestamp of the candidate row data is consistent with the start timestamp of the synchronization time period; determines reference row data from historical synchronization data, wherein the timestamp of the reference row data is consistent with the start timestamp; and compares the candidate row data and the reference row data to determine duplicate row data in the candidate row data. Therefore, it can be applied to source tables without primary key fields and source tables with primary key fields to determine duplicate row data in the source table for deletion, thereby avoiding data duplication problems during data synchronization and improving data synchronization efficiency.

[0108] Corresponding to the data synchronization methods provided in the above embodiments, one embodiment of this application also provides a data synchronization device. Since the data synchronization device provided in this embodiment corresponds to the data synchronization methods provided in the above embodiments, the implementation methods of the data synchronization methods are also applicable to the data synchronization device provided in this embodiment, and will not be described in detail here.

[0109] Figure 3 This is a schematic diagram of a data synchronization device provided in an embodiment of this application.

[0110] like Figure 3 As shown, the data synchronization device 30 may include: a first determining module 31, a second determining module 32, a processing module 33, and a synchronization module 34.

[0111] The first determining module 31 is used to determine the data to be processed in the source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period.

[0112] The second determining module 32 is used to determine duplicate row data in the at least one row of data, wherein the duplicate row data exists in the historical synchronization data of the source table.

[0113] Processing module 33 is used to delete duplicate rows of data in the data to be processed, so as to obtain the data to be synchronized.

[0114] Synchronization module 34 is used to synchronize the data to be synchronized to the destination table.

[0115] As one possible implementation of this application, the first determining module 31 is specifically used to determine the maximum historical synchronization timestamp and the maximum timestamp of the source table.

[0116] The synchronization time period is determined by using the maximum historical synchronization timestamp as the start timestamp and the maximum timestamp as the end timestamp;

[0117] The row data whose timestamps are located within the synchronization time period in the source table are extracted to obtain the data to be processed.

[0118] As another possible implementation of this application embodiment, the first determining module 31 is specifically used to determine the maximum timestamp of the source table;

[0119] Obtain the stored maximum historical synchronization timestamp; or, if the maximum historical synchronization timestamp is not stored, determine the user-configured synchronization timestamp or the preset synchronization timestamp as the maximum historical synchronization timestamp.

[0120] As another possible implementation of this application embodiment, the second determining module 32 is specifically used for,

[0121] Determine candidate row data from the at least one row of data, wherein the timestamp of the candidate row data is consistent with the start timestamp of the synchronization time period;

[0122] Determine a reference row of data in the historical synchronization data, wherein the timestamp of the reference row of data is consistent with the start timestamp;

[0123] The candidate row data and the reference row data are compared to determine the duplicate row data in the candidate row data.

[0124] As another possible implementation of this application embodiment, the source table's row data includes a primary key field, and the second determining module 32 is specifically used for,

[0125] The primary key fields of the candidate row data and the primary key fields of the reference row data are compared to determine the primary key fields that are the same in the candidate row data and the reference row data;

[0126] The candidate row data corresponding to the same primary key field are identified as the duplicate row data.

[0127] As another possible implementation of this application embodiment, the source table's row data does not have a primary key field, and the second determining module 32 is specifically used for,

[0128] Extract the summary information of the candidate row data and the summary information of the reference row data;

[0129] The summary information of the candidate row data and the summary information of the reference row data are compared to determine the same summary information and the number of the same summary information.

[0130] The number of candidate rows with the same summary information are identified as the duplicate rows.

[0131] As another possible implementation of this application embodiment, the data synchronization device 30 further includes: an update module;

[0132] The update module is used to update the stored maximum historical synchronization timestamp using the termination timestamp in the synchronization time period; or, when no maximum historical synchronization timestamp is stored, the termination timestamp is stored as the maximum historical synchronization timestamp.

[0133] The data synchronization device provided in this application embodiment determines the data to be processed in the source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period; determines duplicate row data in the at least one row of data, wherein the duplicate row data exists in the historical synchronization data of the source table; deletes the duplicate row data in the data to be processed to obtain the data to be synchronized; and synchronizes the data to be synchronized to the destination table, thereby solving the problem of possible data duplication and improving data synchronization efficiency.

[0134] To achieve the above embodiments, this application also proposes an electronic device. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0135] The memory 1001, the processor 1002, and the computer program stored on the memory 1001 and capable of running on the processor 1002.

[0136] When the processor 1002 executes the program, it implements the debugging method for the front-end development page provided in the above embodiments.

[0137] Furthermore, electronic devices also include:

[0138] Communication interface 1003 is used for communication between memory 1001 and processor 1002.

[0139] The memory 1001 is used to store computer programs that can run on the processor 1002.

[0140] The memory 1001 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0141] The processor 1002 is used to implement the debugging method for the front-end development page described in the above embodiments when executing the program.

[0142] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, then the communication interface 1003, memory 1001, and processor 1002 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0143] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.

[0144] The processor 1002 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0145] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data synchronization method provided in the above embodiments.

[0146] To implement the above embodiments, this application also proposes a computer program product that, when executed by an instruction processor, implements the data synchronization method provided in the above embodiments.

[0147] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0148] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0149] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0151] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0152] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0153] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0154] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A data synchronization method, characterized in that, include: Determine the data to be processed in the source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period; Determine candidate row data from the at least one row of data, wherein the timestamp of the candidate row data is consistent with the start timestamp of the synchronization time period; Determine a reference row of data in the historical synchronization data, wherein the timestamp of the reference row of data is consistent with the starting timestamp; The candidate row data and the reference row data are compared to determine the duplicate row data in the candidate row data, wherein the duplicate row data exists in the historical synchronized data of the source table; Delete the duplicate rows of data in the data to be processed to obtain the data to be synchronized; Synchronize the data to be synchronized to the destination table.

2. The data synchronization method according to claim 1, characterized in that, The process of determining the data to be processed in the source table includes: Determine the maximum historical synchronization timestamp and the maximum timestamp of the source table; The synchronization time period is determined by using the maximum historical synchronization timestamp as the start timestamp and the maximum timestamp as the end timestamp; The row data whose timestamps are located within the synchronization time period in the source table are extracted to obtain the data to be processed.

3. The data synchronization method according to claim 2, characterized in that, Determining the maximum historical synchronization timestamp and the maximum timestamp of the source table includes: Determine the maximum timestamp of the source table; Obtain the stored maximum historical synchronization timestamp; or, if the maximum historical synchronization timestamp is not stored, determine the user-configured synchronization timestamp or the preset synchronization timestamp as the maximum historical synchronization timestamp.

4. The data synchronization method according to claim 1, characterized in that, The source table has a primary key field in its row data. The process of comparing the candidate row data and the reference row data to determine the duplicate row data in the candidate row data includes: The primary key fields of the candidate row data and the primary key fields of the reference row data are compared to determine the primary key fields that are the same in the candidate row data and the reference row data; The candidate row data corresponding to the same primary key field are identified as the duplicate row data.

5. The data synchronization method according to claim 1, characterized in that, The source table does not have a primary key field in its row data. The process of comparing the candidate row data and the reference row data to determine the duplicate row data in the candidate row data includes: Extract the summary information of the candidate row data and the summary information of the reference row data; The summary information of the candidate row data and the summary information of the reference row data are compared to determine the same summary information and the number of the same summary information. The number of candidate rows with the same summary information are identified as the duplicate rows.

6. The data synchronization method according to claim 2, characterized in that, After synchronizing the data to be synchronized to the destination table, the method further includes: The maximum historical synchronization timestamp stored can be updated using the termination timestamp within the synchronization time period; or, if no maximum historical synchronization timestamp is stored, the termination timestamp can be stored as the maximum historical synchronization timestamp.

7. A data synchronization device, characterized in that, include: The first determining module is used to determine the data to be processed in the source table, wherein the data to be processed includes at least one row of data whose corresponding timestamp is located within the synchronization time period; The second determining module is used to determine candidate row data in the at least one row of data, wherein the timestamp of the candidate row data is consistent with the start timestamp of the synchronization time period; determine reference row data in the historical synchronization data, wherein the timestamp of the reference row data is consistent with the start timestamp; and perform a comparison process on the candidate row data and the reference row data to determine duplicate row data in the candidate row data, wherein the duplicate row data exists in the historical synchronization data of the source table; The processing module is used to delete the duplicate rows of data in the data to be processed to obtain the data to be synchronized; The synchronization module is used to synchronize the data to be synchronized to the destination table.

8. The data synchronization device according to claim 7, characterized in that, The first determining module is specifically used for, Determine the maximum historical synchronization timestamp and the maximum timestamp of the source table; The synchronization time period is determined by using the maximum historical synchronization timestamp as the start timestamp and the maximum timestamp as the end timestamp; The row data whose timestamps are located within the synchronization time period in the source table are extracted to obtain the data to be processed.

9. The data synchronization device according to claim 8, characterized in that, The first determining module is specifically used for, Determine the maximum timestamp of the source table; Obtain the stored maximum historical synchronization timestamp; or, if the maximum historical synchronization timestamp is not stored, determine the user-configured synchronization timestamp or the preset synchronization timestamp as the maximum historical synchronization timestamp.

10. The data synchronization device according to claim 7, characterized in that, The source table contains a primary key field in its row data. The second determining module is specifically used for... The primary key fields of the candidate row data and the primary key fields of the reference row data are compared to determine the primary key fields that are the same in the candidate row data and the reference row data; The candidate row data corresponding to the same primary key field are identified as the duplicate row data.

11. The data synchronization device according to claim 7, characterized in that, The source table does not have a primary key field in its row data. The second determining module is specifically used for: Extract the summary information of the candidate row data and the summary information of the reference row data; The summary information of the candidate row data and the summary information of the reference row data are compared to determine the same summary information and the number of the same summary information. The number of candidate rows with the same summary information are identified as the duplicate rows.

12. The data synchronization device according to claim 8, characterized in that, The device further includes: an update module; The update module is used to update the stored maximum historical synchronization timestamp using the termination timestamp in the synchronization time period; or, when no maximum historical synchronization timestamp is stored, the termination timestamp is stored as the maximum historical synchronization timestamp.

13. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the data synchronization method as described in any one of claims 1-6.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data synchronization method as described in any one of claims 1-6.

Images