Verification method, device, computer equipment, and readable storage medium for incremental synchronization

Abstract

The present application belongs to the field of software testing technology, and relates to a verification method, device, computer equipment, computer-readable storage medium, and computer program product for incremental synchronization. The method includes: creating a first auxiliary table in a source database, and creating a second auxiliary table in a target database; the second auxiliary table includes at least a data write timestamp; writing verification data to the first auxiliary table based on a preset period; in the process of synchronizing the data table to be synchronized in the source database to the target database, synchronizing the verification data in the first auxiliary table to the second auxiliary table, and updating the data write timestamp in the second auxiliary table; determining whether the incremental synchronization process is running normally according to the current system timestamp, data write timestamp, and preset period of the target database. The verification method of the present application only needs to verify the data in the first auxiliary table and the second auxiliary table, and does not need to verify all the synchronized data, thereby improving the verification efficiency.

Description

Technical Field

[0001] The present application relates to the field of software testing technology, and in particular to a verification method, apparatus, computer equipment, computer-readable storage medium and computer program product for incremental synchronization. Background Art

[0002] Data synchronization between databases is a common requirement in modern information systems. Incremental database synchronization means that when synchronizing data between databases, only the changed data is transmitted and synchronized, which can reduce network bandwidth consumption and synchronization time, and improve the efficiency and real-time performance of data synchronization.

[0003] After data synchronization, it is very important to regularly verify the synchronized database. Common database verification methods usually require reading all the data in the database and then comparing them one by one to ensure data consistency and integrity. However, this verification method will occupy a lot of system resources, increase system load pressure, and have low verification efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide an incremental synchronization verification method, device, computer equipment, computer-readable storage medium and computer program product that can improve verification efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a verification method for incremental synchronization, the method comprising:

[0006] Create a first auxiliary table in the source database, and create a second auxiliary table in the target database; the second auxiliary table at least includes a data writing timestamp;

[0007] Writing verification data into the first auxiliary table based on a preset period;

[0008] In the process of synchronizing the data table to be synchronized in the source database to the target database, synchronizing the check data in the first auxiliary table to the second auxiliary table, and updating the data write timestamp in the second auxiliary table;

[0009] Determining whether the incremental synchronization process is running normally according to the current system timestamp of the target database, the data writing timestamp and the preset period;

[0010] In the case that the incremental synchronization process runs normally, a Bloom filter is created according to the first auxiliary table or the second auxiliary table, and based on the Bloom filter, it is determined whether there is any missing data in the incremental synchronization process.

[0011] In one embodiment, the method further comprises:

[0012] When the incremental synchronization process runs normally, obtaining a system time difference between a current system timestamp of the source database and a current system timestamp of the target database;

[0013] Obtaining a checkpoint timestamp in the check data;

[0014] The synchronization delay duration of the incremental synchronization process is obtained according to the data writing timestamp, the checkpoint timestamp and the system time difference.

[0015] In one embodiment, synchronizing the check data in the first auxiliary table to the second auxiliary table and updating the data write timestamp in the second auxiliary table includes:

[0016] Creating a data write trigger in the target database;

[0017] Create a field corresponding to the data writing timestamp in the second auxiliary table;

[0018] In the process of synchronizing the verification data in the first auxiliary table to the second auxiliary table, when new verification data is written into the second auxiliary table, the data write trigger is called to perform an update operation of the data write timestamp.

[0019] In one embodiment, the method further comprises:

[0020] Create a field corresponding to the data update timestamp in the second auxiliary table;

[0021] When the verification data in the second auxiliary table is updated, updating the data update timestamp;

[0022] According to the data update timestamp, it is determined whether there is duplicate data in the incremental synchronization process.

[0023] In one of the embodiments, creating a Bloom filter according to the first auxiliary table or the second auxiliary table, and determining whether there is lost data during the incremental synchronization process based on the Bloom filter, includes:

[0024] For one of the first auxiliary table and the second auxiliary table, obtaining a first verification data set of one of the auxiliary tables within a preset time period;

[0025] Creating a Bloom filter based on the amount of verification data in the first verification data set;

[0026] adding the first verification data set to the Bloom filter;

[0027] Acquire a second verification data set of another auxiliary table within the preset time period;

[0028] Determine whether there is any lost data during the incremental synchronization process according to the second verification data set and the Bloom filter.

[0029] In one embodiment, determining whether the incremental synchronization process is running normally according to the current system timestamp of the target database, the data write timestamp and the preset period includes:

[0030] When the difference between the current system timestamp of the target database and the data writing timestamp is greater than the preset period, determining that an abnormality occurs in the incremental synchronization process;

[0031] When the difference between the current system timestamp of the target database and the data writing timestamp is less than or equal to the preset period, it is determined that the incremental synchronization process is running normally.

[0032] In a second aspect, the present application further provides a verification device for incremental synchronization, the device comprising:

[0033] An auxiliary table creation module, used to create a first auxiliary table in a source database and a second auxiliary table in a target database; the second auxiliary table at least includes a data writing timestamp;

[0034] A data writing module, used for writing verification data into the first auxiliary table based on a preset period;

[0035] A data synchronization module, configured to synchronize the check data in the first auxiliary table to the second auxiliary table and update the data write timestamp in the second auxiliary table during synchronization of the data table to be synchronized in the source database to the target database;

[0036] A consistency judgment module, used to determine whether the incremental synchronization process is running normally according to the current system timestamp of the target database, the data writing timestamp and the preset period;

[0037] The missing data query module is used to create a Bloom filter according to the first auxiliary table or the second auxiliary table when the incremental synchronization process runs normally, and determine whether there is missing data in the incremental synchronization process based on the Bloom filter.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the incremental synchronization verification method provided in any of the above embodiments when executing the computer program.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the incremental synchronization verification method provided in any of the above embodiments is implemented.

[0040] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the incremental synchronization verification method provided in any of the above embodiments.

[0041] The verification method, device, computer equipment, computer-readable storage medium and computer program product of the incremental synchronization described above are as follows: by creating a first auxiliary table in the source database, creating a second auxiliary table in the target database, writing verification data to the first auxiliary table based on a preset period, synchronizing the verification data in the first auxiliary table to the second auxiliary table in the process of synchronizing the data table to be synchronized in the source database to the target database, and updating the data write timestamp in the second auxiliary table; further, according to the current system timestamp, data write timestamp and preset period of the target database, determining whether the incremental synchronization process is running normally, the abnormal situation of the incremental synchronization process can be discovered in time, and the situation that the newly modified data in the source database is not synchronized to the target database due to the stop or abnormal suspension of the data synchronization tool is avoided, so that the target database always saves the last synchronization data. In addition, the verification data is data related to the database ID and time written based on the preset period, and has nothing to do with the business data. In this way, when verifying the incremental synchronization, even if a lot of business data is modified, only the verification data needs to be verified, which improves the verification efficiency. In addition, when the incremental synchronization process runs normally, a Bloom filter can be created according to the first auxiliary table or the second auxiliary table. Based on the Bloom filter, it is determined whether there is lost data in the incremental synchronization process, so that data loss in the incremental synchronization process can be discovered in time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of a flow chart of a verification method for incremental synchronization in one embodiment;

[0044] Figure 2 A schematic diagram of a process for obtaining a synchronization delay time of a source database and a target database in one embodiment;

[0045] Figure 3 A schematic diagram of a process of checking whether there is lost data in one embodiment;

[0046] Figure 4 A schematic diagram of a process of checking whether there is duplicated data in one embodiment;

[0047] Figure 5 It is a flowchart of a verification method for incremental synchronization in a specific embodiment;

[0048] Figure 6 It is a structural block diagram of a verification device for incremental synchronization in one embodiment;

[0049] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It is understood that the terms "first" and "second" used in this application are only used to distinguish a first object from another object. For example, without departing from the scope of this application, a first auxiliary table can be referred to as a second auxiliary table, and similarly, a second auxiliary table can be referred to as a first auxiliary table. Both the first auxiliary table and the second auxiliary table are tables, but they are not the same table.

[0052] In an exemplary embodiment, the present application provides a verification method for incremental synchronization, which can be applied to a computer device, including a service, a script, or a program. Figure 1 As shown, the verification method for incremental synchronization includes the following steps 102-110.

[0053] Step 102: Create a first auxiliary table in the source database and create a second auxiliary table in the target database.

[0054] Among them, the target database is the database to which the data in the source database needs to be synchronized. The first auxiliary table may include information such as the source database ID, the checkpoint timestamp and the reporter ID. Among them, the source database ID is used to identify the source database, and is a unique identifier representing the identity of the source database, which can be represented by the IP address and port of the source database. The checkpoint timestamp points to the moment when the data is inserted into the source database. The reporter ID points to the application that inserts the data into the source database. The application can be a function service or a container, etc. The second auxiliary table is the table to which the data in the first auxiliary table needs to be synchronized. Therefore, the second auxiliary table includes at least the data items and the data write timestamp of the first auxiliary table. The data write timestamp refers to the moment when the data in the first auxiliary table is synchronized to the second auxiliary table.

[0055] Step 104: writing verification data into the first auxiliary table based on a preset period.

[0056] The preset period is a multiple of the system clock period, and can be reasonably set by the user according to database resources and verification accuracy. For example, the preset period can be 1s, 2s or 3s, etc. The verification data includes the IP address and port of the source database, the current database system time (i.e., the verification point timestamp), and the ID of the reporter's application.

[0057] Step 106: in the process of synchronizing the data table to be synchronized in the source database to the target database, synchronize the verification data in the first auxiliary table to the second auxiliary table, and update the data writing timestamp in the second auxiliary table.

[0058] The computer device can call the data synchronization tool to synchronize the data table to be synchronized in the source database to the target database. The data table to be synchronized refers to the table whose data has been modified in the source database. It can be understood that when performing database synchronization, the present application adopts the incremental synchronization method, that is, only the changed data is transmitted and synchronized. Since new verification data is written into the first auxiliary table at regular intervals, in the process of synchronizing the data table to be synchronized in the source database to the target database, it is necessary to synchronize the newly added verification data in the first auxiliary table to the second auxiliary table, and update the data write timestamp corresponding to the newly added verification data in the second auxiliary table.

[0059] Step 108: Determine whether the incremental synchronization process is running normally according to the current system timestamp, data writing timestamp and preset period of the target database.

[0060] The current system timestamp refers to the timestamp of the database system at the time of verification. When a computer device obtains a data write timestamp, it may obtain multiple values. It is necessary to first sort these multiple values ​​in descending order and output only the first value to ensure that the obtained data write timestamp is the latest data write timestamp. The computer device can monitor the delay of the incremental synchronization process based on the current system timestamp and data write timestamp of the target database, and determine whether the incremental synchronization process is running normally based on the delay and the preset period.

[0061] Step 110: When the incremental synchronization process is running normally, a Bloom filter is created according to the first auxiliary table or the second auxiliary table, and based on the Bloom filter, it is determined whether there is any missing data in the incremental synchronization process.

[0062] Bloom Filter is a probabilistic data structure, which consists of a binary vector and a series of random mapping function groups. It can be used to determine whether an element is in a set. It has the advantages of fast running speed and small memory usage. Specifically, a Bloom filter can be created based on the first auxiliary table, and the Bloom filter can be used to quickly query whether the data in the second auxiliary table is in the first auxiliary table to determine whether there is any missing data during the incremental synchronization process. Optionally, a Bloom filter can be created based on the second auxiliary table, and the Bloom filter can be used to quickly query whether the data in the first auxiliary table is in the second auxiliary table to determine whether there is any missing data during the incremental synchronization process.

[0063] In an embodiment of the present application, by creating a first auxiliary table in the source database, creating a second auxiliary table in the target database, writing verification data to the first auxiliary table based on a preset period, in the process of synchronizing the data table to be synchronized in the source database to the target database, synchronizing the verification data in the first auxiliary table to the second auxiliary table, and updating the data write timestamp in the second auxiliary table, further, according to the current system timestamp of the target database, the data write timestamp and the preset period, it is determined whether the incremental synchronization process is running normally, and the abnormal situation of the incremental synchronization process can be discovered in time, so as to avoid the situation that the newly modified data in the source database is not synchronized to the target database due to the stop or abnormal suspension of the data synchronization tool, so that the target database always saves the last synchronization data. In addition, the verification data is data related to the database ID and the write time written based on the preset period, and has nothing to do with the business data. In this way, when verifying the incremental synchronization, even if a lot of business data is modified, only the verification data needs to be verified, which improves the verification efficiency. In addition, when the incremental synchronization process is running normally, a Bloom filter can also be created according to the first auxiliary table or the second auxiliary table. Based on the Bloom filter, data loss in the incremental synchronization process can be quickly discovered.

[0064] In one embodiment, determining whether the incremental synchronization process is running normally is performed based on the current system timestamp, data write timestamp and preset period of the target database, including: when the difference between the current system timestamp and the data write timestamp of the target database is greater than the preset period, determining that an abnormality has occurred in the incremental synchronization process; and when the difference between the current system timestamp and the data write timestamp of the target database is less than or equal to the preset period, determining that the incremental synchronization process is running normally.

[0065] The incremental synchronization process is always ongoing, and incremental synchronization will be triggered as long as there is data change in the source database. The data of the first auxiliary table in the source database is changed according to the preset period. Similarly, the data of the second auxiliary table in the target database will also be inserted with new verification data based on the preset period. Therefore, when the difference between the current system timestamp of the target database and the data write timestamp is greater than the preset period, it means that the second auxiliary table has not synchronized new verification data for more than the preset period, and an abnormality has occurred in the incremental synchronization process. When the difference between the current system timestamp of the target database and the data write timestamp is less than or equal to the preset period, it means that new verification data has been inserted into the second auxiliary table, and the incremental synchronization process is normal.

[0066] In one embodiment, after determining that the incremental synchronization process is proceeding normally according to the current system timestamp, data write timestamp and preset period of the target database, the synchronization delay duration of the source database and the target database can also be obtained. Figure 2 As shown, the incremental synchronization verification method also includes steps 202-206.

[0067] Step 202: When the incremental synchronization process is running normally, the system time difference between the current system timestamp of the source database and the current system timestamp of the target database is obtained.

[0068] The current system timestamp of the source database and the current system timestamp of the target database can be obtained by calling the function for obtaining the current time in the library function. Exemplarily, in a Mysql database system, the current system timestamp of the source database and the current system timestamp of the target database can be obtained by calling the library function now(). In an oracle database system, the current system timestamp of the source database and the current system timestamp of the target database can be obtained by calling the library function sysdate(). Optionally, the current system timestamp of the source database and the current system timestamp of the target database can also be obtained through a custom function or program. If the time zone corresponding to the current system timestamp of the source database is different from the time zone corresponding to the current system timestamp of the target database, it is also necessary to convert the current system timestamp of the source database and the current system timestamp of the target database to the same time zone, and then calculate the system time difference between the two.

[0069] Step 204, obtaining the checkpoint timestamp in the check data.

[0070] There are verification data in both the first auxiliary table and the second auxiliary table. Therefore, the verification point timestamp can be obtained from the first auxiliary table or from the second auxiliary table.

[0071] Step 206: Obtain the synchronization delay duration of the incremental synchronization process according to the data writing timestamp, the checkpoint timestamp and the system time difference.

[0072] The synchronization delay duration is the value obtained by subtracting the checkpoint timestamp and the system time difference from the data write timestamp. Synchronization delay duration = data write timestamp - subtract checkpoint timestamp - system time difference.

[0073] In the embodiment of the present application, when the incremental synchronization process is running normally, the system time difference between the current system timestamp of the source database and the current system timestamp of the target database is obtained, the checkpoint timestamp in the check data is obtained, and the synchronization delay duration of the incremental synchronization process is obtained according to the data writing timestamp, the checkpoint timestamp and the system time difference. When calculating the synchronization delay duration, the system time difference between the source database and the target database is taken into account, thereby improving the accuracy and precision of the synchronization delay duration.

[0074] In one embodiment, the verification data in the first auxiliary table is synchronized to the second auxiliary table, and the data write timestamp is updated in the second auxiliary table, including: creating a data write trigger in the target database; creating a field corresponding to the data write timestamp in the second auxiliary table; during the process of synchronizing the verification data in the first auxiliary table to the second auxiliary table, when new verification data is written in the second auxiliary table, calling the data write trigger to perform the step of updating the data write timestamp.

[0075] A trigger is a database object related to a table, which is triggered when the defined conditions are met and executes the set of statements defined in the trigger. In this embodiment, the write trigger is used to monitor whether there is new data inserted into the second auxiliary table, and when new data is detected to be inserted, the value of the data update timestamp corresponding to the data is updated to the system time at the current moment. It can be understood that the computer device creates a write trigger and creates a field corresponding to the data write timestamp in the second auxiliary table. When new verification data is written in the second auxiliary table, the write trigger is triggered, the system time at the current moment is obtained, and the value of the field corresponding to the data write timestamp is updated to the system time at the current moment.

[0076] In this embodiment, by creating a write trigger to monitor data write events in the second auxiliary table and updating the corresponding data write timestamp, the data in the second auxiliary table can be modified without using a data synchronization tool, which can be decoupled from incremental synchronization to avoid affecting the synchronized data.

[0077] In one embodiment, Figure 3 As shown, the incremental synchronization verification method provided by the present application also includes steps 302-306.

[0078] Step 302: Create a field corresponding to the data update timestamp in the second auxiliary table.

[0079] The data update timestamp refers to the time when the data already existing in the second auxiliary table is updated.

[0080] Step 304: When the verification data in the second auxiliary table is updated, the timestamp of the updated data is updated.

[0081] When the check data already existing in the second auxiliary table is updated, the value of the data update timestamp is updated to the current system time. Exemplarily, an update trigger may be created. The update trigger is used to monitor the data update event of the second auxiliary table, and when the check data already existing in the second auxiliary table is updated, the value of the data update timestamp is updated to the current system time.

[0082] Step 306: Determine whether there is duplicate data in the incremental synchronization process according to the data update timestamp.

[0083] It can be understood that if the value of the data update timestamp is not the initial value, it means that the verification data in the second auxiliary table has been updated, that is, there is duplicate data in the incremental synchronization process; if the value of the data update timestamp is the initial value, it means that the verification data in the second auxiliary table has not been updated, that is, there is no duplicate data in the incremental synchronization process.

[0084] In this embodiment, by creating a field corresponding to the data update timestamp in the second auxiliary table, when the verification data in the second auxiliary table is updated, the data update timestamp is updated, and based on the data update timestamp, it is determined whether there is duplicate data in the incremental synchronization process, so that the duplicate data in the incremental synchronization process can be discovered in time.

[0085] In one embodiment, Figure 4 As shown, creating a Bloom filter according to the first auxiliary table or the second auxiliary table, and determining whether there is lost data in the incremental synchronization process based on the Bloom filter includes steps 402-410.

[0086] Step 402: for one of the first auxiliary table and the second auxiliary table, obtain a first verification data set of one of the auxiliary tables within a preset time period.

[0087] Step 404: Create a Bloom filter based on the number of verification data in the first verification data set.

[0088] Step 406: Add the first verification data set to the Bloom filter.

[0089] Step 408: Obtain a second verification data set of another auxiliary table within a preset time period.

[0090] Step 410: Determine whether there is any lost data during the incremental synchronization process based on the second verification data set and the Bloom filter.

[0091] Among them, the preset time period can be reasonably set according to the amount of data to be verified, for example, it can be half an hour, 1 hour, 2 hours or 3 hours, etc. Bloom Filter is a probabilistic data structure, consisting of a binary vector and a series of random mapping function groups, which can be used to determine whether an element is in a set. It has the advantages of fast running speed and small memory usage. If there is no data loss, the first verification data set and the second verification data set should be the same. Therefore, a Bloom filter can be created based on the first verification data set. By querying each verification data in the second verification data set through the Bloom filter whether it is in the first verification data set, it can be determined whether there is any missing data during the incremental synchronization process.

[0092] Exemplarily, a first verification data set of the first auxiliary table within a preset time period may be obtained, and the capacity of the Bloom filter may be determined according to the number of verification data in the first verification data set, and the Bloom filter may be created. A second verification data set of the second auxiliary table within a preset time period may be obtained, and the Bloom filter may be used to query whether each verification data in the second verification data set is in the first verification data set. If there is a situation where the verification data in the second verification data set is not in the first verification data set, it indicates that there is lost data in the incremental synchronization process.

[0093] Optionally, the first verification data set of the second auxiliary table within a preset time period can be obtained, and the capacity of the Bloom filter can be determined according to the number of verification data in the first verification data set to create the Bloom filter. The second verification data set of the first auxiliary table within the preset time period is obtained, and each verification data in the second verification data set is queried through the Bloom filter to see whether it is in the first verification data set. If there is a situation where the verification data in the second verification data set is not in the first verification data set, it means that there is lost data in the incremental synchronization process.

[0094] In this embodiment, by creating a Bloom filter, it is possible to quickly query whether there is data loss during the incremental synchronization process.

[0095] In order to better illustrate the incremental synchronization verification method provided by the present application, a more detailed embodiment is listed below, including the following steps:

[0096] S1, create the first auxiliary table sync_report_1 in the source database, and create the second auxiliary table sync_report_2 in the target database.

[0097] The attributes in the first auxiliary table include source database ID, checkpoint timestamp and reporter ID. The attributes in the second auxiliary table include source database ID, checkpoint timestamp, reporter ID, data write timestamp and data update timestamp.

[0098] S2, create a write trigger function and an update trigger function for the second auxiliary table in the target database.

[0099] S3. Create a scheduled insert function for the first auxiliary table in the source database.

[0100] The timed insertion function is the reporter application mentioned above. The verification data is inserted into the first auxiliary table every 3 seconds through the timed insertion function. The verification data is the source database IP address and port, the current system timestamp and the reporter ID. Before inserting the verification data into the first auxiliary table through the timed insertion function, it is necessary to establish a secure connection between the timed insertion function and the source database. Specifically, the timed insertion function is called first to generate identity information based on the source database's IP address, port, source database name, database user name and database password, and determine whether there is permission to open the source database. If the source database cannot be opened successfully, the user may be prompted that the database opening failed. If the source database can be opened successfully, the verification data is inserted into the corresponding field of the first auxiliary table.

[0101] S4, start the data synchronization tool, configure the synchronization plan from the first auxiliary table to the second auxiliary table, and start the incremental synchronization process.

[0102] Query the first auxiliary table. As shown in Table 1, data has been written into the first auxiliary table:

[0103] Table 1 The first auxiliary table

[0104]

[0105] Query the second auxiliary table, as shown in Table 2, the data has been synchronized:

[0106] Table 2 Second auxiliary table

[0107]

[0108] S5, create a verification function to verify the incremental synchronization process.

[0109] Specifically, Figure 5 As shown in the figure, the checkpoint timestamp, data write timestamp and current system timestamp in the target database are obtained. The checkpoint timestamp is "2024-5-10 08:23:56.018", the data write timestamp is "2024-5-10 08:23:57.497", and the current system timestamp is "2024-5-10 08:23:57.864". The difference between the current system timestamp and the data write timestamp is 0.367 seconds, which is less than 3 seconds, indicating that the incremental synchronization process is running normally.

[0110] Obtaining the synchronization delay duration: In this embodiment, the target database and the source database are set in the same database system, so the system time difference is 0. According to the data writing timestamp and the checkpoint timestamp, the synchronization delay duration can be obtained as 1.479 seconds.

[0111] Check whether there is any missing data: query all the verification data in the first auxiliary table within one hour to form the first verification data set. Define a Bloom filter according to the number of the first verification data set, and add the verification data in the first verification data set to the Bloom filter. Query all the verification data in the second auxiliary table within one hour to form the second verification data set. Query whether each verification data in the second verification data set is in the Bloom filter. If there is a verification data in the second auxiliary table that does not exist in the Bloom filter, it means that there is missing data; if all the verification data in the second auxiliary table are in the Bloom filter, it means that there is no missing data.

[0112] Check whether there is duplicate data: Get the value of the data update timestamp. If the value of the data update timestamp is the initial value, it means that there is no duplicate data; if the value of the data update timestamp is not the initial value, it means that there is duplicate data.

[0113] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] Based on the same inventive concept, the embodiment of the present application also provides an incremental synchronization verification device for implementing the incremental synchronization verification method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the embodiments of one or more incremental synchronization verification devices provided below can refer to the limitations of the incremental synchronization verification method above, and will not be repeated here.

[0115] In an exemplary embodiment, Figure 6 As shown, a verification device for incremental synchronization is provided, including: an auxiliary table creation module 602, a data writing module 604, a data synchronization module 606, a consistency judgment module 608 and a lost data query module 610, wherein:

[0116] Auxiliary table creation module 602, used to create a first auxiliary table in the source database and a second auxiliary table in the target database; the second auxiliary table at least includes a data writing timestamp;

[0117] A data writing module 604, configured to write verification data into the first auxiliary table based on a preset period;

[0118] The data synchronization module 606 is used to synchronize the check data in the first auxiliary table to the second auxiliary table and update the data writing timestamp in the second auxiliary table during the process of synchronizing the data table to be synchronized in the source database to the target database;

[0119] The consistency judgment module 608 is used to determine whether the incremental synchronization process is running normally according to the current system timestamp, data writing timestamp and preset period of the target database;

[0120] The missing data query module 610 is used to create a Bloom filter according to the first auxiliary table or the second auxiliary table when the incremental synchronization process runs normally, and determine whether there is missing data in the incremental synchronization process based on the Bloom filter.

[0121] In one embodiment, the verification device for incremental synchronization also includes a delay acquisition module, which is used to obtain the system time difference between the current system timestamp of the source database and the current system timestamp of the target database when the incremental synchronization process runs normally; obtain the verification point timestamp in the verification data; and obtain the synchronization delay duration of the incremental synchronization process based on the data write timestamp, the verification point timestamp and the system time difference.

[0122] In one embodiment, the data synchronization module is also used to create a data write trigger in the target database; create a field corresponding to the data write timestamp in the second auxiliary table; during the process of synchronizing the verification data in the first auxiliary table to the second auxiliary table, when new verification data is written in the second auxiliary table, call the data write trigger and execute the data write timestamp update operation.

[0123] In one embodiment, the verification device for incremental synchronization also includes a duplicate data query module, which is used to create a field corresponding to the data update timestamp in the second auxiliary table; when the verification data in the second auxiliary table is updated, the data update timestamp is updated; and based on the data update timestamp, it is determined whether there is duplicate data in the incremental synchronization process.

[0124] In one embodiment, the verification device for incremental synchronization also includes a missing data query module, which is used to obtain a first verification data set of one of the auxiliary tables within a preset time period for the first auxiliary table and the second auxiliary table; create a Bloom filter based on the number of verification data in the first verification data set; add the first verification data set to the Bloom filter; obtain a second verification data set of the other auxiliary table within a preset time period; and determine whether there is missing data during the incremental synchronization process based on the second verification data set and the Bloom filter.

[0125] In one embodiment, the consistency judgment module is also used to determine that an abnormality has occurred in the incremental synchronization process when the difference between the current system timestamp of the target database and the data write timestamp is greater than a preset period; and to determine that the incremental synchronization process is running normally when the difference between the current system timestamp of the target database and the data write timestamp is less than or equal to a preset period.

[0126] Each module in the above-mentioned incremental synchronization verification device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0127] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data obtained from a source database and a target database. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a verification method for incremental synchronization is implemented.

[0128] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the incremental synchronization verification method provided in any of the above embodiments is implemented.

[0130] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the incremental synchronization verification method provided in any of the above embodiments is implemented.

[0131] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the incremental synchronization verification method provided in any of the above embodiments.

[0132] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A verification method for incremental synchronization, characterized in that: The method comprises: Create a first auxiliary table in the source database, and create a second auxiliary table in the target database; the second auxiliary table at least includes a data writing timestamp; Writing verification data into the first auxiliary table based on a preset period; In the process of synchronizing the data table to be synchronized in the source database to the target database, synchronizing the check data in the first auxiliary table to the second auxiliary table, and updating the data write timestamp in the second auxiliary table; Determining whether the incremental synchronization process is running normally according to the current system timestamp of the target database, the data writing timestamp and the preset period; When the incremental synchronization process is running normally, for one of the first auxiliary table and the second auxiliary table, a first verification data set of one of the auxiliary tables within a preset time period is obtained; Creating a Bloom filter based on the amount of verification data in the first verification data set; adding the first verification data set to the Bloom filter; Acquire a second verification data set of another auxiliary table within the preset time period; Determine whether there is any lost data during the incremental synchronization process according to the second verification data set and the Bloom filter.

2. The method according to claim 1, characterized in that The method further comprises: When the incremental synchronization process runs normally, obtaining a system time difference between a current system timestamp of the source database and a current system timestamp of the target database; Obtaining a checkpoint timestamp in the check data; The synchronization delay duration of the incremental synchronization process is obtained according to the data writing timestamp, the checkpoint timestamp and the system time difference.

3. The method according to claim 1, characterized in that The step of synchronizing the check data in the first auxiliary table to the second auxiliary table and updating the data writing timestamp in the second auxiliary table includes: Creating a data write trigger in the target database; Create a field corresponding to the data writing timestamp in the second auxiliary table; In the process of synchronizing the verification data in the first auxiliary table to the second auxiliary table, when new verification data is written into the second auxiliary table, the data write trigger is called to perform an update operation of the data write timestamp.

4. The method according to claim 1, characterized in that: The method further comprises: Create a field corresponding to the data update timestamp in the second auxiliary table; When the check data in the second auxiliary table is updated, updating the data update timestamp; According to the data update timestamp, it is determined whether there is duplicate data in the incremental synchronization process.

5. The method according to claim 1, characterized in that The step of determining whether the incremental synchronization process is running normally according to the current system timestamp of the target database, the data writing timestamp and the preset period includes: When the difference between the current system timestamp of the target database and the data writing timestamp is greater than the preset period, determining that an abnormality occurs in the incremental synchronization process; When the difference between the current system timestamp of the target database and the data writing timestamp is less than or equal to the preset period, it is determined that the incremental synchronization process is running normally.

6. A verification device for incremental synchronization, characterized in that: The device comprises: An auxiliary table creation module, used to create a first auxiliary table in a source database and a second auxiliary table in a target database; the second auxiliary table at least includes a data writing timestamp; A data writing module, used for writing verification data into the first auxiliary table based on a preset period; A data synchronization module, configured to synchronize the check data in the first auxiliary table to the second auxiliary table and update the data write timestamp in the second auxiliary table during synchronization of the data table to be synchronized in the source database to the target database; A consistency judgment module, used to determine whether the incremental synchronization process is running normally according to the current system timestamp of the target database, the data writing timestamp and the preset period; A missing data query module is used to, when the incremental synchronization process is operating normally, obtain a first verification data set of one of the first auxiliary table and the second auxiliary table within a preset time period; create a Bloom filter based on the number of verification data in the first verification data set; add the first verification data set to the Bloom filter; obtain a second verification data set of the other auxiliary table within the preset time period; and determine whether there is missing data in the incremental synchronization process based on the second verification data set and the Bloom filter.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images