Data comparison method and device, electronic equipment, chip and medium

Abstract

The invention provides a data comparison method and device, electronic equipment, a chip and a medium. The method comprises the steps of obtaining source data of a source database, target data of a target database and a data source of the source data or the target data, wherein the data source comprises the source data, the target data and user operation; performing full-amount comparison on the source data and the target data, and adding the data which are inconsistent in comparison into a difference data set; determining comparison delay time according to the time difference between the write-in time and the transmission time of the source data or the target data; monitoring execution commands and data sources of the source data and the target data, comparing the source data with the target data based on the comparison delay time, and adding inconsistent data into a difference data set; and performing data comparison on the difference data set by taking a preset time as a period. Through the technical scheme provided by the invention, the problem of low data comparison efficiency in the database synchronization process is solved, the data comparison efficiency is improved, and the data consistency of the synchronous database is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of databases, and in particular to a data comparison method, device, electronic device, chip and medium. Background Art

[0002] In the use of databases, it is very important to maintain the data consistency of the database. In the related art, the method of maintaining the consistency of the database is to compare all the data through a single comparison. Some data will be compared repeatedly, which is inefficient on the one hand and wastes resources on the other. Although comparing sampled data can improve efficiency, it cannot guarantee the correctness of all data. In other words, data comparison in the related art has the problems of low efficiency and difficulty in ensuring data consistency. Summary of the invention

[0003] The present disclosure provides a data comparison method, device, electronic device, chip and medium to solve the problems of low efficiency and difficulty in ensuring data consistency in database data comparison. By comparing the full amount of the source database and the target database, the data writing status of the source database and the target database is monitored, and data comparison is performed on the newly added, deleted and updated data. During the comparison process, the delay time of the system is calculated to delay the data comparison. During the entire data comparison process, the data that is inconsistent with the comparison is recorded, and the comparison is performed again regularly, and the data that is consistent with the comparison is removed. Through this dynamic comparison method, the database replication is not affected, and the problem of inconsistent data comparison caused by system delays is reduced. By dynamically maintaining inconsistent data, only inconsistent data will be compared repeatedly, which improves the efficiency of data comparison.

[0004] A first aspect of the present disclosure provides a data comparison method, the method comprising:

[0005] Obtain source data of a source database, target data of a target database, and data sources of source data or target data, where the data sources include source data, target data, and user operations;

[0006] Perform a full comparison of the source data and the target data, and add the inconsistent data to the difference data set;

[0007] Determine the comparison delay time according to the time difference between the writing time and the transmission time of the source data or the target data;

[0008] Monitor the execution commands and data sources of source data and target data, and compare the source data and target data based on the comparison delay time, and add inconsistent data to the difference data set;

[0009] The data comparison is performed on the difference data sets at a preset time period.

[0010] In one embodiment of the present disclosure, obtaining source data of a source database and target data of a target database includes:

[0011] Obtain source data and the first serial number of the source data, where the first serial number is the serial number of the latest execution statement for the source data;

[0012] The target data and the second serial number of the target data are obtained, where the second serial number is the serial number of the latest execution statement for the target data.

[0013] In one embodiment of the present disclosure, determining the comparison delay time according to the time difference between the writing time and the transmission time of the source data or the target data includes:

[0014] Obtaining a first delay time, and continuously recording a time difference between a writing time to a source database or a target database and a transmission time;

[0015] A comparison delay time is determined according to the first delay time, a preset number of time differences and a preset time offset value.

[0016] In one embodiment of the present disclosure, the execution commands and data sources of source data and target data are monitored, and based on the comparison delay time, the source data and target data are compared, and inconsistent data are added to the difference data set, including:

[0017] Monitor the execution commands of source data and target data and the data sources corresponding to the execution commands;

[0018] If the sequence number of the current execution command for the source database is greater than the first sequence number, or the sequence number of the current execution command for the target database is greater than the second sequence number, the source data and the target data are compared according to the data source after the comparison delay time is postponed, and the inconsistent data is added to the difference data set.

[0019] In one embodiment of the present disclosure, comparing source data and target data according to the data source includes:

[0020] For the first data to be compared in the source database, if the data source is the target data, skip this comparison; if the data source is a user operation, add the first data to be compared to the comparison queue;

[0021] For the second data to be compared in the target database, if the data source is source data, this comparison is skipped; if the data source is user operation, the second data to be compared is added to the comparison queue.

[0022] In one embodiment of the present disclosure, after comparing the source data and the target data according to the data source, the following steps are further included:

[0023] The first data to be compared in the comparison queue is synchronized to the target database with the first time period, or,

[0024] The second data to be compared in the comparison queue is synchronized to the source database.

[0025] In one embodiment of the present disclosure, data comparison is performed on the difference data set at a preset time period, including:

[0026] Based on the preset time period, the data in the difference data set are compared pairwise according to the data source and data content, and the data with consistent comparison is deleted from the difference data set.

[0027] A second aspect of the present disclosure provides a data comparison device, the device comprising:

[0028] An acquisition module is used to acquire source data of a source database, target data of a target database, and data sources of source data or target data, where the data sources include source data, target data, and user operations;

[0029] The full comparison module is used to perform a full comparison of the source data and the target data, and add the inconsistent data to the difference data set;

[0030] A delay determination module, used to determine a comparison delay time according to a time difference between a writing time and a transmission time of source data or target data;

[0031] The monitoring and filtering module is used to monitor the execution commands and data sources of the source data and the target data, and compare the source data and the target data based on the comparison delay time, and add the inconsistent data to the difference data set;

[0032] The difference comparison module is used to compare the difference data sets at a preset time period.

[0033] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods in the first aspect embodiment of the present disclosure.

[0034] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the method in the first aspect embodiment of the present disclosure.

[0035] The fifth aspect embodiment of the present disclosure proposes a computer program product, characterized in that it includes a computer program, and the computer program implements any method in the first aspect embodiment of the present disclosure when executed by a processor.

[0036] The sixth aspect embodiment of the present disclosure proposes a chip, comprising at least one processor and a communication interface; the communication interface is used to receive signals input into the chip or signals output from the chip, the processor communicates with the communication interface and implements any one of the methods in the first aspect embodiment of the present disclosure through logic circuits or execution code instructions.

[0037] In summary, according to the data comparison method proposed in the present disclosure, the following beneficial effects can be achieved:

[0038] Obtain the source data of the source database, the target data of the target database, and the data sources of the source data or target data, including the source data, target data and user operations, to provide a data source for the data comparison of the database; perform a full comparison of the source data and the target data, add the inconsistent data to the difference data set, realize the comparison of the full data of the database, and determine the inconsistent data; determine the comparison delay time based on the time difference between the write time and the transmission time of the source data or the target data, and provide the delay time caused by data transmission for accurate data comparison; monitor the execution commands and data sources of the source data and the target data, and compare the source data and the target data based on the comparison delay time, add the inconsistent data to the difference data set, avoid repeated comparison caused by data transmission, and improve the efficiency of data comparison results; perform data comparison on the difference data set at a preset time period to improve data consistency.

[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0041] Figure 1 It is a schematic diagram of a database transmission service system in the related art;

[0042] Figure 2 A flow chart of a data comparison method according to an embodiment of the present disclosure;

[0043] Figure 3 A flow chart of performing data comparison based on data sources according to an embodiment of the present disclosure;

[0044] Figure 4 A flowchart of a data dynamic maintenance comparison result according to an embodiment of the present disclosure;

[0045] Figure 5 is a schematic diagram of a data comparison system according to an embodiment of the present disclosure;

[0046] Figure 6 It is a structural schematic diagram of a data comparison device according to an embodiment of the present disclosure;

[0047] Figure 7 is a block diagram of an electronic device for implementing the data comparison method disclosed herein according to an exemplary embodiment;

[0048] Figure 8 It is a schematic diagram of the structure of the chip of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout identify the same or similar originals or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0050] In database transmission services, in normal business data modification scenarios, or scenarios with more reads and less data modification, such as content display platforms, e-commerce websites, etc., data comparison is required to ensure data consistency between the two databases.

[0051] like Figure 1 This is a schematic diagram of a database transmission service system in the related art. As shown in the figure, the source database transmits all data to the target end. When the source end adds, deletes, or modifies data, it is synchronized to the target end to perform the same addition, deletion, or modification operations. When the target end performs addition, deletion, or modification operations, it is also transmitted to the source end to perform the same addition, deletion, or modification operations.

[0052] When comparing all the data in the database, some data will be compared repeatedly, which is inefficient and results in a waste of computing resources.

[0053] In addition, the related technical solutions need to stop the ongoing data storage when performing data comparison, and enter the data into the target database according to the comparison results. This method makes it impossible for the database to synchronize data normally during the data comparison, and will cause data delays caused by data comparison. This makes it difficult to ensure consistency in data comparison.

[0054] The method proposed in the present disclosure is applied to data comparison tasks, and its application has a rich variety of scenarios. In the process of migrating data from one database to another, the method disclosed in the present disclosure can be used to ensure the integrity and consistency of data. For example, when an enterprise upgrades a data warehouse or switches a database management system, it is necessary to ensure that the data in the new and old systems are consistent. In the scenario of real-time data analysis, the method disclosed in the present disclosure can be used to monitor and compare real-time data to ensure the accuracy of the analysis results. For example, an e-commerce platform monitors user behavior data to adjust the recommendation algorithm in real time. In the process of data backup and recovery, the method disclosed in the present disclosure can be used to verify the integrity and consistency of the backup data. For example, when an enterprise performs regular data backup, it is necessary to confirm that the backup data is consistent with the source data. By comparing and analyzing business data in different time periods, system performance bottlenecks can be discovered and optimized. For example, an e-commerce platform optimizes server configuration to improve response speed by analyzing user access data. When performing data governance and compliance checks, this method can be used to compare and verify data to meet regulatory requirements. For example, financial institutions need to regularly check whether customer data meets regulatory requirements. In scientific research and experimental data analysis, this method can be used to compare data under different experimental conditions to verify research hypotheses. For example, in medical research, the effects of different treatment options are compared. In market research and competition analysis, this method can be used to compare the product features and market share of different competitors. For example, a company analyzes the sales strategies of competitors to formulate its own market strategy. The application scenario is not limited in the embodiments of the present disclosure.

[0055] The data comparison method provided by the present disclosure is described in detail below with reference to the accompanying drawings.

[0056] Figure 2 FIG. 1 is a flow chart of a data comparison method according to an embodiment of the present disclosure. Figure 2 In the embodiment shown, the data comparison method includes:

[0057] Step 201, obtaining source data of a source database, target data of a target database, and data sources of source data or target data, where the data sources include source data, target data, and user operations.

[0058] The source database refers to one of the two databases that need to be compared, and the source data refers to the data stored in the source database. The target database refers to another database relative to the source database, and the target data refers to the data stored in the target database. The data source refers to the source of the data in the source database or the target database. In other words, the source of the target data or the source data before it is written in the target database or the source database. The data source includes the source database or the target database for data synchronization between databases, and the data generated by user operations. Before performing a database data comparison, first read the source data from the source database, read the target data from the target database, and read the data source of the source data or the target data to provide data preparation for data comparison.

[0059] Step 202 : perform a full comparison of the source data and the target data, and add the inconsistent data to the difference data set.

[0060] The difference data set refers to a data set consisting of inconsistent data between the source data and the target data. After obtaining the source data and the target data, the source data and the target data are fully compared, that is, the source data and the target data are comprehensively compared line by line and field by field. The inconsistent data in the comparison results are added to the difference data set as a small range of data sets that can be further compared. Optionally, the data format of the difference data set can be a list, a set, or a database or a data table.

[0061] Step 203: Determine a comparison delay time according to the time difference between the writing time and the transmission time of the source data or the target data.

[0062] The comparison delay time refers to the time required for the two databases to be compared to complete data synchronization. In order to avoid data comparison inconsistency caused by data replication delays caused by network factors, the comparison delay time is determined by monitoring the time difference between the writing time and transmission time of the source data or target data, providing a time basis for postponing data comparison to the appropriate time.

[0063] Step 204 , monitoring the execution commands and data sources of the source data and the target data, and comparing the source data and the target data based on the comparison delay time, and adding inconsistent data to the difference data set.

[0064] Monitor the operations that occur when source data is executed in the source database by commands, or monitor the operations that occur when target data is executed in the target database by commands, and optionally, the execution commands include adding, deleting, and modifying. According to the execution commands and data sources of the source data and target data, the source data and target data are compared after the comparison delay time to ensure the reliability of data comparison, and the inconsistent data is added to the difference data set.

[0065] Step 205 , performing data comparison on the difference data sets at a preset time period.

[0066] Regularly compare the difference data sets, and recheck whether there is any data that is consistent with the comparison from the difference data sets. If there is any data that is consistent with the comparison, it can be removed from the difference data sets, so as to dynamically maintain the comparison results.

[0067] In summary, according to the data comparison method proposed in the present invention, the source data of the source database, the target data of the target database, and the data source of the source data or the target data are obtained, and the data source includes the source data, the target data and the user operation, which provides a data source for the data comparison of the database; the source data and the target data are fully compared, and the inconsistent data is added to the difference data set, so that the comparison of the full amount of data in the database is realized and the inconsistent data is determined; the comparison delay time is determined according to the time difference between the writing time and the transmission time of the source data or the target data, and the delay time caused by the data transmission is provided for the data comparison to achieve accuracy; the execution commands and data sources of the source data and the target data are monitored, and the source data and the target data are compared based on the comparison delay time, and the inconsistent data is added to the difference data set, so as to avoid repeated comparison caused by data transmission and improve the efficiency of the data comparison results; the difference data set is compared with the preset time as a period, so as to improve the consistency of the data.

[0068] In a possible embodiment of the present disclosure, obtaining source data of a source database and target data of a target database can be further interpreted as obtaining source data and a first serial number of the source data, the first serial number being the serial number of the latest execution statement for the source data; and obtaining target data and a second serial number of the target data, the second serial number being the serial number of the latest execution statement for the target data.

[0069] In this embodiment, the execution statement refers to the instruction code executed on the source database or the target database. The first serial number refers to the serial number of the latest execution statement that completes the full comparison and accesses the source data, and the second serial number refers to the serial number of the latest execution statement that completes the full comparison and accesses the target data. In the process of obtaining the source data of the source database, the first serial number of the source data in the source database is obtained at the same time. Similarly, in the process of obtaining the target data of the target database, the second serial number of the target data in the target database is obtained at the same time.

[0070] In one implementation of this embodiment, in the full comparison stage, when obtaining source data, the latest execution statement sourceFullId of the source database is recorded, and the latest execution statement targetFullId of the target database is recorded. Then, the same data is obtained from the target database based on the source data. If the data is not obtained from the target database or the comparison result is inconsistent after the data is obtained, it is added to the difference data set with different comparison results.

[0071] In this embodiment, the first serial number and the second serial number can reflect the serial numbers of the latest executed commands of the source database and the target database, and provide a basis for judging whether operations such as writing, deleting, and modifying the database will occur thereafter.

[0072] In a possible embodiment of the present disclosure, determining the comparison delay time based on the time difference between the writing time and the transmission time of the source data or the target data can be further interpreted as obtaining a first delay time, and continuously recording the time difference between the writing time and the transmission time of the source database or the target database; determining the comparison delay time based on the first delay time, a preset number of time differences and a preset time offset value.

[0073] In this embodiment, the first delay time refers to the initial delay time of the data comparison system for data comparison. Optionally, the first delay time can be directly obtained through the system, or the delay time of the data comparison system for data comparison can be tested through program instructions. The preset time offset value refers to a preset offset value for multiple accumulations of the time difference between the write time and the transmission time. In the first data comparison, the first delay time can be used as the comparison delay time. In the second data comparison, half of the sum of the first delay time, the first time difference between the write time and the transmission time and the preset time offset value can be used as the delay time for the next data comparison. In the third data comparison, the second time difference between the write time and the transmission time is monitored, and one third of the sum of the first delay time, the first time difference, the second time difference and the preset time offset value can be used as the delay time for the next data comparison. It can be seen that in the second comparison, the time difference has a preset number of 1. In the third comparison, the time difference has a preset number of 2.

[0074] In one implementation of this embodiment, due to different transmission systems or different network environments, the data transmission delay is different. When data comparison is performed before the data is transmitted, the result of the data comparison is definitely inconsistent. In order to solve this problem, the system adds data transmission delay monitoring to automatically adjust the delay comparison time. The initial delay time t1 is manually set to be slightly larger than the observed value. The observed value here is the delay value obtained by the system. When comparing the data again, the time difference t2 between the writing time and the transmission time of the data is obtained, and Δt is set to the preset time offset value. (t1+t2+Δt) / 2 is used as the delay time for the next data comparison. When comparing the data next time, the time difference t3 between the writing time and the transmission time of the data is obtained at the same time, and (t1+t2+t3+Δt) / 3 is used as the next comparison delay time. By analogy, the final data comparison delay time t_delay will gradually approach the transmission delay time of the system, and the formula is as follows:

[0075] t_delay=(t n +t n-1 +t n-2 +...+t n-m+1 +Δt) / m

[0076] Where n is the sequence number of the current comparison, and m is the preset number of time differences.

[0077] In this embodiment, the comparison delay time t_delay that affects the consistency of data comparison is determined, providing a more accurate delay time for the next data comparison, which helps to obtain a more accurate data comparison result next time. At the same time, as the number of comparisons increases, the comparison delay time approaches the transmission delay time of the system, and the obtained data comparison result is more reliable.

[0078] In a possible embodiment of the present disclosure, the execution commands and data sources of source data and target data are monitored, and the source data and target data are compared based on a comparison delay time. The inconsistent data is added to a difference data set, which can be further interpreted as monitoring the execution commands of the source data and target data and the data sources corresponding to the execution commands; if the sequence number of the current execution command for the source database is greater than the first sequence number, or the sequence number of the current execution command for the target database is greater than the second sequence number, then after postponing the comparison delay time, the source data and target data are compared according to the data source, and the inconsistent data is added to the difference data set.

[0079] In this embodiment, the execution command of the target data of the source data is monitored, and the data source of the data corresponding to the execution command is monitored. According to the comparison delay time obtained by the last calculation of data comparison, the current data comparison is delayed by the comparison delay time, and then the source data and the target data are compared. Specifically, if the sequence number of the current execution command for the source database is greater than the first sequence number, or the sequence number of the current execution command for the target database is greater than the second sequence number, it means that at least one of the operations of adding, deleting, and modifying the data of the source database or the target database has occurred, then after the comparison delay time is postponed, the source data and the target data are compared according to the data source of the data, and the inconsistent data is added to the difference data set.

[0080] In this embodiment, the comparison delay time is used to delay the data comparison, thereby avoiding the problem of inconsistent data comparison caused by the delay, and improving the reliability and accuracy of the data comparison.

[0081] In a possible embodiment of the present disclosure, according to the data source, the comparison of source data and target data can be further interpreted as for the first data to be compared in the source database, if the data source is the target data, this comparison is skipped; if the data source is the user operation, the first data to be compared is added to the comparison queue; for the second data to be compared in the target database, if the data source is the source data, this comparison is skipped; if the data source is the user operation, the second data to be compared is added to the comparison queue.

[0082] In this embodiment, the first data to be compared refers to the data to be compared in the source data of the source database. The second data to be compared refers to the data to be compared in the target data of the target database. The comparison queue is a queue for temporarily caching the comparison data. If the data source of the first data to be compared is the target data, it means that the first data to be compared already exists in the target database, then the comparison of the first data to be compared is skipped, thereby avoiding repeated comparison of the first data to be compared. If the data source of the first data to be compared is a user operation, then the first data to be compared is added to the comparison queue for temporary storage.

[0083] Similarly, if the data source of the second data to be compared is the source data, it means that the second data to be compared already exists in the source database, so the comparison of the second data to be compared is skipped to avoid repeated comparison of the second data to be compared. If the data source of the second data to be compared is a user operation, the second data to be compared is added to the comparison queue for temporary storage.

[0084] In one implementation of this embodiment, Figure 3 FIG. 1 is a flow chart of performing data comparison based on data sources according to an embodiment of the present disclosure. Figure 3As shown, monitor the target database and source database and pull data.

[0085] For the writing of monitoring data, firstly, the data with source-side monitoring execution ID greater than sourceFullId and the data with target-side monitoring execution ID greater than targetFullId are checked, and then the data source is determined.

[0086] To determine the source of data, the execution statement of the transmission needs to be modified. For the data transmitted between the source and the destination, the source flag sourceId of the data source is added on the basis of the original operation before the transmission. For the statements executed by the user connecting to the database, no modification is made, that is, sourceId is empty. When the sourceId of the execution statement is empty, it is added to the data comparison queue. When sourceId is not empty, it is not added to the data comparison queue. By filtering sourceId, repeated comparison caused by data transmission is avoided.

[0087] In this embodiment, by performing different processing on data from different sources, the efficiency of data comparison is improved and computing resources are saved.

[0088] In a possible embodiment of the present disclosure, after comparing the source data and the target data according to the data source, it can be further interpreted as synchronizing the first data to be compared in the comparison queue to the target database with the first time period, or synchronizing the second data to be compared in the comparison queue to the source database.

[0089] In this embodiment, after comparing the source data and the target data according to the data source, the first data to be compared that is temporarily stored in the comparison queue and is generated by the user operation and stored in the source database is synchronized to the target database with the first time period to complete the synchronization of the first data to be compared. Similarly, the second data to be compared that is temporarily stored in the comparison queue and is generated by the user operation and stored in the target database is synchronized to the source database to complete the synchronization of the second data to be compared. The consistency of data between databases is achieved.

[0090] In a possible embodiment of the present disclosure, performing data comparison on the difference data set at a preset time period can be further interpreted as performing pairwise comparison on the data in the difference data set according to the data source and data content at a preset time period, and deleting the data with consistent comparison from the difference data set.

[0091] In this embodiment, the data in the difference data set are compared regularly based on the data source and data content at a preset time period, and the data with consistent comparison is removed from the difference data set.

[0092] In one implementation of this embodiment, Figure 4FIG. 1 is a flow chart of a data dynamic maintenance comparison result according to an embodiment of the present disclosure. Figure 4 As shown in the figure, during the whole dynamic data comparison process, a data set with inconsistent comparison (difference data set) is recorded. For these inconsistent data sets, it is necessary to compare them again regularly. At the same time, another process is added to compare these inconsistent data again regularly. After comparison, the consistent data is deleted from the difference data set.

[0093] Figure 5 FIG. 1 is a schematic diagram of a data comparison system according to an embodiment of the present disclosure. Figure 5 As shown, the data comparison system includes a comparison module, a monitoring and filtering module, and a transmission delay module.

[0094] For the comparison module, first perform a full comparison, then monitor the data written to the source and target ends, and compare the new data written. The detailed design steps are as follows:

[0095] (1) In the full comparison phase, when obtaining source data, the latest execution statement sourceFullId of the source database is recorded, and the latest execution statement targetFullId of the target end is recorded. Then, the same data is obtained from the target database based on the source data. If the data is not obtained from the target database or the comparison results are inconsistent after the data is obtained, it is added to the data set with different comparison results.

[0096] (2) Monitor the data writing status of the source database and the target database.

[0097] If the id of the execution statement of the source database is greater than sourceFullId, it means that new data has been written, updated, or deleted in the source database. The source of the execution statement is determined. If it is transmitted from the peer (target database), the comparison of the data is skipped. If the data comes from user operations, it is added to the data comparison queue.

[0098] If the id of the execution statement of the target database is greater than targetFullId, it means that the data of the target database is written, updated or deleted. The source of the execution statement is determined. If it is transmitted from the peer (source database), the comparison of the data is skipped. If the data comes from user operation, it is added to the data comparison queue.

[0099] During the whole process, the comparison results were inconsistent, and different difference data sets with different comparison results were added.

[0100] For monitoring and filtering modules, such as Figure 3As shown, the monitoring and filtering module compares data by monitoring the data source, completes the identification and filtering of repeated comparisons in the data comparison, temporarily stores and performs specific processing on the data whose source is user operation, thereby improving the efficiency of data comparison.

[0101] For the transmission delay module, the monitoring and filtering module is used to monitor the delay time of each data comparison. After the delay time is extended, the source data in the monitored source database and the target data in the target database are compared, and the inconsistent data is saved in the difference data set.

[0102] Corresponding to the methods provided in the above-mentioned embodiments, the present disclosure also provides a data comparison device. Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation method of the method is also applicable to the device provided in this embodiment and will not be described in detail in this embodiment.

[0103] Figure 6 FIG. 6 is a schematic diagram of the structure of a data comparison device 600 according to an embodiment of the present disclosure. Figure 6 As shown, the data comparison device comprises:

[0104] An acquisition module 610 is used to acquire source data of a source database, target data of a target database, and data sources of source data or target data, where the data sources include source data, target data, and user operations;

[0105] A full comparison module 620 is used to perform a full comparison of the source data and the target data, and add the inconsistent data to the difference data set;

[0106] Determine delay module 630, used to determine the comparison delay time according to the time difference between the writing time and the transmission time of the source data or the target data;

[0107] A monitoring and filtering module 640 is used to monitor the execution commands and data sources of the source data and the target data, and compare the source data and the target data based on the comparison delay time, and add the inconsistent data to the difference data set;

[0108] The difference comparison module 650 is used to compare the difference data sets at a preset time period.

[0109] In some embodiments, the acquisition module 610 is used to:

[0110] Obtain source data and the first serial number of the source data, where the first serial number is the serial number of the latest execution statement for the source data;

[0111] The target data and the second serial number of the target data are obtained, where the second serial number is the serial number of the latest execution statement for the target data.

[0112] In some embodiments, the delay determination module 630 is used to:

[0113] Obtaining a first delay time, and continuously recording a time difference between a writing time to a source database or a target database and a transmission time;

[0114] A comparison delay time is determined according to the first delay time, a preset number of time differences and a preset time offset value.

[0115] The monitoring and filtering module 640 is used to:

[0116] Monitor the execution commands of source data and target data and the data sources corresponding to the execution commands;

[0117] If the sequence number of the current execution command for the source database is greater than the first sequence number, or the sequence number of the current execution command for the target database is greater than the second sequence number, the source data and the target data are compared according to the data source after the comparison delay time is postponed, and the inconsistent data is added to the difference data set.

[0118] The monitoring and filtering module 640 compares the source data and the target data according to the data source in the following manner:

[0119] For the first data to be compared in the source database, if the data source is the target data, skip this comparison; if the data source is a user operation, add the first data to be compared to the comparison queue;

[0120] For the second data to be compared in the target database, if the data source is source data, this comparison is skipped; if the data source is user operation, the second data to be compared is added to the comparison queue.

[0121] After comparing source data and target data according to the data source, the monitoring and filtering module 640 is also used to:

[0122] The first data to be compared in the comparison queue is synchronized to the target database with the first time period, or,

[0123] The second data to be compared in the comparison queue is synchronized to the source database.

[0124] In some embodiments, the difference comparison module 650 is used to:

[0125] Based on the preset time period, the data in the difference data set are compared pairwise according to the data source and data content, and the data with consistent comparison is deleted from the difference data set.

[0126] In summary, through the data comparison device 600, the source data of the source database, the target data of the target database, and the data source of the source data or the target data are obtained, and the data source includes the source data, the target data and the user operation; the source data and the target data are fully compared, and the inconsistent data is added to the difference data set; the comparison delay time is determined according to the time difference between the writing time and the transmission time of the source data or the target data; the execution commands and data sources of the source data and the target data are monitored, and the source data and the target data are compared based on the comparison delay time, and the inconsistent data is added to the difference data set; the difference data set is compared with a preset time as a period.

[0127] The device solves the problem of low data comparison efficiency during database synchronization, improves data comparison efficiency, and ensures data consistency of the synchronized database.

[0128] In the above embodiments provided by the present disclosure, the methods and devices provided by the embodiments of the present disclosure are introduced. In order to implement the functions in the methods provided by the above embodiments of the present disclosure, the electronic device may include a hardware structure and a software module, and implement the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. One of the above functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0129] Figure 7 is a block diagram of an electronic device 700 for implementing the above data comparison method according to an exemplary embodiment.

[0130] For example, the electronic device 700 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0131] Reference Figure 7 , the electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0132] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0133] The memory 704 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0134] The power supply component 706 provides power to the various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.

[0135] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0136] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0137] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0138] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of components, such as the display and keypad of the electronic device 700, and the sensor assembly 714 can also detect the position change of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and the temperature change of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0139] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0140] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0141] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by a processor 720 of an electronic device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0142] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the data comparison method described in the above embodiments of the present disclosure.

[0143] The embodiments of the present disclosure further provide a computer program product, including a computer program, which executes the data comparison method described in the above embodiments of the present disclosure when a processor is used.

[0144] Figure 8 is a schematic structural diagram of a chip 800 for implementing the above data comparison method according to an exemplary embodiment.

[0145] Reference Figure 8 The chip 800 includes at least one communication interface 801 and a processor 802; the communication interface 801 is used to receive a signal input to the chip 800 or a signal output from the chip 800, and the processor 802 communicates with the communication interface 801 and implements the data comparison method described in the above embodiment through a logic circuit or executing code instructions.

[0146] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0147] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0148] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0149] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0150] It should be understood that the various parts of the embodiments of the present disclosure can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0151] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0152] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0153] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A data comparison method, characterized in that: The method comprises: Acquire source data of a source database, target data of a target database, and data sources of the source data or the target data, wherein the data sources include the source data, the target data, and user operations; Performing a full comparison of the source data and the target data, and adding inconsistent data to a difference data set; Determine a comparison delay time according to a time difference between a writing time and a transmission time of the source data or the target data; monitoring the execution commands of the source data and the target data, and the data sources, and comparing the source data and the target data based on the comparison delay time, and adding inconsistent data to the difference data set; The difference data sets are compared with each other at a preset time period.

2. The method according to claim 1, characterized in that The step of obtaining source data of a source database and target data of a target database includes: Acquire the source data and a first sequence number of the source data, where the first sequence number is a sequence number of a latest execution statement for the source data; The target data and a second sequence number of the target data are obtained, where the second sequence number is a sequence number of a latest execution statement for the target data.

3. The method according to claim 2, characterized in that The determining of the comparison delay time according to the time difference between the writing time and the transmission time of the source data or the target data comprises: Obtaining a first delay time, and continuously recording a time difference between a writing time to the source database or the target database and a transmission time; The comparison delay time is determined according to the first delay time, a preset number of the time differences and a preset time offset value.

4. The method according to claim 2, characterized in that The monitoring of the execution commands of the source data and the target data, the data sources, and comparing the source data and the target data based on the comparison delay time, and adding inconsistent data to the difference data set includes: Monitoring the execution commands of the source data and the target data and the data sources corresponding to the execution commands; If the sequence number of the current execution command for the source database is greater than the first sequence number, or the sequence number of the current execution command for the target database is greater than the second sequence number, then after postponing the comparison delay time, the source data and the target data are compared according to the data source, and the inconsistent data are added to the difference data set.

5. The method according to claim 4, characterized in that The comparing the source data and the target data according to the data source includes: For the first data to be compared in the source database, if the data source is the target data, skip the current comparison; if the data source is the user operation, add the first data to be compared to the comparison queue; For the second data to be compared in the target database, if the data source is the source data, skip this comparison; if the data source is the user operation, add the second data to be compared to the comparison queue.

6. The method according to claim 5, characterized in that After comparing the source data and the target data according to the data source, the method further includes: The first to-be-compared data in the comparison queue is synchronized to the target database with a first time period, or, The second to-be-compared data in the comparison queue is synchronized to the source database.

7. The method according to claim 1, characterized in that The performing data comparison on the difference data set at a preset time period includes: The data in the difference data set are compared in pairs according to the data source and data content at a preset time period, and the data with consistent comparison is deleted from the difference data set.

8. A data comparison device, characterized in that: The device comprises: An acquisition module, used to acquire source data of a source database, target data of a target database, and a data source of the source data or the target data, wherein the data source includes the source data, the target data, and user operations; A full comparison module, used to perform a full comparison of the source data and the target data, and add inconsistent data to a difference data set; A delay determination module, used to determine a comparison delay time according to a time difference between a writing time and a transmission time of the source data or the target data; A monitoring and filtering module, used for monitoring the execution commands of the source data and the target data, the data source, and comparing the source data and the target data based on the comparison delay time, and adding inconsistent data to the difference data set; The difference comparison module is used to compare the difference data sets at a preset time period.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

11. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

12. A chip, characterized in that: It includes at least one processor and a communication interface; the communication interface is used to receive a signal input to the chip or a signal output from the chip, and the processor communicates with the communication interface and implements the method according to any one of claims 1-7 through a logic circuit or executing code instructions.

Images