Data comparison method, computing device and computer storage medium

Abstract

The present invention discloses a data comparison method, computing device, and computer storage medium. The method includes: obtaining first data from a first data source, determining whether there is second data consistent with the first data in a second data source; if not, determining the first data as difference data; obtaining second data from a second data source, and determining whether there is first data consistent with the second data in the first data source; if not, determining the second data as difference data; storing relevant information of the difference data in a delayed comparison queue; when the storage time of the relevant information of the difference data in the delayed comparison queue reaches a preset time, extracting relevant information of the difference data from the delayed comparison queue, and performing data comparison on the data in the first data source and the second data source based on the extracted relevant information of the difference data. The adoption of this solution can save storage resources, reduce the misjudgment rate of abnormal data, and improve data comparison accuracy.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data comparison method, a computing device, and a computer storage medium. Background Art

[0002] With the increase in Internet business, the single computer room architecture can no longer meet the huge service needs, so the multi-computer room architecture (especially the dual-computer room architecture) is increasingly used in Internet business.

[0003] In a multi-computer room architecture, multiple computer rooms can provide services to users, improving service quality and efficiency. Furthermore, if a service outage occurs in one computer room, other rooms can continue to provide services, enhancing service stability. Therefore, in a multi-computer room architecture, business data across multiple computer rooms must generally remain consistent. To ensure consistency across multiple computer rooms, data comparison is often required.

[0004] The existing technology typically extracts data from multiple computer rooms and compares it once, identifying any inconsistent business data as abnormal. However, this data comparison method in the existing technology has a high error rate for abnormal data and poor data comparison accuracy. Furthermore, the existing technology requires additional storage space to store the data to be compared, increasing storage overhead. Summary of the Invention

[0005] In view of the above problems, the present invention is proposed to provide a data comparison method, a computing device and a computer storage medium that overcome the above problems or at least partially solve the above problems.

[0006] According to one aspect of the present invention, there is provided a data comparison method, comprising:

[0007] Obtaining first data from a first data source, and determining whether second data consistent with the first data exists in a second data source; if not, determining the first data as difference data;

[0008] Acquire second data from a second data source, and determine whether there is first data in the first data source that is consistent with the second data; if not, determine the second data as difference data;

[0009] Storing relevant information of the difference data in a delayed comparison queue;

[0010] When the storage time of the relevant information of the difference data in the delay comparison queue reaches a preset time, the relevant information of the difference data is extracted from the delay comparison queue, and data comparison is performed on the data in the first data source and the second data source based on the extracted relevant information of the difference data.

[0011] According to another aspect of the present invention, there is provided a computing device comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0012] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the following operations:

[0013] Obtaining first data from a first data source, and determining whether second data consistent with the first data exists in a second data source; if not, determining the first data as difference data;

[0014] Acquire second data from a second data source, and determine whether there is first data in the first data source that is consistent with the second data; if not, determine the second data as difference data;

[0015] Storing relevant information of the difference data in a delayed comparison queue;

[0016] When the storage time of the relevant information of the difference data in the delay comparison queue reaches a preset time, the relevant information of the difference data is extracted from the delay comparison queue, and data comparison is performed on the data in the first data source and the second data source based on the extracted relevant information of the difference data.

[0017] According to another aspect of the present invention, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned data comparison method.

[0018] According to the data comparison method, computing device and computer storage medium disclosed in the present invention, first data is obtained from a first data source, and it is determined whether there is second data consistent with the first data in the second data source; if not, the first data is determined as difference data; second data is obtained from a second data source, and it is determined whether there is first data consistent with the second data in the first data source; if not, the second data is determined as difference data; relevant information of the difference data is stored in a delayed comparison queue; when the storage time of relevant information of the difference data in the delayed comparison queue reaches a preset time length, relevant information of the difference data is extracted from the delayed comparison queue, and data comparison is performed on the data in the first data source and the second data source based on the extracted relevant information of the difference data. The adoption of this solution can save storage resources, reduce the misjudgment rate of abnormal data, and improve the accuracy of data comparison.

[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0021] Figure 1 A schematic flow chart of a data comparison method according to the first embodiment of the present invention is shown;

[0022] Figure 2 A schematic flow chart of a method for obtaining difference data applied in the first embodiment of the present invention is shown;

[0023] Figure 3 A schematic flow chart of another method for obtaining difference data applied in the first embodiment of the present invention is shown;

[0024] Figure 4 A schematic flow chart of a data comparison method according to a second embodiment of the present invention is shown;

[0025] Figure 5 A schematic structural diagram of a computing device provided according to a fourth embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0027] Example 1

[0028] Figure 1 A flow chart of a data comparison method provided according to the first embodiment of the present invention is shown.

[0029] The data comparison method provided in this embodiment can be applied to a variety of databases, especially to memory databases such as Redis.

[0030] like Figure 1 As shown, the method includes:

[0031] Step S110: obtaining first data from a first data source, and determining whether second data consistent with the first data exists in a second data source; if not, determining the first data as difference data.

[0032] The first data source and the second data source are two different data sources to be tested for data consistency. In actual application scenarios, the first data source and the second data source can be servers or clusters in two different computer rooms, for example, the first data source can be a slave node in the main computer room, and the second data source can be a master node in the hot standby computer room. Alternatively, the first data source and the second data source can be servers or clusters in the same computer room, for example, the first data source can be a master node in the main computer room, and the second data source can be a slave node in the main computer room.

[0033] In the process of detecting data consistency between the first data source and the second data source, the first data to be compared can be obtained from the first data source. The first data can be one or more pieces of data. For any first data, it is determined whether there is second data consistent with the first data in the second data source; if not, the first data is determined as difference data. This step can be used to preliminarily screen out first data from the first data source that is inconsistent with the data in the second data source. In addition, during the execution of this step, the second data in the second data source is not retrieved, so there is no need to allocate a large storage space to store the data to be compared, thereby saving storage resources.

[0034] In an optional way, you can use Figure 2 The method shown in FIG. 1 filters out difference data from the first data source, such as Figure 2 As shown, the method includes the following steps S111-S115.

[0035] S111: Obtain a data key of first data from a first data source.

[0036] In an in-memory database, data is typically stored in a key-value format, with each piece of data having a corresponding data key and data details. The data details include at least one of the following: data type, data length, and data value. Because the data key has a small amount of data and the data key scanning efficiency in an in-memory database is high, this step can quickly obtain the data key of the first data from the first data source. The data key of each piece of first data can be obtained one by one, or the data keys of multiple pieces of first data can be obtained in batches.

[0037] S112, determining whether a data key consistent with the data key of the first data can be found in the second data source; if so, executing step S113, if not, executing step S115.

[0038] After obtaining the data key of the first data, for any first data, a data key that is consistent with the data key of the first data is searched in the second data source, and subsequent steps are executed according to the search result.

[0039] Optionally, in the actual implementation process, due to the large amount of business data, the second data source can be a cluster containing multiple second data nodes, and the second data can be stored in any second data node. In order to be able to quickly find the data key that is consistent with the data key of the first data from the second data source, the second data source can establish a mapping relationship between the data key and the second data node where the data is stored according to the corresponding rules (such as the second operation rule) when storing the second data. Therefore, in this step, the data key of the first data can be operated using the second operation rule in the process of searching for the data key that is consistent with the data key of the first data, and the target second data node corresponding to the data key of the first data is determined according to the operation result, and the data key that is consistent with the data key of the first data is searched in the target second data node.

[0040] S113 , extracting data details of the first data and data details of the second data corresponding to the data key from the first data source and the second data source respectively, and performing a detail comparison between the data details of the first data and the data details of the second data.

[0041] If a data key that matches the data key of the first data can be found in the second data source, the data details of the first data are obtained from the first data source based on the data key, and the data details of the second data corresponding to the data key are obtained from the second data source. The data details of the first data are compared with the data details of the second data.

[0042] Optionally, the data details include multiple types of detail information, such as data type, data length, and data value. Each type of detail information has a corresponding comparison priority, specifically data type > data length > data value. The data details of the first data and the data details of the second data can be obtained according to the comparison priority corresponding to each type of detail information, and the data details of the first data and the data details of the second data can be compared according to the comparison priority corresponding to each type of detail information. For example, for the first data A and the second data B with the same data key, first obtain the data type of the first data A and the data type of the second data B. If the data types are inconsistent, determine that the data details are inconsistent; if the data types are consistent, continue to obtain the data length of the first data A and the data length of the second data B. If the data lengths are inconsistent, determine that the data details are inconsistent; if the data lengths are consistent, continue to obtain the data value of the first data A and the data value of the second data B. If the data lengths are inconsistent, determine that the data details are inconsistent. If the data lengths are consistent, determine that the data details are consistent. This method of obtaining and comparing detailed data step by step in order of priority from high to low can reduce overall memory overhead and avoid affecting the normal business services of the data source.

[0043] Further optionally, in the process of obtaining data values, since the data amounts of data values ​​corresponding to different data vary greatly, in the process of obtaining data values, the method of obtaining the data values ​​may be determined based on the data type and data length. If the data amount of the data value determined based on the data type and data length is greater than a preset threshold, the data value is obtained by batch acquisition, and the batch comparison method is also used in the subsequent data value comparison process; if the data amount of the data value determined based on the data type and data length is less than or equal to the preset threshold, the data value is obtained by sufficient acquisition.

[0044] S114, determining a comparison result of the data details of the first data and the data details of the second data; if they are inconsistent, executing S115.

[0045] If all data details of the first data match all data details of the second data, it is determined that the data details of the first data are consistent with the data details of the second data; if any data detail of the first data and the second data is inconsistent, it is determined that the data details of the first data are inconsistent with the data details of the second data.

[0046] S115: Determine the first data as difference data.

[0047] In one case, if a data key that matches the data key of the first data cannot be found in the second data source, the first data is determined to be difference data. This eliminates the need to obtain the data details of the first and second data for comparison, allowing for quick identification of difference data and saving storage resources.

[0048] In another case, if a data key that is consistent with the data key of the first data can be found in the second data source, but the data details of the first data are inconsistent with the data details of the second data, the first data is determined to be differential data.

[0049] As an optional method of this step, in order to improve the overall data comparison efficiency, different working threads can respectively execute the acquisition of the data key of the first data, the acquisition of the detailed data of the first data and the second data, and the comparison between the data keys and the comparison of the detailed data.

[0050] Step S120: obtaining second data from the second data source, and determining whether there is first data consistent with the second data in the first data source; if not, determining the second data as difference data.

[0051] The execution process of this step can refer to the description of the corresponding part in step S110. Figure 3 The method shown in FIG. 1 filters out difference data from the second data source. Specifically, Figure 3 As shown, the method includes the following steps S121-S125.

[0052] S121: Obtain a data key of second data from a second data source.

[0053] S122, determine whether a data key consistent with the data key of the second data can be found in the first data source; if so, execute step S123, if not, execute step S125.

[0054] Optionally, if the first data source is a cluster containing multiple first data nodes, the data key of the second data is operated using the first operation rule, the target first data node corresponding to the data key of the second data is determined based on the operation result, and the data key consistent with the data key of the second data is searched in the target first data node.

[0055] S123 , extracting data details of the first data and data details of the second data corresponding to the data key from the first data source and the second data source respectively, and performing a detail comparison between the data details of the first data and the data details of the second data.

[0056] If a data key that is consistent with the data key of the second data can be found in the first data source, the data details of the second data are obtained from the second data source based on the data key, and the data details of the first data corresponding to the data key are obtained from the first data source, and the data details of the first data are compared with the data details of the second data.

[0057] S124, determining a comparison result between the data details of the first data and the data details of the second data; if they are inconsistent, executing S125.

[0058] S125: Determine the second data as difference data.

[0059] In one case, if a data key that is consistent with the data key of the second data cannot be found in the first data source, the second data is determined to be difference data.

[0060] In another case, if a data key that is consistent with the data key of the second data can be found in the first data source, but the data details of the first data are inconsistent with the data details of the second data, the second data is determined to be difference data.

[0061] In addition, step S110 and step S120 may be executed concurrently or sequentially in a corresponding order.

[0062] Step S130: storing relevant information of the difference data into a delayed comparison queue.

[0063] After identifying the difference data, this embodiment does not directly determine the difference data as abnormal data, but stores relevant information of the identified difference data in a delayed comparison queue for subsequent re-comparison of the difference data.

[0064] Optionally, the relevant information of the difference data includes a data key, that is, this embodiment does not store the data details of the difference data in the delayed comparison queue, but stores the data key of the difference data. Based on the data key, not only can the subsequent data comparison be completed, but the data key has a small amount of data and is easy to store and retrieve, thereby saving storage resources and improving the overall efficiency of data comparison.

[0065] Further optionally, the relevant information of the difference data also includes a difference category, which can identify which information of the difference data in the first data source and the second data source is different during a comparison process before the difference data is placed in the delayed comparison queue. The difference category includes at least one of the following categories: data key difference, data type difference, data length difference, and data value difference. For example, with respect to step S110, if a data key that is consistent with the data key of the first data cannot be found in the second data source, the first data is determined to be difference data, and the difference type of the difference data is a data key difference; or with respect to step S120, if a data key that is consistent with the data key of the second data cannot be found in the first data source, the second data is determined to be difference data, and the difference type of the difference data is a data key difference; if the first data and the second data corresponding to the same data key can be found, and the data types of the first data and the second data are inconsistent, the difference type of the difference data is determined to be a data type difference; if the data lengths of the first data and the second data are inconsistent, the difference type of the difference data is determined to be a data length difference; if the data values ​​of the first data and the second data are inconsistent, the difference type of the difference data is determined to be a data value difference.

[0066] Step S140: When the storage time of the relevant information of the difference data in the delay comparison queue reaches a preset time, the relevant information of the difference data is extracted from the delay comparison queue, and data comparison is performed on the data in the first data source and the second data source based on the extracted relevant information of the difference data.

[0067] The relevant information of the difference data is stored in the delay comparison queue for a preset period of time and then retrieved again. Based on the relevant information of the difference data extracted from the delay comparison queue, the data in the first data source and the second data source can be compared again to determine whether the difference data is truly abnormal data.

[0068] Optionally, during the data comparison process of this step, for any difference data, if only one of the first data source and the second data source has the data key of the difference data, the comparison result corresponding to the difference data is determined to be a failure; if the data key of the difference data exists in both the first data source and the second data source, the data details of the first data corresponding to the data key of the difference data are obtained from the first data source, and the data details of the second data corresponding to the data key of the difference data are obtained from the second data source, and the data details of the first data are compared with the data details of the second data. If the data details of the first data are inconsistent with the data details of the second data, the comparison result corresponding to the difference data is determined to be a failure. If the data key of the difference data exists in both the first data source and the second data source, and the data details of the first data and the second data corresponding to the data key are consistent, the comparison result corresponding to the difference data is determined to be a success, or if the data key of the difference data does not exist in both the first data source and the second data source, the comparison result corresponding to the difference data is determined to be a success. The comparison process of the data details can refer to the description of the corresponding part in step S110, and this step will not be repeated here.

[0069] Further optionally, if the relevant information of the difference data also includes a difference category, the starting point for data comparison corresponding to the difference data can be determined based on the difference category of the difference data. For example, if the difference category of the difference data is a data length difference, the data length is used as the starting point for data comparison corresponding to the difference data. That is, after obtaining the first data and the second data corresponding to the data key of the difference data, the data lengths of the first data and the second data are first obtained and compared.

[0070] It can be seen that the present embodiment obtains first data from a first data source and determines whether there is second data consistent with the first data in the second data source, and obtains second data from a second data source and determines whether there is first data consistent with the second data in the first data source. This two-way comparison method can completely extract the difference data in the first data and the second data; and in the data comparison process of the present embodiment, there is no need to take all the data to be compared from the data source, so there is no need to open up a large storage space to store the data to be compared, thereby saving storage resources; furthermore, the present embodiment stores the relevant information of the identified difference data in a delayed comparison queue, and after a preset delay, performs data comparison again on the data in the first data source and the second data source based on the relevant information of the difference data, thereby reducing the misjudgment rate of abnormal data and improving the accuracy of data comparison.

[0071] Example 2

[0072] Figure 4 A flow chart of a data comparison method provided according to the second embodiment of the present invention is shown.

[0073] The data comparison method provided in this embodiment is a further optimization of the data comparison method in the first embodiment.

[0074] like Figure 4 As shown, the method includes:

[0075] Step S410: obtaining first data from a first data source, and determining whether second data consistent with the first data exists in a second data source; if not, determining the first data as difference data.

[0076] Step S420: Obtain second data from the second data source, and determine whether there is first data consistent with the second data in the first data source; if not, determine the second data as difference data.

[0077] The specific implementation process of steps S410 and S420 can refer to the description of the corresponding parts in embodiment 1, and will not be repeated in this embodiment.

[0078] Step S430: storing relevant information of the difference data into a delayed comparison queue.

[0079] If multiple difference data are identified, relevant information of the multiple difference data are stored in the delayed comparison queue in the order in which the difference data are identified.

[0080] Optionally, to facilitate management of differential data, information related to differential data identified from the first data source can be stored in a first delay comparison queue, and information related to differential data identified from the second data source can be stored in a second delay comparison queue. Thus, when a data error occurs and data comparison is no longer required, the corresponding delay comparison queue can be directly deleted.

[0081] Step S440: When the storage time of the relevant information of the difference data in the delay comparison queue reaches a preset time, the relevant information of the difference data is extracted from the delay comparison queue, and data comparison is performed on the data in the first data source and the second data source based on the extracted relevant information of the difference data.

[0082] In the actual implementation process, the data in the delay comparison queue can be consumed by the corresponding working thread. When the working thread determines that the single storage duration of the relevant information of the difference data at the head of the delay comparison queue has reached a preset duration, the relevant information of the difference data at the head of the queue is taken out from the delay comparison queue, and then the data in the first data source and the second data source are compared based on the relevant information of the extracted difference data. Among them, the single storage duration of the relevant information of the difference data specifically refers to the duration between the time T1 when the relevant information of the difference data was last added to the delay comparison queue and the current time T2.

[0083] In one optional manner, the preset duration can be a fixed value, such as 1 minute, etc. In another optional manner, the preset duration can be determined based on the number of times the relevant information of the difference data is stored in the delay comparison queue. Specifically, the relevant information of the same difference data has different storage times in the delay comparison queue, and the corresponding preset duration is different. For example, when the relevant information of the difference data is stored once in the delay comparison queue, the corresponding preset duration is 1 minute; when the relevant information of the difference data is stored twice in the delay comparison queue, the corresponding preset duration is 3 minutes, and so on. Therefore, in the actual implementation process, before extracting the relevant information of the difference data from the delay comparison queue, the preset duration is first determined based on the number of times the relevant information of the difference data is stored in the delay comparison queue.

[0084] Step S450: Determine the data comparison result corresponding to the difference data; if the data comparison result is failure, execute step S460.

[0085] If it is determined that the data comparison result corresponding to the difference data is a failure, it is necessary to further determine whether the difference data is truly abnormal data through subsequent steps S460; if it is determined that the data comparison result corresponding to the difference data is a success, it indicates that the difference data is not truly abnormal data, and the relevant information of the difference data will no longer be stored in the delayed comparison queue.

[0086] Step S460: Determine whether the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than a preset number threshold; if so, execute step S470; if not, execute step S480.

[0087] If the data comparison result corresponding to the difference data fails, it indicates that the difference data is still suspected abnormal data. In order to further determine whether the difference data is truly abnormal data, the relevant information of the difference data can be stored again in the delayed comparison queue.

[0088] Among them, in order to prevent the real abnormal data from being repeatedly compared, this embodiment further determines whether the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than a preset number threshold after determining that the re-data comparison result corresponding to the difference data is a failure. If the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than the preset number threshold, it indicates that the difference data is still suspected abnormal data, and step S470 is further executed to re-store the relevant information of the difference data into the delayed comparison queue; if the number of times the relevant information of the difference data is stored in the delayed comparison queue is greater than or equal to the preset number threshold, it indicates that the difference data is abnormal data, and step S480 is further executed to record the relevant information of the difference data in the abnormal data list.

[0089] Step S470: Restore the relevant information of the extracted difference data into the delayed comparison queue.

[0090] If the re-data comparison result corresponding to the difference data is a failure, and the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than the preset threshold, it is still impossible to determine whether the difference data is truly abnormal data, so the relevant information of the extracted difference data can be re-stored in the delayed comparison queue, and step S440 and subsequent steps can be further executed.

[0091] Step S480: Record the relevant information of the difference data in the abnormal data list.

[0092] If the data comparison result corresponding to the difference data is failed, and the number of times the relevant information of the difference data is stored in the delayed comparison queue is greater than or equal to the preset number threshold, the difference data is determined to be abnormal data, and then the relevant information of the difference data is recorded in the abnormal data list.

[0093] As can be seen, in this embodiment, when the data comparison result corresponding to the difference data fails, the method further determines the number of times the relevant information of the difference data has been stored in the delayed comparison queue. If the number of storage times is small, the relevant information of the difference data is placed back into the delayed comparison queue. When the data comparison result corresponding to the difference data fails again, and the number of times the relevant information of the difference data has been stored in the delayed comparison queue is large, the difference data is determined to be abnormal data. This method can accurately identify true abnormal data on the one hand, and prevent true abnormal data from being repeatedly compared on the other hand, thereby saving system resources.

[0094] Example 3

[0095] According to a third embodiment of the present invention, a non-volatile computer storage medium is provided, wherein the computer storage medium stores at least one executable instruction.

[0096] The executable instructions can specifically be used to enable the processor to perform the following operations: obtain first data from a first data source, and determine whether second data consistent with the first data exists in a second data source; if not, determine the first data as difference data; obtain second data from a second data source, and determine whether first data consistent with the second data exists in the first data source; if not, determine the second data as difference data; store relevant information of the difference data in a delay comparison queue; when the storage time of relevant information of the difference data in the delay comparison queue reaches a preset time, extract relevant information of the difference data from the delay comparison queue, and perform data comparison on the data in the first data source and the second data source based on the extracted relevant information of the difference data.

[0097] In an optional embodiment, the executable instructions cause the processor to perform the following operations: obtain a data key of the first data from a first data source, and search for a data key consistent with the data key of the first data in a second data source; if a data key consistent with the data key of the first data cannot be found in the second data source, the first data is determined to be difference data; if a data key consistent with the data key of the first data can be found in the second data source, the data details of the first data are obtained from the first data source based on the data key, and the data details of the second data corresponding to the data key are obtained from the second data source, and the data details of the first data are compared with the data details of the second data; if the data details of the first data are inconsistent with the data details of the second data, the first data is determined to be difference data.

[0098] In an optional embodiment, the executable instructions cause the processor to perform the following operations: obtain a data key of the second data from the second data source, and search for a data key consistent with the data key of the second data in the first data source; if a data key consistent with the data key of the second data cannot be found in the first data source, the second data is determined to be difference data; if a data key consistent with the data key of the second data can be found in the first data source, the data details of the second data are obtained from the second data source based on the data key, and the data details of the first data corresponding to the data key are obtained from the first data source, and the data details of the first data are compared with the data details of the second data; if the data details of the first data are inconsistent with the data details of the second data, the second data is determined to be difference data.

[0099] In an optional embodiment, the executable instructions cause the processor to perform the following operations: the second data source is a cluster comprising multiple second data nodes; the data key of the first data is operated using a second operation rule, and the target second data node corresponding to the data key of the first data is determined based on the operation result, and a data key consistent with the data key of the first data is searched in the target second data node.

[0100] In an optional embodiment, the executable instructions cause the processor to perform the following operations: the first data source is a cluster comprising multiple first data nodes; the data key of the second data is operated using a first operation rule, and the target first data node corresponding to the data key of the second data is determined based on the operation result, and a data key consistent with the data key of the second data is searched in the target first data node.

[0101] In an optional embodiment, the executable instructions cause the processor to perform the following operations: the data details include multiple types of detail information, and each type of detail information has a corresponding comparison priority; according to the comparison priority corresponding to each type of detail information, the data details of the first data and the data details of the second data are compared in detail.

[0102] In an optional embodiment, the executable instruction causes the processor to perform the following operations: after performing data comparison on the data in the first data source and the second data source based on the relevant information of the extracted difference data, if the data comparison result corresponding to the difference data is failure, the relevant information of the extracted difference data is re-stored in the delayed comparison queue.

[0103] In an optional embodiment, the executable instruction causes the processor to perform the following operations: if the data comparison result corresponding to the difference data is a failure, then determine whether the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than a preset number threshold; if the number of storages is less than the preset number threshold, then re-store the relevant information of the difference data into the delayed comparison queue.

[0104] In an optional embodiment, the executable instruction causes the processor to perform the following operations: after determining whether the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than a preset number threshold, if the number of times the storage is greater than or equal to the preset number threshold, the relevant information of the difference data is recorded in the abnormal data list.

[0105] In an optional embodiment, the executable instruction causes the processor to perform the following operations: when the storage time of the relevant information of the difference data in the delay comparison queue reaches a preset time, before extracting the relevant information of the difference data from the delay comparison queue, determine the preset time based on the number of times the relevant information of the difference data is stored in the delay comparison queue.

[0106] In an optional embodiment, the executable instructions cause the processor to perform the following operations: the relevant information includes a data key; if only one of the first data source and the second data source has the data key of the difference data, it is determined that the comparison result corresponding to the difference data is a failure; if the data key of the difference data exists in both the first data source and the second data source, the data details of the first data corresponding to the data key of the difference data are obtained from the first data source, and the data details of the second data corresponding to the data key of the difference data are obtained from the second data source, and the data details of the first data are compared with the data details of the second data; if the data details of the first data are inconsistent with the data details of the second data, it is determined that the comparison result corresponding to the difference data is a failure.

[0107] In an optional embodiment, the executable instructions cause the processor to perform the following operations: the relevant information also includes a difference category; the difference category includes at least one of the following categories: data key difference, data type difference, data length difference, and data value difference; based on the difference category of the difference data, determine the starting point of the data comparison corresponding to the difference data.

[0108] It can be seen that this embodiment obtains the first comparison data set and the second comparison data set from the first data source and the second data source respectively, and further identifies the difference data between the first comparison data set and the second comparison data set, and stores the relevant information of the identified difference data in the delayed comparison queue. After a preset delay period, the data in the first data source and the second data source are compared again based on the relevant information of the difference data, thereby reducing the misjudgment rate of abnormal data and improving the data comparison accuracy.

[0109] Figure 5 A schematic structural diagram of a computing device provided according to a fifth embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0110] like Figure 5As shown, the computing device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508. The processor 502, communications interface 504, and memory 506 communicate with each other via the communications bus 508. The communications interface 504 is used to communicate with other devices, such as clients or other server network elements. The processor 502 is used to execute a program 510, which may specifically perform the relevant steps in the above-described method embodiments. Specifically, the program 510 may include program code, which includes computer operating instructions. The processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be of the same type, such as one or more CPUs, or may be of different types, such as one or more CPUs and one or more ASICs. The memory 506 is used to store the program 510. The memory 506 may include high-speed RAM memory or non-volatile memory, such as at least one disk drive.

[0111] Program 510 can be specifically used to enable the processor 502 to perform the following operations: obtain first data from a first data source, and determine whether second data consistent with the first data exists in the second data source; if not, determine the first data as difference data; obtain second data from a second data source, and determine whether first data consistent with the second data exists in the first data source; if not, determine the second data as difference data; store relevant information of the difference data in a delay comparison queue; when the storage time of relevant information of the difference data in the delay comparison queue reaches a preset time, extract relevant information of the difference data from the delay comparison queue, and perform data comparison on the data in the first data source and the second data source based on the extracted relevant information of the difference data.

[0112] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: obtain the data key of the first data from the first data source, and search for a data key consistent with the data key of the first data in the second data source; if the data key consistent with the data key of the first data cannot be found in the second data source, the first data is determined to be difference data; if the data key consistent with the data key of the first data can be found in the second data source, the data details of the first data are obtained from the first data source based on the data key, and the data details of the second data corresponding to the data key are obtained from the second data source, and the data details of the first data are compared with the data details of the second data; if the data details of the first data are inconsistent with the data details of the second data, the first data is determined to be difference data.

[0113] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: obtain the data key of the second data from the second data source, and search for a data key consistent with the data key of the second data in the first data source; if the data key consistent with the data key of the second data cannot be found in the first data source, the second data is determined to be difference data; if the data key consistent with the data key of the second data can be found in the first data source, the data details of the second data are obtained from the second data source based on the data key, and the data details of the first data corresponding to the data key are obtained from the first data source, and the data details of the first data are compared with the data details of the second data; if the data details of the first data are inconsistent with the data details of the second data, the second data is determined to be difference data.

[0114] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: the second data source is a cluster including multiple second data nodes; the data key of the first data is operated using a second operation rule, and the target second data node corresponding to the data key of the first data is determined according to the operation result, and a data key consistent with the data key of the first data is searched in the target second data node.

[0115] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: the first data source is a cluster including multiple first data nodes; the data key of the second data is operated using a first operation rule, and the target first data node corresponding to the data key of the second data is determined according to the operation result, and a data key consistent with the data key of the second data is searched in the target first data node.

[0116] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: the data details include multiple types of detail information, and each type of detail information has a corresponding comparison priority; according to the comparison priority corresponding to each type of detail information, the data details of the first data and the data details of the second data are compared in detail.

[0117] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: after performing a data comparison on the data in the first data source and the second data source based on the relevant information of the extracted difference data, if the data comparison result corresponding to the difference data is a failure, the relevant information of the extracted difference data is re-stored in the delayed comparison queue.

[0118] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: if the data comparison result corresponding to the difference data is a failure, then determine whether the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than a preset number threshold; if the number of storages is less than the preset number threshold, then the relevant information of the difference data is re-stored in the delayed comparison queue.

[0119] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: after determining whether the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than a preset number threshold, if the number of storages is greater than or equal to the preset number threshold, the relevant information of the difference data is recorded in the abnormal data list.

[0120] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: when the storage time of the relevant information of the difference data in the delay comparison queue reaches a preset time, before the relevant information of the difference data is extracted from the delay comparison queue, the preset time is determined based on the number of times the relevant information of the difference data is stored in the delay comparison queue.

[0121] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: the relevant information includes a data key; if only one of the first data source and the second data source has the data key of the difference data, then the comparison result corresponding to the difference data is determined to be a failure; if the data key of the difference data exists in both the first data source and the second data source, the data details of the first data corresponding to the data key of the difference data are obtained from the first data source, and the data details of the second data corresponding to the data key of the difference data are obtained from the second data source, and the data details of the first data are compared with the data details of the second data; if the data details of the first data are inconsistent with the data details of the second data, then the comparison result corresponding to the difference data is determined to be a failure.

[0122] In an optional embodiment, the program 510 can be specifically used to enable the processor 502 to perform the following operations: the relevant information also includes a difference category; the difference category includes at least one of the following categories: data key difference, data type difference, data length difference, and data value difference; according to the difference category of the difference data, determine the starting point of the data comparison corresponding to the difference data.

[0123] It can be seen that this embodiment obtains the first comparison data set and the second comparison data set from the first data source and the second data source respectively, and further identifies the difference data between the first comparison data set and the second comparison data set, and stores the relevant information of the identified difference data in the delayed comparison queue. After a preset delay period, the data in the first data source and the second data source are compared again based on the relevant information of the difference data, thereby reducing the misjudgment rate of abnormal data and improving the data comparison accuracy.

[0124] The algorithm or demonstration provided herein are not inherently relevant to any particular computer, virtual system or other equipment. Various general-purpose systems may also be used together with the teachings based on this. According to the above description, it is apparent that the structure required for constructing this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0125] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0126] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0127] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0128] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0129] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components according to an embodiment of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing a part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0130] It should be noted that the above embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by a person skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

Claims

1. A data comparison method, comprising: Obtaining first data from a first data source, and determining whether second data consistent with the first data exists in a second data source; If not, determining the first data as difference data; Acquire second data from a second data source, and determine whether there is first data consistent with the second data in the first data source; If not, the second data is determined to be difference data; wherein, a data key of the second data is obtained from the second data source, and a data key that is consistent with the data key of the second data is searched in the first data source; if a data key that is consistent with the data key of the second data can be found in the first data source, data details of the second data are obtained from the second data source based on the data key, the data details including multiple types of detail information, each type of detail information having a corresponding comparison priority; data details of the first data corresponding to the data key are obtained from the first data source, and the data details of the first data are compared with the data details of the second data according to the comparison priority corresponding to each type of detail information; the comparison priority is data type > data length > data value; storing relevant information of the difference data in a delayed comparison queue; the relevant information includes a difference category, and the difference category includes at least one of a data key difference, a data type difference, a data length difference, and a data value difference; When the storage time of the relevant information of the difference data in the delay comparison queue reaches a preset time, the relevant information of the difference data is extracted from the delay comparison queue; according to the difference category of the difference data, the starting point of the data comparison corresponding to the difference data is determined, and data comparison is performed on the data in the first data source and the second data source based on the relevant information of the extracted difference data; wherein, the preset time is determined according to the number of times the relevant information of the difference data is stored in the delay comparison queue, and if the relevant information of the same difference data is stored in the delay comparison queue for different times, the corresponding preset time is different.

2. The method according to claim 1, wherein The first data is obtained from the first data source, and it is determined whether the second data that is consistent with the first data exists in the second data source; If not, determining the first data as difference data further includes: Obtaining a data key of first data from a first data source, and searching a second data source for a data key that is consistent with the data key of the first data; If a data key that is consistent with the data key of the first data cannot be found in the second data source, the first data is determined to be difference data; If a data key consistent with the data key of the first data can be found in the second data source, the data details of the first data are obtained from the first data source based on the data key, and the data details of the second data corresponding to the data key are obtained from the second data source. The data details of the first data are compared with the data details of the second data. If the data details of the first data are inconsistent with the data details of the second data, the first data is determined to be difference data.

3. The method according to claim 1, wherein acquiring second data from the second data source, and determining whether there is first data consistent with the second data in the first data source; If not, determining the second data as difference data further includes: If a data key that is consistent with the data key of the second data cannot be found in the first data source, the second data is determined to be difference data; If the data details of the first data are inconsistent with the data details of the second data, the second data is determined to be difference data.

4. The method according to claim 2, wherein: The second data source is a cluster comprising a plurality of second data nodes; Then searching the second data source for a data key that is consistent with the data key of the first data further includes: The data key of the first data is operated using a second operation rule, a target second data node corresponding to the data key of the first data is determined based on the operation result, and a data key consistent with the data key of the first data is searched in the target second data node.

5. The method according to claim 3, wherein: The first data source is a cluster comprising a plurality of first data nodes; Then searching the first data source for a data key that is consistent with the data key of the second data further includes: The data key of the second data is operated using the first operation rule, the target first data node corresponding to the data key of the second data is determined according to the operation result, and the data key consistent with the data key of the second data is searched in the target first data node.

6. The method according to any one of claims 1 to 4, wherein After comparing the data in the first data source and the data in the second data source based on the relevant information of the extracted difference data, the method further includes: If the data comparison result corresponding to the difference data is a failure, the relevant information of the extracted difference data is stored again in the delayed comparison queue.

7. The method according to claim 6, wherein: If the data comparison result corresponding to the difference data is a failure, re-storing the relevant information of the extracted difference data into the delayed comparison queue further includes: If the data comparison result corresponding to the difference data is failure, determining whether the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than a preset number threshold; If the storage times are less than the preset times threshold, the relevant information of the difference data is stored again in the delayed comparison queue.

8. The method according to claim 7, wherein: After determining whether the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than a preset number threshold, the method further includes: If the storage times are greater than or equal to a preset times threshold, the relevant information of the difference data is recorded in an abnormal data list.

9. The method according to claim 6, wherein: The relevant information includes a data key; The comparing the data in the first data source and the data in the second data source based on the extracted relevant information of the difference data further includes: If only one of the first data source and the second data source has the data key of the difference data, determining that the comparison result corresponding to the difference data is failure; If the data key of the difference data exists in both the first data source and the second data source, the data details of the first data corresponding to the data key of the difference data are obtained from the first data source, and the data details of the second data corresponding to the data key of the difference data are obtained from the second data source. The data details of the first data are compared with the data details of the second data. If the data details of the first data are inconsistent with the data details of the second data, it is determined that the comparison result corresponding to the difference data is failed.

10. A computing device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the following operations: Obtaining first data from a first data source, and determining whether second data consistent with the first data exists in a second data source; if not, determining the first data as difference data; Acquire second data from a second data source, and determine whether there is first data consistent with the second data in the first data source; If not, the second data is determined to be difference data; wherein, a data key of the second data is obtained from the second data source, and a data key that is consistent with the data key of the second data is searched in the first data source; if a data key that is consistent with the data key of the second data can be found in the first data source, data details of the second data are obtained from the second data source based on the data key, the data details including multiple types of detail information, each type of detail information having a corresponding comparison priority; data details of the first data corresponding to the data key are obtained from the first data source, and the data details of the first data are compared with the data details of the second data according to the comparison priority corresponding to each type of detail information; the comparison priority is data type > data length > data value; storing relevant information of the difference data in a delayed comparison queue; the relevant information includes a difference category, and the difference category includes at least one of a data key difference, a data type difference, a data length difference, and a data value difference; When the storage time of the relevant information of the difference data in the delay comparison queue reaches a preset time, the relevant information of the difference data is extracted from the delay comparison queue; according to the difference category of the difference data, the starting point of the data comparison corresponding to the difference data is determined, and data comparison is performed on the data in the first data source and the second data source based on the relevant information of the extracted difference data; wherein, the preset time is determined according to the number of times the relevant information of the difference data is stored in the delay comparison queue, and if the relevant information of the same difference data is stored in the delay comparison queue for different times, the corresponding preset time is different.

11. The computing device of claim 10, wherein: The executable instructions cause the processor to perform the following operations: Obtaining a data key of first data from a first data source, and searching a second data source for a data key that is consistent with the data key of the first data; If a data key that is consistent with the data key of the first data cannot be found in the second data source, the first data is determined to be difference data; If a data key consistent with the data key of the first data can be found in the second data source, the data details of the first data are obtained from the first data source based on the data key, and the data details of the second data corresponding to the data key are obtained from the second data source. The data details of the first data are compared with the data details of the second data. If the data details of the first data are inconsistent with the data details of the second data, the first data is determined to be difference data.

12. The computing device of claim 10, wherein: The executable instructions cause the processor to perform the following operations: If a data key that is consistent with the data key of the second data cannot be found in the first data source, the second data is determined to be difference data; If the data details of the first data are inconsistent with the data details of the second data, the second data is determined to be difference data.

13. The computing device of claim 11, wherein: The executable instructions cause the processor to perform the following operations: The second data source is a cluster comprising a plurality of second data nodes; The data key of the first data is operated using a second operation rule, a target second data node corresponding to the data key of the first data is determined based on the operation result, and a data key consistent with the data key of the first data is searched in the target second data node.

14. The computing device of claim 12, wherein: The executable instructions cause the processor to perform the following operations: The first data source is a cluster comprising a plurality of first data nodes; The data key of the second data is operated using the first operation rule, the target first data node corresponding to the data key of the second data is determined according to the operation result, and the data key consistent with the data key of the second data is searched in the target first data node.

15. The computing device according to any one of claims 10 to 13, wherein: The executable instructions cause the processor to perform the following operations: After comparing the data in the first data source and the second data source based on the relevant information of the extracted difference data, if the data comparison result corresponding to the difference data is failure, the relevant information of the extracted difference data is stored again in the delayed comparison queue.

16. The computing device of claim 15, wherein: The executable instructions cause the processor to perform the following operations: If the data comparison result corresponding to the difference data is failure, determining whether the number of times the relevant information of the difference data is stored in the delayed comparison queue is less than a preset number threshold; If the storage times are less than the preset times threshold, the relevant information of the difference data is stored again in the delayed comparison queue.

17. The computing device of claim 16, wherein: The executable instructions cause the processor to perform the following operations: After determining whether the number of times the relevant information of the difference data is stored in the delay comparison queue is less than a preset number threshold, if the number of times is greater than or equal to the preset number threshold, the relevant information of the difference data is recorded in the abnormal data list.

18. The computing device of claim 15, wherein: The executable instructions cause the processor to perform the following operations: The relevant information includes a data key; If only one of the first data source and the second data source has the data key of the difference data, determining that the comparison result corresponding to the difference data is failure; If the data key of the difference data exists in both the first data source and the second data source, the data details of the first data corresponding to the data key of the difference data are obtained from the first data source, and the data details of the second data corresponding to the data key of the difference data are obtained from the second data source. The data details of the first data are compared with the data details of the second data. If the data details of the first data are inconsistent with the data details of the second data, it is determined that the comparison result corresponding to the difference data is failed.

19. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and wherein the executable instruction enables a processor to execute an operation corresponding to the data comparison method according to any one of claims 1 to 9.

Images