Data comparison method and device, electronic equipment, medium and program product

Abstract

The invention provides a data comparison method and device, electronic equipment, a medium and a program product, and can be applied to the field of data processing. The method comprises the following steps: acquiring a first data identifier of each piece of streaming data transmitted by a source end data system in a cache region; based on a plurality of threads and each first data identifier, concurrently obtaining to-be-compared data matched with each piece of streaming data in a target data system, the first data identifier of the streaming data being the same as a second data identifier of the to-be-compared data matched with the streaming data; and comparing each piece of streaming data with the to-be-compared data matched with the streaming data. According to the invention, the data comparison efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data comparison method, device, electronic device, medium and program product. Background Art

[0002] In the merchant online query service, the merchant information needs to be synchronized from the network access platform to different online systems, so that the same data will be stored in different systems. Due to the large number of nodes and long links in the data collection, transmission, storage and processing process, data inconsistency may occur during the data synchronization process, which seriously affects the credibility of the data and further affects the downstream system business. In this regard, it is necessary to compare the data with the same data identifier in different systems to ensure the consistency of the same data in different systems.

[0003] In the exemplary technology, the data in the two data systems are downloaded offline, and after the data download is completed, the data in the two data systems are compared.

[0004] However, the data system stores a large amount of data, and it takes a long time to download the data offline. In addition, a large number of data comparisons are required after the data download is completed, resulting in low data comparison efficiency. Summary of the invention

[0005] The present application provides a data comparison method, device, electronic device, medium and program product, which solve the problem of low data comparison efficiency.

[0006] In a first aspect, the present application provides a data comparison method, comprising:

[0007] Obtaining a first data identifier of each stream data transmitted by the source end data system in the buffer area;

[0008] Based on multiple threads and each of the first data identifiers, in the target data system, concurrently obtaining the data to be compared that matches each of the streaming data, the first data identifier of the streaming data being the same as the second data identifier of the data to be compared that matches the streaming data;

[0009] Each of the streaming data is compared with the data to be compared that matches the streaming data.

[0010] In some embodiments, based on multiple threads and each of the first data identifiers, concurrently acquiring the data to be compared that matches each of the streaming data in the target data system includes:

[0011] Acquire a target parameter, wherein the target parameter includes at least one of the amount of current cached data in the cache area and the remaining computing resources of the data comparison device;

[0012] Determine the target number of threads for data comparison according to the target parameter;

[0013] Based on the target number of threads and each of the first data identifiers, in the target data system, the data to be compared that matches each of the streaming data is concurrently acquired.

[0014] In some embodiments, the target parameter includes the amount of cached data, and determining the target number of threads for data comparison according to the target parameter includes:

[0015] In response to the data volume being greater than or equal to a preset threshold, closing some threads, and determining a target number according to the number of threads that are not closed;

[0016] In response to the data volume being less than a preset threshold, a plurality of threads are created, and a target number is determined according to the number of current threads and the number of created threads.

[0017] In some embodiments, comparing each of the streaming data with the data to be compared that matches the streaming data includes:

[0018] Obtaining a data deformation strategy, wherein the data deformation strategy is used to convert the data into a set format;

[0019] According to the data deformation strategy, each of the streaming data and each of the to-be-compared data is deformed to obtain each of the first data and each of the second data, wherein the first data is used to indicate the deformed streaming data, and the second data is used to indicate the deformed to-be-compared data;

[0020] Each of the first data is compared with the second data matching the first data.

[0021] In some embodiments, comparing each of the streaming data with the data to be compared that matches the streaming data includes:

[0022] Get data comparison scenario;

[0023] According to the comparison method associated with the data comparison scenario, each of the streaming data is compared with the data to be compared that matches the streaming data.

[0024] In some embodiments, comparing each of the streaming data with the data to be compared that matches the streaming data according to the comparison method associated with the data comparison scenario includes:

[0025] In response to the data comparison scenario being a difference record scenario, determining that the comparison mode is a summary comparison mode;

[0026] Determine, according to the summary comparison method, a first summary feature value of each of the streaming data and a second summary feature value of each of the to-be-compared data;

[0027] The first summary feature value of the streaming data is compared with the second summary feature value of the data to be compared that matches the streaming data.

[0028] In some embodiments, comparing each of the streaming data with the data to be compared that matches the streaming data according to the comparison method associated with the data comparison scenario includes:

[0029] In response to the data comparison scenario being a custom scenario, determining that the comparison mode is a script comparison mode;

[0030] Obtain a target script associated with the script comparison method;

[0031] Each of the streaming data and each of the data to be compared is input into a target script to compare the streaming data with the data to be compared that matches the streaming data.

[0032] In some embodiments, the step of obtaining a first data identifier of each stream data transmitted by the source data system in the buffer area includes:

[0033] In the buffer area, each stream data updated within the first time window transmitted by the source data system is obtained;

[0034] The data of the source data system compared by the data comparison device last time is located in a second time window, and there is an overlapping time window between the second time window and the first time window.

[0035] In some embodiments, after comparing each of the streaming data with the data to be compared that matches the streaming data, the method further includes:

[0036] Obtaining difference data obtained after comparing each of the streaming data with the matching data to be compared;

[0037] Determining a difference type of each of the difference data, and determining a difference processing strategy according to the difference type and the type of the target data system;

[0038] According to the difference processing strategy, the data to be compared corresponding to the difference data in the target data system is processed.

[0039] In a second aspect, the present application provides a data comparison device, comprising:

[0040] A first acquisition module, used to acquire a first data identifier of each stream data transmitted by the source end data system in the buffer area;

[0041] A second acquisition module is used for concurrently acquiring, in a target data system, data to be compared that matches each of the streaming data based on multiple threads and each of the first data identifiers, wherein the first data identifier of the streaming data is the same as the second data identifier of the data to be compared that matches the streaming data;

[0042] The comparison module is used to compare each of the streaming data with the data to be compared that matches the streaming data.

[0043] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory and a communication interface communicatively connected to the processor;

[0044] The communication interface is used to communicate with other communication devices;

[0045] The memory is used to store computer-executable instructions;

[0046] The processor is used to execute the computer-executable instructions stored in the memory to implement the data comparison method provided in the first aspect.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the data comparison method provided in the first aspect is implemented.

[0048] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data comparison method provided in the first aspect.

[0049] The data comparison method, device, electronic device, medium and program product provided by the present application obtain the first data identifier of each streaming data transmitted by the source data system from the cache area, and concurrently obtain the data to be compared matched by each streaming data in the target data system based on multiple threads and each first data identifier, the second data identifier of the data to be compared is the same as the first data identifier, and then compare each streaming data with the data to be compared that matches the streaming data. In the present application, the data comparison device obtains the streaming data from the cache area and compares it with the data in the target data system. There is no need to download all the data from the source data system and the target data system before comparing, which reduces the data comparison time. In addition, by concurrently obtaining the data to be compared of the target data system through multiple threads, the data comparison time is further reduced and the data comparison efficiency is improved. In addition, by obtaining data from the cache area, there is no need to obtain the data of the source data system from the disk, which avoids the disk's read and write rate affecting the data comparison rate, and further improves the data comparison efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] Figure 1 This is a schematic diagram of the scenario involved in the data comparison method of this application;

[0052] Figure 2 The following is a schematic diagram of the steps of the data comparison method in the embodiment of the present application: Figure 1 ;

[0053] Figure 3 The following is a schematic diagram of the steps of the data comparison method in the embodiment of the present application: Figure 2 ;

[0054] Figure 4 The following is a schematic diagram of the steps of the data comparison method in the embodiment of the present application: Figure 3 ;

[0055] Figure 5 The following is a schematic diagram of the steps of the data comparison method in the embodiment of the present application: Figure 4 ;

[0056] Figure 6 The following is a schematic diagram of the steps of the data comparison method in the embodiment of the present application: Figure 5 ;

[0057] Figure 7 A schematic diagram of a program module of a data comparison device provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0059] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. In addition, although the disclosure in the present application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation method separately.

[0061] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0062] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0063] The term "module" used in the embodiments of the present application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware and / or software code that can perform the functions associated with the element.

[0064] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0065] In the merchant online query service, the merchant information needs to be synchronized from the network access platform to different online systems, so that the same data will be stored in different systems. Due to the large number of nodes and long links in the data collection, transmission, storage and processing process, data inconsistency may occur during the data synchronization process, which seriously affects the credibility of the data and further affects the downstream system business. In this regard, it is necessary to compare the data with the same data identifier in different systems to ensure the consistency of the same data in different systems.

[0066] In the exemplary technology, the data in the two data systems are downloaded offline, and after the data download is completed, the data in the two data systems are compared.

[0067] The inventors of the present application have discovered that the data system stores a large amount of data, and it takes a long time to download the data offline. In addition, a large number of data comparisons are required after the data download is completed, resulting in low data comparison efficiency.

[0068] The inventor of the present application therefore conceived of obtaining the first data identifier of each streaming data transmitted by the source data system from the cache area, and based on multiple threads and each first data identifier, concurrently obtaining the data to be compared that matches each streaming data in the target data system, the second data identifier of the data to be compared is the same as the first data identifier, and then comparing each streaming data with the data to be compared that matches the streaming data. In the present application, the data comparison device obtains streaming data from the cache area and compares it with the data in the target data system, without having to download all the data from the source data system and the target data system before comparing, thereby reducing the data comparison time. In addition, by concurrently obtaining the data to be compared of the target data system through multiple threads, the data comparison time is further reduced and the data comparison efficiency is improved. In addition, by obtaining data from the cache area, there is no need to obtain the data of the source data system from the disk, thereby avoiding the disk read and write rate affecting the data comparison rate and further improving the data comparison efficiency.

[0069] Reference Figure 1 , Figure 1 The following is a scenario diagram of the data comparison method of the present application. The data comparison device 100 obtains the first data identifier of each streaming data transmitted by the source data system 200 from its own cache area (not marked), and then obtains the data to be compared that matches the streaming data from the target data system 300 based on each first data identifier. The first data identifier of the streaming data is the same as the second data identifier of the data to be compared that matches the streaming data. The data comparison device 100 compares each streaming data with the matching data to be compared to obtain difference data, and the data comparison device 100 then outputs the difference data.

[0070] Combination Figure 1 The following embodiments describe the technical solutions shown in the present application in detail. It should be noted that the following embodiments can exist independently or in combination with each other, and the same or similar contents will not be described repeatedly in different embodiments.

[0071] Reference Figure 2 , Figure 2 The process diagram of the data comparison method provided in the embodiment of the present application is as follows Figure 1 , data comparison methods include:

[0072] Step S201, obtaining a first data identifier of each stream data transmitted by a source data system in a buffer area.

[0073] In this embodiment, the execution subject is a data comparison device. For the convenience of description, the device is used below to refer to the data comparison device. The device can be a server or a terminal device with a data comparison tool or data comparison function.

[0074] The same data identifier is stored in different data systems. In order to ensure data consistency, it is necessary to compare the data in the target data system with the data in the source data system. The target data system refers to the system or component in the system that needs to be compared with the source data system.

[0075] In one example, the device periodically compares data between the source data system and the target data system. The device sends information to the source data system, and the source data system transmits information to the device. The source data system transmits data to the device in a streaming manner. Streaming means that the source data system continuously transmits data. After receiving each streaming data from the source data system, the device stores each streaming data in a cache area, and instantly obtains data from the target data system to compare with the streaming data. The device uses the cache area as a buffer area for data, temporarily stores the streaming data in the cache area, and then obtains the data identifier of each streaming data transmitted by each source data system in the cache area. The data identifier is defined as the first data identifier.

[0076] In another example, the binlog (binary) log function is enabled in the source database, and the binlog log function can record the addition and deletion of all data in the database to obtain a log. The device obtains the log of the source data system. When it is determined based on the log that the source data system has data additions and deletions, the source data system sends a data comparison request to the device. When the device agrees to the source data system to compare the data, the source data system transmits streaming data to the device, and the data comparison request contains the system identifier of the target data system that needs to compare the data, so that the device can obtain data from the target data system based on the system identifier and compare it with the streaming data.

[0077] Step S202, based on multiple threads and each first data identifier, concurrently obtain the data to be compared that matches each streaming data in the target data system, the first data identifier of the streaming data is the same as the second data identifier of the data to be compared that matches the streaming data.

[0078] The device includes multiple threads, each of which is used to concurrently obtain the data to be compared that matches the streaming data from the target data system. After the device obtains the first data identifier of each streaming data, the device controls each thread to obtain the data to be compared that matches the streaming data from the target data system, and the second data identifier of the data to be compared is the same as the first data identifier of the streaming data that matches the data to be compared.

[0079] When the target data system is not a database or the data stored in the target data system is sensitive data, the device sends information to the target data system to request an interface address. The device obtains the data scanning interface or data search interface of the target data system through the interface address. The device obtains each data to be compared from the target data scanning interface or data search interface based on each thread. By obtaining data from the interface of the target data system, the security of data comparison is effectively improved, the risk of sensitive data transmission is avoided, and the comparison use scope of the device is effectively improved. In addition, after the target data system processes the data through the data acquisition method, it transmits the processed data to the device through the data scanning interface or data search interface. The data acquisition method, for example, processes the data as private data.

[0080] Step S203: compare each stream data with the data to be compared that matches the stream data.

[0081] After obtaining multiple data to be compared from the target data system through various threads concurrently, the device compares each stream data with the matching data to be compared. After the device compares each stream data, it discards each stream data and obtains each unmatched stream data from the buffer area for comparison.

[0082] In the process of obtaining streaming data in the cache area for comparison, the device only needs to obtain single data. Even if the number of different data is obtained between the streaming data to be compared and the data to be compared, it does not affect the data acquisition speed, and high-speed data comparison can be achieved even in the scenario where all the data are different.

[0083] The design of the memory cache area eliminates the need for the device to store streaming data on disk, thus avoiding the consumption of additional disk resources, reducing disk I / O operations (read and write operations) during the comparison process, and improving the data acquisition rate of the device.

[0084] In this embodiment, the first data identifier of each streaming data transmitted by the source data system is obtained from the cache area, and based on multiple threads and each first data identifier, the data to be compared that matches each streaming data is concurrently obtained in the target data system, and the second data identifier of the data to be compared is the same as the first data identifier, and each streaming data is compared with the data to be compared that matches the streaming data. In this embodiment, the data comparison device compares the streaming data obtained from the cache area with the data in the target data system, and there is no need to compare after downloading all the data from the source data system and the target data system, which reduces the data comparison time. In addition, by concurrently obtaining the data to be compared of the target data system through multiple threads, the data comparison time is further reduced and the data comparison efficiency is improved. In addition, by obtaining data from the cache area, there is no need to obtain the data of the source data system from the disk, which avoids the disk read and write rate affecting the data comparison rate, and further improves the data comparison efficiency.

[0085] Reference Figure 3 , Figure 3 The process diagram of the data comparison method for this application is shown in FIG. Figure 2 ,based on Figure 2 In the illustrated embodiment, step S203 includes:

[0086] Step S301, obtaining target parameters, where the target parameters include at least one of the amount of data currently cached in the cache area and the remaining computing resources of the data comparison device.

[0087] In this embodiment, the device can control the data acquisition speed, thereby controlling the working pressure of the target data system and the device itself. Specifically, the device acquires a target parameter, which includes at least one of the amount of current cached data in the cache area and the remaining computing resources of the device.

[0088] Step S302, determining the target number of threads for data comparison according to the target parameters.

[0089] After determining the target parameters, the device determines the target number of threads for data comparison based on the target parameters.

[0090] In one example, the target parameter includes the remaining computing resources. If the remaining computing resources are greater than or equal to the preset computing resources, the device creates multiple threads based on the current thread to obtain the target number, that is, the number of current threads and the number of created threads to obtain the target number. It should be noted that the target number is less than the maximum number of threads. When the remaining computing resources are less than the preset computing resources, the workload of the device is large, and the device shuts down some threads. The number of remaining threads is the target number.

[0091] In another example, the target parameter includes the amount of data converted to data, and the cached data refers to the amount of unmatched streaming data accumulated in the cache area. When the amount of cached data is less than a preset threshold, it can be determined that there is not too much unmatched streaming data accumulated in the cache area, and the device can slow down the data comparison rate to reduce the working pressure of the device. The device then closes some threads to determine the target number based on the number of unclosed threads, that is, the number of unclosed threads is used as the target number. When the amount of cached data is greater than or equal to the preset threshold, it can be determined that there is a lot of unmatched streaming data accumulated in the cache area, and the data comparison rate needs to be increased. The device then creates multiple threads, and determines the target number based on the number of created threads and the current number of threads, that is, the sum of the number of created threads and the current number of threads is the target number. The current thread refers to the number of threads of the device before the thread is created.

[0092] Step S303 : based on the target number of threads and each first data identifier, concurrently obtain the data to be compared that matches each streaming data in the target data system.

[0093] After determining the target number, the device concurrently obtains the to-be-compared data matched by each streaming data in the target data system based on the target number of threads and each first data identifier.

[0094] In this embodiment, the acquisition rate of data in the cache area is controlled by controlling the number of threads, thereby ensuring that the working pressure in the device during data comparison is not too large and ensuring that the cache data in the device cache area is not accumulated too much.

[0095] Reference Figure 4 , Figure 4 The process diagram of the data comparison method for this application is shown in FIG. Figure 3 ,based on Figure 2 or Figure 3 In the illustrated embodiment, step S403 includes:

[0096] Step S401, obtaining a data deformation strategy, where the data deformation strategy is used to convert data into a set format.

[0097] In this embodiment, in order to support effective comparison of heterogeneous data, it is necessary to transform the data of the source data system and the target data system. Transforming the data refers to the format conversion of the data or the replacement of the privacy data in the data.

[0098] The device obtains a data deformation strategy, which is used to convert the data into a set format. Specifically, the user can configure deformation rules, edit deformation rule scripts, or call deformation services provided by the user. The deformation rules, deformation rule scripts, and deformation services are stored in the device as data deformation strategies. The data deformation strategy can be stored in an external device, which can be connected and disconnected from the device by plugging and unplugging.

[0099] The data deformation script requires the user to upload the corresponding deformation script and select the function to be called. The currently supported script languages ​​include Lua script and Python script. In these scripts, the user needs to provide a deformation function to receive the source data and output the deformed data. In this script, the user can write a complex mapping relationship with certain logic to realize data mapping deformation in complex situations.

[0100] User-defined deformation service, which provides a solution for deformation requirements in scenarios that require complex operations such as encryption operations. The deformation process assembles the data to be deformed using a certain format, sends it to the user-provided interface for deformation processing, and returns it in the agreed format to complete the custom deformation.

[0101] The deformation rules run the mapping relationship and processing logic of the fields specified in the configuration file, and support a sharp and complex data deformation function to meet user needs. The deformation functions are as follows:

[0102] Table 1

[0103]

[0104] Step S402, deforming each streaming data and each to-be-compared data according to the data deformation strategy to obtain each first data and each second data, wherein the first data is used to indicate the deformed streaming data, and the second data is used to indicate the deformed to-be-compared data.

[0105] The device obtains a data deformation strategy, and according to the data deformation strategy, deforms each streaming data and each data to be compared to obtain each first data and each second data, wherein the first data is used to indicate the deformed streaming data, and the second data refers to the deformed data to be compared.

[0106] In one example, when the data deformation strategy is a deformation rule, the device uses a deformation function associated with the deformation rule to deform each streaming data and each to-be-compared data into each first data and each second data.

[0107] In another example, when the data deformation strategy is a deformation script, the device uses the deformation script to deform each streaming data and each to-be-compared data to obtain each first data and each second data.

[0108] In another example, when the data deformation strategy is a custom deformation service, the device sends each streaming data and each to-be-compared data to an interface corresponding to the custom deformation service for deformation, and then receives each first data and each second data fed back by the interface.

[0109] Step S403: compare each first data with the second data matching the first data.

[0110] After acquiring each first data and each second data, the device compares each first data with the second data matching the first data.

[0111] In this embodiment, a variety of data deformation schemes are provided, and deformation can be defined according to actual needs to complete subsequent comparison work. This process realizes pluggable data deformation, which has the characteristics of high flexibility and good scalability, thereby meeting the heterogeneous data comparison needs in various scenarios.

[0112] Reference Figure 5 , Figure 5 The process diagram of the data comparison method for this application is shown in FIG. Figure 4 ,based on Figures 2 to 4 In any of the embodiments shown in , step S203 includes:

[0113] Step S501, obtaining a data comparison scenario.

[0114] In this embodiment, the device has multiple comparison modes when comparing data, and each data comparison scenario is associated with a comparison mode. The comparison modes include full-text comparison, summary comparison, and custom rule comparison. When the data comparison scenario is a scenario that requires detailed comparison of differences, a full-text comparison is required; when the data comparison scenario is a scenario that only needs to record differences, a summary comparison is performed; when the data comparison scenario is a comparison scenario with special rules such as custom rules, data comparison is performed based on user-defined scripts.

[0115] Step S502 : comparing each stream data with the to-be-compared data that matches the stream data according to the comparison method associated with the data comparison scenario.

[0116] After the device determines the data comparison scenario, the device compares each stream data with the data to be compared that matches the stream data based on the comparison method associated with the data comparison scenario.

[0117] In one example, when the data comparison scenario is a difference record scenario, it is determined that the comparison method is a summary comparison method. The device determines the first summary feature value of each streaming data and the second summary feature value of each data to be compared based on the summary comparison method. The first summary feature value refers to obtaining a summary field from the streaming data and calculating a hash value of the summary field as the first summary feature value; the device obtains the summary field of the data to be compared and calculates the hash value of the summary field as the second summary feature value. The device compares the first summary feature value of the streaming data with the second summary feature value of the data to be compared that matches the streaming data.

[0118] In another example, when the data comparison is a custom scene, the comparison method is a script comparison method. The script corresponding to the script comparison method is a user-defined script. The device obtains the custom script associated with the script comparison method as the target script. The device inputs each streaming data and each data to be compared into the target script to perform a comparison between the streaming data and the data to be compared that matches the streaming data.

[0119] In yet another example, the data comparison scenario is a detailed comparison difference scenario, and the device performs a full-text comparison of the streaming data with the matching data to be compared.

[0120] In this embodiment, through multiple comparison methods, the device can be applicable to different data comparison scenarios to meet users' diverse data comparison needs.

[0121] In one embodiment, when the binlog logging function is not enabled in the source database, when performing data comparison, the user can specify the data update timestamp, that is, during the comparison process, obtain the data within a certain timestamp and compare it with the data in the target data system. After the comparison of the current time window is completed, the time window is updated to continue the comparison. In order to improve the coverage of data acquisition, there is a certain overlap in the time window during the update process. In this regard, the device obtains each streaming data updated within the first time window transmitted by the source data system in the cache area to perform data comparison. The data of the source data system compared by the data comparison device last time is located in the second time window, and there is an overlapping time window between the first time window and the second time window.

[0122] Reference Figure 6 , Figure 6 The process diagram of the data comparison method for this application is shown in FIG. Figure 5 ,based on Figures 2 to 5 In any of the embodiments shown in the figure, after step S203, the method further includes:

[0123] Step S601, obtaining difference data obtained after comparing each stream data with the matching data to be compared.

[0124] In this embodiment, the device compares each stream data with the matching data to be compared to obtain difference data, and the device needs to perform data processing based on the difference data.

[0125] Step S602: determine the difference type of each difference data, and determine the difference processing strategy according to the difference type and the type of the target data system.

[0126] Step S603: Process the to-be-compared data corresponding to the difference data in the target data system according to the difference processing strategy.

[0127] In this embodiment, the device compares each stream data with the matching data to be compared to obtain difference data, and the device needs to perform data processing based on the difference data.

[0128] For processing the difference data, the device needs to select different processing methods based on the type of the difference data and the type of the data system. To this end, the device obtains the difference data between each streaming data and the matching data to be compared.

[0129] In addition, the difference data is first sorted out to generate a comparison report and a list of specific difference information. The difference comparison report includes the number of consistent data, the number of differences, and the differences in each field, providing users with the results of the comparison task execution. The specific difference information records and stores the details of each difference information. The difference comparison report includes the total number of scanned data, the number of identical data, the number of lost data, the start and end time of the comparison task, the time taken to compare the data, and the difference records for each field of the data, and is stored in the form of a json file. In the detailed record, each difference data information is recorded one by one, including the unique primary key of the source and destination, the difference type, the detailed data of the source data system and the target data system, as well as the differences and specific timestamps found. This information is stored as a table file. The device displays the difference comparison report.

[0130] The device determines the difference type of each difference data, and determines the difference processing strategy based on the difference type and the type of the target data system. The device processes the to-be-compared data corresponding to the difference data in the target data system based on the difference processing strategy.

[0131] In one example, two types of differences are found during the data comparison process, namely, missing data and data differences. For missing data, the missing data needs to be inserted into the target data system, and for data with differences, the target data system data needs to be updated. In this regard, when the difference type of the difference data is missing data, the difference processing strategy is to insert the missing data into the target data system, that is, based on the missing data corresponding to the difference data, the missing data is inserted into the target data system. When the difference data is a data difference, the difference processing strategy is to update the data in the target data system, and the device obtains the streaming data corresponding to the difference data, and replaces the data to be compared in the target data system with the matching streaming data.

[0132] In another example, different difference processing operation and maintenance scripts are generated for different types of target data systems, and the data of the target data system is updated through the difference processing operation and maintenance scripts. The user can select the difference data to be processed, and generate a difference operation and maintenance script file based on these selected contents. When the target data system is a relational database, the corresponding data manipulation language is generated according to the difference content of the difference data, and the INSERT statement is generated for the missing data to insert the missing data, and the UPDATE statement is generated for the inconsistent data to update the data in the target data system. In this regard, the difference processing strategy is that the difference processing operation and maintenance script generates the statement of the target data system, and the data in the target data system is updated based on the statement.

[0133] When the target data system is Redis (cache database), the device generates a corresponding appended file based on the difference content of the difference data. The appended file generates different command statements according to the difference type of the difference data. For example, for the hash data type in the difference data, the hmset command is generated, and for the List type data in the difference data, the original data is first deleted and then the data is added by executing the lpush command, etc.

[0134] While the Redis system may not be able to directly import the aof file into the Redis database, it can generate corresponding Redis operation commands, which can be executed through the Redis console to process the difference data.

[0135] The user can further confirm and modify the above difference processing operation and maintenance script. Finally, the difference processing operation and maintenance script can connect to the target data system and perform the import operation, thereby eliminating the inconsistent data between the target data system and the source data system, and ensuring the consistency of the data between the source data system and the target data system.

[0136] In this embodiment, the device determines a difference processing strategy based on the difference type of the difference data and the type of the target data system, and processes the to-be-compared data corresponding to the difference data in the target data system based on the difference processing strategy.

[0137] Based on the data comparison method provided in this application, the data comparison performance was tested. The details are as follows:

[0138] In actual use, taking the comparison between MySQL database and Redis database as an example, without speed limit, the actual execution speed and performance of the data comparison process are shown in Table 2. Through data analysis, it can be seen that the data comparison device has a high data processing and comparison speed in different comparison scenarios, and the comparison speed is relatively stable, and the memory occupied is small. In Table 2, Table A-single record averages 1K, Table B-single record averages 1.5K, and Table C-single record averages 4.5K.

[0139] Table 2

[0140]

[0141] The performance comparison between the data comparison device and the comparison tool in the exemplary technology is shown in Table 3. It also has a good comparison speed in the scenario where there are differences in data. Through the comparison, it can be seen that the performance of the method of the present application is significantly better than other comparison tools.

[0142] Table 3

[0143] This application plan Comparison tool 1 Comparison Tool 2 Scenarios where data differ 14000TPS 3800TPS 1800TPS

[0144] The present application provides a general database data consistency comparison method, which realizes flexible and efficient data comparison process, comparison data deformation rule configuration, online data comparison method, and diversified difference processing operation and maintenance script generation, and can efficiently complete the data consistency check tasks of most commonly used databases.

[0145] Based on the contents described in the above embodiments, a data comparison device is also provided in the embodiments of the present application. Figure 7 , Figure 7 This is a schematic diagram of a program module of a data comparison device provided in an embodiment of the present application. In some implementations, the data comparison device 700 includes:

[0146] A first acquisition module 710 is used to acquire a first data identifier of each stream data transmitted by the source data system in the buffer area;

[0147] A second acquisition module 720 is used to concurrently acquire, in the target data system, data to be compared that matches each streaming data based on multiple threads and each first data identifier, wherein the first data identifier of the streaming data is the same as the second data identifier of the data to be compared that matches the streaming data;

[0148] The comparison module 730 is used to compare each stream data with the data to be compared that matches the stream data.

[0149] In some embodiments, the data comparison device 700 is specifically used for:

[0150] Acquiring a target parameter, the target parameter including at least one of the amount of current cached data in the cache area and remaining computing resources of the data comparison device;

[0151] Determine the target number of threads for data comparison according to the target parameter;

[0152] Based on the target number of threads and each first data identifier, the data to be compared that matches each streaming data is concurrently acquired in the target data system.

[0153] In some embodiments, the data comparison device 700 is specifically used for:

[0154] In response to the data volume being greater than or equal to a preset threshold, some threads are closed, and a target number is determined according to the number of threads that are not closed;

[0155] In response to the data volume being less than a preset threshold, a plurality of threads are created, and a target number is determined according to the number of current threads and the number of created threads.

[0156] In some embodiments, the data comparison device 700 is specifically used for:

[0157] Get the data deformation strategy, which is used to convert the data into a set format;

[0158] According to the data deformation strategy, each streaming data and each to-be-compared data are deformed to obtain each first data and each second data, wherein the first data is used to indicate the deformed streaming data, and the second data is used to indicate the deformed to-be-compared data;

[0159] Each first data is compared with the second data matching the first data.

[0160] In some embodiments, the data comparison device 700 is specifically used for:

[0161] Get data comparison scenario;

[0162] According to the comparison method associated with the data comparison scenario, each streaming data is compared with the data to be compared that matches the streaming data.

[0163] In some embodiments, the data comparison device 700 is specifically used for:

[0164] In response to the data comparison scenario being a difference record scenario, determining that the comparison mode is a summary comparison mode;

[0165] Determine, according to the summary comparison method, a first summary feature value of each stream data and a second summary feature value of each to-be-compared data;

[0166] The first summary feature value of the streaming data is compared with the second summary feature value of the data to be compared that matches the streaming data.

[0167] In some embodiments, the data comparison device 700 is specifically used for:

[0168] In response to the data comparison scenario being a custom scenario, determining that the comparison mode is a script comparison mode;

[0169] Get the target script associated with the script comparison method;

[0170] Each piece of streaming data and each piece of data to be compared are input into a target script to compare the streaming data with the piece of data to be compared that matches the streaming data.

[0171] In some embodiments, the data comparison device 700 is specifically used for:

[0172] In the buffer area, each stream data updated within the first time window transmitted by the source data system is obtained;

[0173] The data of the source data system compared by the data comparison device last time is located in the second time window, and there is an overlapping time window between the second time window and the first time window.

[0174] In some embodiments, the data comparison device 700 is specifically used for:

[0175] Obtaining difference data obtained after comparing each streaming data with the matching data to be compared;

[0176] Determine the difference type of each difference data, and determine the difference processing strategy based on the difference type and the type of target data system;

[0177] According to the difference processing strategy, the data to be compared corresponding to the difference data in the target data system is processed.

[0178] It should be noted that the various steps in the data comparison method executed by the data comparison device refer to the above embodiments for details, and will not be described in detail here.

[0179] Furthermore, based on the contents described in the above embodiments, an electronic device is also provided in an embodiment of the present application, which includes at least one processor, and a communication interface and a memory connected to the processor; wherein the communication interface is used to communicate with other communication devices, and the memory stores computer execution instructions; the above at least one processor executes the computer execution instructions stored in the memory to implement each step in the data comparison method described in the above embodiments.

[0180] In order to better understand the embodiments of the present application, refer to Figure 8 , Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0181] like Figure 8 As shown, the electronic device 800 of this embodiment includes: a processor 801, a memory 802, and a communication interface 804; wherein:

[0182] Memory 802, used to store computer-executable instructions;

[0183] The communication interface 804 is used to communicate with other communication devices;

[0184] The processor 801 is used to execute the computer-executable instructions stored in the memory to implement the various steps in the query optimization method described in the above embodiment.

[0185] Optionally, the memory 802 may be independent or integrated with the processor 801 .

[0186] When the memory 802 is independently provided, the device further includes a bus 803 for connecting the memory 802 , the communication interface 804 and the processor 801 .

[0187] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, each step in the data comparison method described in the above embodiment is implemented.

[0188] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the computer program implements the various steps in the data comparison method described in the above embodiment.

[0189] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0190] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0192] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0193] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0194] The memory may include high-speed memory, and may also include non-volatile storage, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0195] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0196] The storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The storage medium may be any available medium that can be accessed by a general or special purpose computer.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data comparison method, characterized in that: Applied to a data comparison device, the data comparison method comprises: Obtaining a first data identifier of each stream data transmitted by the source end data system in the buffer area; Based on multiple threads and each of the first data identifiers, in the target data system, concurrently obtaining the data to be compared that matches each of the streaming data, the first data identifier of the streaming data being the same as the second data identifier of the data to be compared that matches the streaming data; Each of the streaming data is compared with the data to be compared that matches the streaming data.

2. The method according to claim 1, characterized in that The method of concurrently acquiring, in the target data system, data to be compared that matches each of the streaming data based on the multiple threads and each of the first data identifiers includes: Acquiring a target parameter, wherein the target parameter includes at least one of the amount of current cached data in the cache area and the remaining computing resources of the data comparison device; Determine the target number of threads for data comparison according to the target parameter; Based on the target number of threads and each of the first data identifiers, in the target data system, the data to be compared that matches each of the streaming data is concurrently acquired.

3. The method according to claim 2, characterized in that The target parameter includes the amount of cached data, and determining the target number of threads for data comparison according to the target parameter includes: In response to the data volume being greater than or equal to a preset threshold, closing some threads, and determining a target number according to the number of threads that are not closed; In response to the data volume being less than a preset threshold, a plurality of threads are created, and a target number is determined according to the number of current threads and the number of created threads.

4. The method according to claim 1, characterized in that: The step of comparing each of the streaming data with the data to be compared that matches the streaming data includes: Obtaining a data deformation strategy, wherein the data deformation strategy is used to convert the data into a set format; According to the data deformation strategy, each of the streaming data and each of the to-be-compared data is deformed to obtain each of the first data and each of the second data, wherein the first data is used to indicate the deformed streaming data, and the second data is used to indicate the deformed to-be-compared data; Each of the first data is compared with the second data matching the first data.

5. The method according to claim 1, characterized in that The step of comparing each of the streaming data with the data to be compared that matches the streaming data includes: Get data comparison scenario; According to the comparison method associated with the data comparison scenario, each of the streaming data is compared with the data to be compared that matches the streaming data.

6. The method according to claim 5, characterized in that The step of comparing each of the streaming data with the data to be compared that matches the streaming data according to the comparison method associated with the data comparison scenario includes: In response to the data comparison scenario being a difference record scenario, determining that the comparison mode is a summary comparison mode; Determine, according to the summary comparison method, a first summary feature value of each of the streaming data and a second summary feature value of each of the to-be-compared data; The first summary feature value of the streaming data is compared with the second summary feature value of the data to be compared that matches the streaming data.

7. The method according to claim 5, characterized in that The step of comparing each of the streaming data with the data to be compared that matches the streaming data according to the comparison method associated with the data comparison scenario includes: In response to the data comparison scenario being a custom scenario, determining that the comparison mode is a script comparison mode; Obtain a target script associated with the script comparison method; Each of the streaming data and each of the data to be compared is input into a target script to compare the streaming data with the data to be compared that matches the streaming data.

8. The method according to claim 1, characterized in that: The obtaining of the first data identifier of each stream data transmitted by the source end data system in the buffer area includes: In the buffer area, each stream data updated within the first time window transmitted by the source data system is obtained; The data of the source data system compared by the data comparison device last time is located in a second time window, and there is an overlapping time window between the second time window and the first time window.

9. The method according to claim 1, characterized in that: After comparing each of the streaming data with the data to be compared that matches the streaming data, the method further includes: Obtaining difference data obtained after comparing each of the streaming data with the matching data to be compared; Determining a difference type of each of the difference data, and determining a difference processing strategy according to the difference type and the type of the target data system; According to the difference processing strategy, the data to be compared corresponding to the difference data in the target data system is processed.

10. A data comparison device, characterized in that: include: A first acquisition module, used to acquire a first data identifier of each stream data transmitted by the source end data system in the buffer area; A second acquisition module is used for concurrently acquiring, in a target data system, data to be compared that matches each of the streaming data based on multiple threads and each of the first data identifiers, wherein the first data identifier of the streaming data is the same as the second data identifier of the data to be compared that matches the streaming data; The comparison module is used to compare each of the streaming data with the data to be compared that matches the streaming data.

11. An electronic device, characterized in that: include: A processor, and a memory and a communication interface communicatively connected to the processor; The communication interface is used to communicate with other communication devices; The memory is used to store computer-executable instructions; The processor is used to execute the computer-executable instructions stored in the memory to implement the data comparison method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the data comparison method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data comparison method according to any one of claims 1 to 9 is implemented.

Images