Data comparison method, device, business data comparison system and electronic device

Abstract

The present application relates to a data comparison method, device, business data comparison system and electronic device, wherein the data comparison method includes: reading first-party target data and second-party target data from a comparison database through a target comparison task; filtering the first-party target data and the second-party target data by rules to obtain first-party comparison data and second-party comparison data; batch comparing the generated first comparison elements and second comparison elements, and storing the comparison elements of both parties with differences in the first comparison elements and the second comparison elements in a first result database; through the target comparison task, comparing the newly read new comparison elements with the corresponding comparison elements in the first result database one by one to obtain the comparison results. It realizes the adaptation of the data comparison function in different business scenarios in the form of rule configuration, thereby reducing the difficulty of data comparison and improving the flexibility of data comparison.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data comparison method, device, business data comparison system and electronic device. Background Art

[0002] As a payment solution based on the Internet and financial technology, the cross-border payment real-time system is designed to handle cross-border payment transactions. In the cross-border payment real-time system, each payment request will generate a corresponding payment order, and each system has its own order. The consistency of payment business data between systems has an important impact on the normal operation of the system. Therefore, it is necessary to compare the consistency of business data between different systems.

[0003] In related technologies, different data comparison logics are often developed for different business scenarios and business needs. As the scale of real-time payment systems expands and the amount of data increases, the performance and efficiency of the above methods will be difficult to meet actual business needs. In order to cope with the growing amount of data and increasingly complex business needs, the data comparison system needs to be continuously expanded and optimized, resulting in higher difficulty and lower flexibility in data comparison.

[0004] Therefore, in the relevant technology, there are still problems that need to be solved, such as high difficulty in data comparison and low flexibility in comparison. Summary of the invention

[0005] In this embodiment, a data comparison method, device, business data comparison system and electronic device are provided to solve the problems of high difficulty and low comparison flexibility in data comparison in related technologies.

[0006] In a first aspect, a data comparison method is provided in this embodiment, comprising:

[0007] Execute a target comparison task pre-configured for a target business scenario, so as to read the first-party target data and the second-party target data from a comparison database through the target comparison task; wherein the comparison database obtains the first-party target data from the first-party system and obtains the second-party target data from the second-party system;

[0008] By means of the target comparison task, based on the pre-configured target comparison task rules, the first-party target data and the second-party target data are subjected to rule filtering to obtain the first-party comparison data and the second-party comparison data;

[0009] Generate a first comparison element according to the first-party comparison data, generate a second comparison element according to the second-party comparison data, perform batch comparison on the first comparison element and the second comparison element, and store the comparison elements of both parties that are different between the first comparison element and the second comparison element in a first result database;

[0010] Through the target comparison task, the newly read new comparison elements are compared one by one with the corresponding comparison elements in the first result database to obtain a comparison result.

[0011] In some embodiments, the method further comprises:

[0012] The binary log messages of the first-party system and the second-party system are transmitted to the comparison database in real time through the message queue, and the first-party target data of the first-party system and the second-party target data of the second-party system are stored in the comparison database according to the binary log messages.

[0013] In some embodiments, according to the binary log message, storing the first-party target data of the first-party system and the second-party target data of the second-party system in the comparison database includes:

[0014] Normalizing the first-party initial data carried by the binary log message originating from the first-party system to obtain the first-party target data;

[0015] Normalizing the second-party initial data carried by the binary log message originating from the second-party system to obtain the second-party target data;

[0016] The first-party target data and the second-party target data are stored in the comparison database.

[0017] In some embodiments, the target comparison task rule includes a first comparison rule and a second comparison rule; through the target comparison task, based on the pre-configured target comparison task rule, the first-party target data and the second-party target data are subjected to rule filtering to obtain the first-party comparison data and the second-party comparison data, including:

[0018] Through the target comparison task, based on the first comparison rule and a preset comparison time cursor, the first party target data and the second party target data within a preset time range are read from the comparison database respectively;

[0019] Through the target comparison task, based on the second comparison rule, the first-party comparison data whose business type matches the comparison requirements in the target business scenario is filtered out from the first-party target data within the preset time range, and the second-party comparison data whose business type matches the comparison requirements is filtered out from the second-party target data within the preset time range.

[0020] In some of the embodiments, the newly read new comparison elements are compared one by one with the corresponding comparison elements in the first result database through the target comparison task to obtain the comparison results, including:

[0021] In the case where the reason for the inconsistency of the comparison elements is missing data on one side, if a new missing comparison element is received within a preset review period after the batch comparison is completed, the new comparison element is compared with the corresponding comparison element in the first result database one by one to obtain a comparison result;

[0022] If the comparison result indicates that the corresponding comparison element is consistent with the new comparison element, removing the corresponding comparison element from the first result database;

[0023] Through the target comparison task, the first result database is scanned according to a preset scanning cycle, and the comparison elements retained in the first result database are stored in the second result database; the interval time period of the scanning cycle is longer than the review time period.

[0024] In some embodiments, the method further comprises:

[0025] According to the data in the second result database, an alarm message is pushed to the associated business party.

[0026] In a second aspect, a data comparison device is provided in this embodiment, including: a data reading module, a data filtering module, a batch comparison module and a line-by-line comparison module; wherein:

[0027] The data reading module is used to execute a target comparison task pre-configured for a target business scenario, so as to read the first-party target data and the second-party target data from the comparison database through the target comparison task; wherein the comparison database obtains the first-party target data from the first-party system and obtains the second-party target data from the second-party system;

[0028] The data filtering module is used to perform rule filtering on the first-party target data and the second-party target data through the target comparison task based on pre-configured target comparison task rules to obtain first-party comparison data and second-party comparison data;

[0029] The batch comparison module is used to generate a first comparison element according to the first comparison data, generate a second comparison element according to the second comparison data, perform batch comparison on the first comparison element and the second comparison element, and store the comparison elements of both parties that are different between the first comparison element and the second comparison element in a first result database;

[0030] The item-by-item comparison module is used to compare the newly read new comparison elements with the corresponding comparison elements in the first result database item by item through the target comparison task to obtain a comparison result.

[0031] In a third aspect, a business data comparison system is provided in this embodiment, including: a comparison database, a first result database, and a server device; the server device is connected to the comparison database and the first result database respectively;

[0032] The server device is used to execute the data comparison method described in the first aspect above.

[0033] In a fourth aspect, an electronic device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data comparison method described in the first aspect when executing the computer program.

[0034] In a fifth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the data comparison method described in the first aspect is implemented.

[0035] Compared with the related art, a data comparison method, device, business data comparison system and electronic device are provided in this embodiment. Among them, the data comparison method first executes a target comparison task pre-configured for a target business scenario to read the first-party target data and the second-party target data from the comparison database through the target comparison task; wherein the comparison database obtains the first-party target data from the first-party system and obtains the second-party target data from the second-party system; then, through the target comparison task, based on the pre-configured target comparison task rules, the first-party target data and the second-party target data are rule-filtered to obtain the first-party comparison data and the second-party comparison data; a first comparison element is generated according to the first-party comparison data, a second comparison element is generated according to the second-party comparison data, and the first comparison element and the second comparison element are batch-compared, and the comparison elements of both parties that have differences in the first comparison element and the second comparison element are stored in the first result database; finally, through the target comparison task, the newly read new comparison elements are compared one by one with the corresponding comparison elements in the first result database to obtain the comparison result. It realizes the adaptation of data comparison function in different business scenarios in the form of rule configuration, thereby reducing the difficulty of data comparison and improving the flexibility of data comparison.

[0036] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 is a hardware structure block diagram of a terminal of the data comparison method of this embodiment;

[0039] Figure 2 is a flow chart of the data comparison method of this embodiment;

[0040] Figure 3 is a flow chart of a data comparison method in some embodiments;

[0041] Figure 4 is a schematic diagram of data flow in the data comparison process of some embodiments;

[0042] Figure 5 is a schematic diagram of a comparison task execution process in some embodiments;

[0043] Figure 6 is a structural block diagram of the data comparison device of this embodiment;

[0044] Figure 7 is a schematic diagram of the structure of the business data comparison system of this embodiment;

[0045] Figure 8 It is a schematic diagram of the comparison function of the business data comparison system in some embodiments. DETAILED DESCRIPTION

[0046] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0047] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0048] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the data comparison method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0049] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data comparison method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0050] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0051] In this embodiment, a data comparison method is provided. Figure 2 is a flow chart of the data comparison method of this embodiment, such as Figure 2 As shown, the process includes the following steps:

[0052] Step S210, execute the target comparison task pre-configured for the target business scenario to read the first-party target data and the second-party target data from the comparison database through the target comparison task; wherein the comparison database obtains the first-party target data from the first-party system and obtains the second-party target data from the second-party system.

[0053] Specifically, a configurable comparison task setting mechanism can be provided for all business scenarios, and corresponding comparison tasks can be configured for each business scenario in advance according to the business needs of each business scenario. When data comparison is required for the business scenario, the corresponding comparison task is executed to implement the data comparison logic. Among them, the above-mentioned comparison task can be implemented based on the form of configuration file and computer program script. The user can configure the field mapping method of the comparison data and the data filtering rules through the configuration file. For the specific comparison task to be performed, the data comparison cycle, delay time, and specific rules and logic for data comparison can be further configured. For example, the order number is used to associate the values ​​of the amount field, currency field, and order status field in the data comparison between the two parties. Afterwards, the computer script reads and parses the above-mentioned configuration file, and performs corresponding data processing and comparison steps based on the content set in the configuration file to complete the data comparison.

[0054] The comparison database may be a preset intermediate database independent of the two systems that need to perform data comparison. The first-party system and the second-party system may be systems in which any two parties need to perform data comparison in a real-time payment scenario. Exemplarily, the first-party system may be a payment system used by a user in a payment scenario, and the second-party system may be some payment channels associated with the payment scenario, such as an e-commerce platform, a payment gateway, a third-party payment institution, etc. The first-party target data and the second-party target data may specifically be data to be compared that are respectively derived from the first-party system and the second-party system. It is understandable that due to differences in data formats and other aspects between different systems, there may be differences in the format and form of data directly read from the first-party system and the second-party system. Therefore, the first-party target data may be data after format conversion of the first-party initial data read from the first-party system, and the second-party target data may be data after format conversion of the second-party initial data read from the second-party system. The formats of the above-mentioned two-party data after format conversion are consistent.

[0055] In particular, the first-party data may be data related to payment orders in transactions, and the second-party data may be data related to channel orders. Payment orders are orders generated by the payment system during the payment process. When the user confirms the transaction and selects the payment method, the payment system will generate a payment order based on the transaction information. Channel orders refer to orders generated through specific channels or platforms during the transaction process. Such channels may include e-commerce platforms, payment gateways, third-party payment institutions, etc.

[0056] Among them, the target comparison task can be set as a scheduled and periodically executed task. When the preset execution time of the target comparison task is reached, the target comparison task will start automatically and read the first-party target data and second-party target data that need to be compared from the comparison database.

[0057] Before executing the target comparison task, you can configure the target comparison task in the background, set the name, start time, delay time, comparison cycle, comparison interval time, threshold, responsible personnel information, etc. of the target comparison task. When configuring specific comparison task rules, set the unique identifier (Identity document, referred to as id) of the data source filtering of both parties, the data source status of both parties, the data source filtering rules of both parties, the comparison matching rules, and the comparison factor, etc. After configuration, when executing the target comparison task, the target comparison task will read the first-party target data and the second-party target data from the comparison database to the local memory of the running server device based on the pre-configured content.

[0058] Step S220 , through the target comparison task, based on the pre-configured target comparison task rules, the first-party target data and the second-party target data are rule-filtered to obtain the first-party comparison data and the second-party comparison data.

[0059] After reading the data, the first-party target data and the second-party target data are filtered according to the target comparison task rules in the target comparison task to obtain filtered first-party comparison data and second-party comparison data. In this step, the target comparison task rules can be pre-configured, wherein the data within a certain time period that needs to be compared, the data corresponding to certain orders, or the data corresponding to a certain transaction type can be configured. By filtering the first-party target data and the second-party target data according to the rules, the interference of irrelevant data can be reduced, and the amount of data that needs to be processed in the comparison process can be reduced, thereby improving the accuracy of the comparison results and improving the comparison efficiency.

[0060] Step S230, generating a first comparison element according to the first-party comparison data, generating a second comparison element according to the second-party comparison data, performing batch comparison on the first comparison element and the second comparison element, and storing the comparison elements of both parties that are different between the first comparison element and the second comparison element in the first result database.

[0061] Among them, the first comparison element and the second comparison element are both key elements or indicators used to compare and verify data consistency during the data comparison process. For example, when reconciling data between two parties in a transaction scenario, it can be a reconciliation element string, which contains a reconciliation factor and a verification field (such as an amount or amount details). The reconciliation factor can specifically be the association relationship between the two parties of the reconciliation. The first comparison element is generated based on the first-party comparison data, and the second comparison element is generated based on the second-party comparison data. The first comparison element and the second comparison element can be represented in the form of a string. After that, the difference method can be used to perform batch comparison on the first comparison element and the second comparison element. For example, the two-set difference method of guava can be used for batch comparison, where guava is an open source tool library.

[0062] After the batch comparison is completed, the inconsistent comparison elements are saved in the first result database. The first result database can be regarded as a coarse-grained inconsistency database, which is used to store inconsistent data found after the above coarse-grained batch comparison. The reasons for data inconsistency can also be recorded in the first result database, such as missing first-party data, missing second-party data, inconsistent data between the two parties, etc.

[0063] Step S240, through the target comparison task, the newly read new comparison elements are compared one by one with the corresponding comparison elements in the first result database to obtain a comparison result.

[0064] Through the target comparison task, the newly read new comparison elements can be compared one by one with the corresponding comparison elements put into the first result database, thereby realizing two data comparisons and obtaining the final comparison result, which indicates that the data comparison is successful (consistent) or the data comparison fails (inconsistent). In this step, by combining coarse-grained batch comparison and fine-grained line-by-line comparison, the two data comparisons can be covered, thereby avoiding data loss caused by delayed data arrival or inconsistency of data between the two parties caused by other external factors, thereby reducing the misjudgment of inconsistent data and improving the accuracy of the comparison results.

[0065] For real-time payment systems, data consistency has a far-reaching impact on the normal operation of the system and the user experience of the system. For a certain product, the corresponding channel order status failed, while the corresponding payment order status was successful, which resulted in the actual user not paying successfully but the merchant shipping the goods, causing damage to the merchant's interests. Or, the channel order status was successful but the payment order failed, resulting in the actual user paying successfully but the merchant not shipping the goods, causing damage to the user's interests. It can be seen that data inconsistency will have a great impact on user payment experience and merchant transactions.

[0066] In related technologies, traditional data comparison methods are usually used to verify data consistency. That is, corresponding data comparison functions are developed for different business scenarios and business needs. In this case, every time a new business scenario is added or business needs are adjusted, the data comparison function needs to be redeveloped. In addition, in order to cope with the growing amount of data and increasingly complex business needs, the existing data comparison function needs to be continuously expanded and optimized, which leads to increased difficulty and complexity of data comparison in actual applications, and low flexibility of data comparison.

[0067] Accordingly, in this embodiment, through the above steps S210 to S240, a comparison task and a corresponding comparison task rule are configured for each business scenario. The differentiated needs of business scenarios can be met through the configuration items of the comparison tasks and the dynamic expressions in the comparison task rules. This embodiment abstracts the comparison tasks and comparison task rules and realizes configuration to match the different business needs of different business scenarios, thereby realizing the reuse of the data comparison function in different business scenarios and realizing the flexible expansion of the data comparison function. Therefore, it is possible to reduce the difficulty of data comparison, improve the flexibility of data comparison, and enhance the user experience.

[0068] Moreover, in practical applications, this embodiment can also reduce the development cost of data comparison by decoupling the data sources of the two parties to be compared through a comparison database that is independent of the two parties to be compared, thereby achieving zero-intrusive access comparison.

[0069] The above steps S210 to S240 execute the target comparison task pre-configured for the target business scenario, so as to read the first-party target data and the second-party target data from the comparison database through the target comparison task; wherein, the comparison database obtains the first-party target data from the first-party system and obtains the second-party target data from the second-party system; through the target comparison task, based on the pre-configured target comparison task rules, the first-party target data and the second-party target data are rule-filtered to obtain the first-party comparison data and the second-party comparison data; a first comparison element is generated according to the first-party comparison data, a second comparison element is generated according to the second-party comparison data, and the first comparison element and the second comparison element are batch-compared, and the comparison elements of both parties that have differences in the first comparison element and the second comparison element are stored in the first result database; through the target comparison task, the newly read new comparison element is compared one by one with the corresponding comparison element in the first result database to obtain the comparison result. It realizes the adaptation of the data comparison function in different business scenarios in the form of rule configuration, thereby reducing the difficulty of data comparison and improving the flexibility of data comparison.

[0070] In one embodiment, the data comparison method may further include:

[0071] The binary log messages of the first-party system and the second-party system are transmitted to the comparison database in real time through the message queue, and the first-party target data of the first-party system and the second-party target data of the second-party system are stored in the comparison database according to the binary log messages.

[0072] Specifically, based on the message push method, when new data is generated in the first-party system and the second-party system, the data can be pushed to the comparison database in the form of binary log messages (binlog messages), thereby realizing real-time update of data in the comparison database. Among them, the first-party system and the second-party system associated with the comparison database can be set by configuring the data source of the comparison.

[0073] In this embodiment, by parsing the data of the business system, the corresponding binlog messages are stored in the comparison database, and then the target comparison task scans the comparison database and filters the data according to the comparison task rules, so that data inconsistencies can be discovered in seconds, thereby improving the efficiency of data comparison.

[0074] In particular, in one embodiment, according to the binary log message, storing the first-party target data of the first-party system and the second-party target data of the second-party system in the comparison database may include:

[0075] The first-party initial data carried by the binary log message from the first-party system is normalized to obtain the first-party target data; the second-party initial data carried by the binary log message from the second-party system is normalized to obtain the second-party target data; the first-party target data and the second-party target data are stored in the comparison database.

[0076] Understandably, in a distributed system, due to the differences in data formats, storage methods, and access methods between different business systems, traditional data comparison processing requires processing and comparing heterogeneous data from different business systems separately, which results in high complexity of data comparison processing and high difficulty in comparison implementation.

[0077] In this embodiment, the formats of the first-party initial data and the second-party initial data can be normalized respectively through a pre-configured data source field mapping method, thereby realizing data format conversion, achieving isomorphism of business data, and generating normalized data with a unified format, thereby reducing the complexity and difficulty of data comparison.

[0078] In addition, in one embodiment, the target comparison task rule includes a first comparison rule and a second comparison rule; based on the above step S220, through the target comparison task, based on the pre-configured target comparison task rule, the first-party target data and the second-party target data are subjected to rule filtering to obtain the first-party comparison data and the second-party comparison data, which may include:

[0079] Through the target comparison task, based on the first comparison rule and the preset comparison time cursor, the first-party target data and the second-party target data within the preset time range are read from the comparison database respectively; through the target comparison task, based on the second comparison rule, the first-party comparison data whose business type matches the comparison requirements in the target business scenario are filtered out from the first-party target data within the preset time range, and the second-party comparison data whose business type matches the comparison requirements are filtered out from the second-party target data within the preset time range.

[0080] Specifically, when configuring the rules, the first comparison rule and the second comparison rule can be configured separately. Among them, the first comparison rule is used as a basic rule to implement the filtering of basic data such as strong field comparison association id and comparison status; the second comparison rule is used as an advanced rule to calculate the fields that need to be compared through the rule engine, and then filter. The above-mentioned comparison requirements can specifically be the transaction type corresponding to the data. Each reconciliation task reads the data within a preset time range from the comparison database to the local memory of the server device according to the first comparison rule and the comparison time cursor, and then uses the second comparison rule to filter out the comparison data required for the target business scenario. For example, the second comparison rule is used to calculate the transaction type through the rule engine, and then the order data of the specified transaction type is filtered out, so that in the subsequent comparison process, only the data of this transaction type is compared.

[0081] This embodiment separates the focus of comparison by dividing the comparison rules into a first comparison rule and a second comparison rule, so that scenarios with lower comparison complexity can be satisfied by the first comparison rule, while scenarios with higher comparison complexity can be flexibly expanded using the second comparison rule.

[0082] Additionally, in one embodiment, based on the above step S240, the newly read new comparison elements are compared one by one with the corresponding comparison elements in the first result database through the target comparison task to obtain the comparison results, which may specifically include:

[0083] When the reason for the inconsistency of the comparison elements is the missing data of one party, if the missing new comparison elements are received within the preset review time period after the batch comparison is completed, the new comparison elements will be compared with the corresponding comparison elements in the first result database one by one to obtain the comparison results; if the comparison result indicates that the corresponding comparison elements are consistent with the new comparison elements, the corresponding comparison elements will be removed from the first result database; through the target comparison task, the first result database is scanned according to the preset scanning cycle, and the comparison elements retained in the first result database are stored in the second result database; the interval time period of the scanning cycle is longer than the review time period.

[0084] When storing the inconsistent data after batch comparison in the first result database, the inconsistency reasons can be recorded accordingly, such as missing first-party data, missing second-party data, or inconsistent data of both parties. In order to avoid inconsistent data comparison caused by data delay, the inconsistent data comparison caused by missing data of one party in the first result database can be checked again. For example, the first-party comparison data is the source data, and the second-party comparison data is the target data. The source data refers to the data of the system with the front comparison order set in the target comparison task when performing data comparison, and the target data refers to the data of the system with the back comparison order. If the source data is missing during the first comparison, the corresponding source data and the target data fail to be compared during the first comparison, then it can be determined within a preset time period, such as within 30 minutes after the first comparison fails, whether the corresponding source data is generated, and if so, the newly generated source data that was originally missing is compared with the corresponding data in the first result database again. If the comparison is consistent, it is confirmed that the data comparison is successful and it is removed from the first result database. The comparison status can be modified to be automatically repaired.

[0085] The target comparison task can scan the first result database according to a preset scanning cycle, for example, every 30 minutes, and store the data that has not been automatically repaired in the second result database. The second result database can be regarded as a fine-grained inconsistency database, and the data falling into the fine-grained inconsistency database is regarded as inconsistent data finally confirmed after two comparisons.

[0086] It can be understood that all the specific time values ​​shown in this embodiment are only examples, and those skilled in the art can also set other values ​​according to the requirements of actual application scenarios, and this embodiment does not specifically limit this.

[0087] This embodiment uses two data comparisons as a backup, which can effectively reduce the problem of comparison inconsistency caused by data delay, thereby improving the accuracy of the comparison result.

[0088] In addition, in some of the embodiments, after the target comparison task completes the comparison of data within a certain time cursor range, it can also record the start time, end time and comparison result of the time cursor currently completed, thereby forming a complete, orderly and traceable data comparison record.

[0089] Optionally, in one embodiment, the data comparison method may further include: pushing an alarm message to the associated business party based on the data in the second result database. After determining that there are inconsistent data in the two systems, an alarm message may be sent to the business party by means of message push. For example, the application of each associated business party may subscribe to the alarm message based on the message queue, and after the inconsistent data is confirmed through data comparison, the corresponding alarm message is pushed to the application of each business party, so as to facilitate the staff to handle it in time and reduce the impact of data inconsistency on data and information flow.

[0090] Figure 3 is a flow chart of a data comparison method in some of the embodiments, such as Figure 3 As shown, the data comparison method includes the following steps:

[0091] Step S301, configure the comparison task for each business scenario; specifically, the name, start time, delay time, comparison cycle, comparison interval, various comparison thresholds and information of the responsible person of the comparison task can be configured in the background.

[0092] Step S302, configure specific comparison task rules for the comparison task; wherein, the data source filter IDs of the two parties to be compared, the data source states of the two parties, the data filtering rules of the two parties, the comparison rules and the comparison factors, etc. are set.

[0093] Step S303, execute the target comparison task pre-configured for the target business scenario, so as to read the first-party target data and the second-party target data from the comparison database through the target comparison task; wherein, the target comparison task, according to the first comparison rule and the preset comparison time cursor, reads the first-party target data and the second-party target data within the preset time range from the comparison database to the local memory of the server device, and then based on the second comparison rule, uses the rule engine to filter out the first comparison data and the second comparison data that match the target business scenario.

[0094] Step S304, generating a first comparison element according to the first comparison data, generating a second comparison element according to the second comparison data, performing batch comparison on the first comparison element and the second comparison element, and storing the first comparison element and the second comparison element that have differences in the first comparison element and the second comparison element in the first result database. Specifically, two combined difference methods of guava can be used to perform batch comparison, and when inconsistent comparison data is stored in the first result database, the cause of the inconsistency is recorded.

[0095] Step S305, when the reason for the inconsistency of the comparison elements is missing data on one side, if the missing new comparison elements are received within the preset review time period after the batch comparison is completed, the new comparison elements are compared one by one with the corresponding comparison elements in the first result database.

[0096] Step S306: If the comparisons are consistent, the corresponding comparison elements are removed from the first result database.

[0097] Step S307, through the target comparison task, the first result database is scanned according to the preset scanning cycle, and the comparison elements retained in the first result database are stored in the second result database; the interval time period of the scanning cycle is longer than the review time period.

[0098] Step S308: Push an alarm message to the associated business party based on the data in the second result database.

[0099] The above steps S301 to S308 implement the adaptation of the data comparison function in different business scenarios in the form of rule configuration, thereby reducing the difficulty of data comparison and improving the flexibility of data comparison.

[0100] Figure 4 Schematic diagram of data flow in the data comparison process of some embodiments. Figure 4First, each business system can push the data with changed status through its own application (such as application 1 to 3) in the form of message queue (MQ) message push, and after the comparison data is normalized, transmit it to the comparison database. After that, the pre-configured comparison task reads the first-party target data and the second-party target data to be compared from the comparison database by scanning the table. Then, the comparison task filters out the first-party comparison data and the second-party comparison data by matching the pre-configured comparison task rules, and puts the inconsistent data into the first result database through the remote dictionary service (redis) or the local cache through coarse-grained batch comparison. After that, if there is data inconsistency caused by the missing data of one party, if the missing data arrives within the preset time, the first result database is scanned by the comparison task, and the newly arrived data is compared with the corresponding data in the first result database one by one. If they are consistent, the corresponding data is removed. The comparison task periodically scans the first result database by scanning the table, puts the data that still remains after a preset time period into the second result database, and subscribes to the inconsistent data and pushes it to the applications of each business party (applications 1 to 3).

[0101] Figure 5 Schematic diagram of the comparison task execution process in some embodiments. Figure 5 As shown, when the comparison task is executed, the data to be compared (first-party target data and second-party target data) is read from the comparison database, and then the data is converted and filtered to perform business comparison. Specifically, the two-way comparison of data can be achieved by batch comparison, and the identified inconsistent data is stored in the first result database. After that, the comparison task compares the data in the first result database by scanning one by one, and stores the inconsistent data in the second result database.

[0102] In this embodiment, a data comparison device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0103] Figure 6 is a structural block diagram of the data comparison device 60 of this embodiment, as shown in Figure 6 As shown, the data comparison device 60 includes: a data reading module 62, a data filtering module 64, a batch comparison module 66 and a line-by-line comparison module 68; wherein:

[0104] The data reading module 62 is used to execute the target comparison task pre-configured for the target business scenario, so as to read the first-party target data and the second-party target data from the comparison database through the target comparison task; wherein, the comparison database obtains the first-party target data from the first-party system and obtains the second-party target data from the second-party system; the data filtering module 64 is used to perform rule filtering on the first-party target data and the second-party target data based on the pre-configured target comparison task rules through the target comparison task to obtain the first-party comparison data and the second-party comparison data; the batch comparison module 66 is used to generate a first comparison element according to the first-party comparison data, generate a second comparison element according to the second-party comparison data, perform batch comparison on the first comparison element and the second comparison element, and store the comparison elements of both parties that have differences in the first comparison element and the second comparison element in the first result database; the line-by-line comparison module 68 is used to compare the newly read new comparison elements with the corresponding comparison elements in the first result database line by line through the target comparison task to obtain the comparison result.

[0105] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination. For specific examples in this embodiment, reference can be made to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0106] This embodiment also provides a business data comparison system. Figure 7 FIG. 7 is a schematic diagram of the structure of the business data comparison system 70 of this embodiment. Figure 7 As shown, the business data comparison system 70 includes: a comparison database 72, a first result database 74, and a server device 76; the server device 76 is connected to the comparison database 72 and the first result database 74 respectively; the server device 76 is used to execute the data comparison method provided in the above embodiment. Among them, the business data comparison system can also include a second result database (not shown in the figure) for storing data confirmed to be inconsistent after the second comparison.

[0107] Figure 8 FIG. 1 is a schematic diagram of the comparison function of the business data comparison system in some embodiments, such as Figure 8 As shown, the functions of the business data comparison system include business data isomorphism, scheduled tasks, comparison methods, configuration center and comparison result processing.

[0108] Among them, the business data isomorphism function can read the data source configuration, receive the binlog messages of the data sources of each business system, filter the business data in the rule engine, and then perform format conversion processing on the filtered business data to generate unified reconciliation normalization data. Therefore, the business data isomorphism function can realize message reception and data conversion.

[0109] The scheduled task function can execute specific comparison tasks and realize coarse-grained data scanning. The coarse-grained data scanning refers to scanning the data in the first result database. The comparison method is a two-way comparison of data. The comparison method can realize a two-way comparison of business data.

[0110] The configuration center can configure comparison tasks, filter rules, and field mapping. The field mapping configuration is used to map normalized data fields to business data source fields, so as to standardize the fields of data from different business data sources.

[0111] The comparison result processing function includes background data display, which is used to display inconsistent data identified after comparison, and message subscription notification, which promptly informs the business party of any data inconsistencies through message push.

[0112] In this embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0113] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0114] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0115] S1, executing a target comparison task pre-configured for a target business scenario, so as to read first-party target data and second-party target data from a comparison database through the target comparison task; wherein the comparison database obtains the first-party target data from the first-party system and obtains the second-party target data from the second-party system;

[0116] S2, through the target comparison task, based on the pre-configured target comparison task rules, the first-party target data and the second-party target data are rule-filtered to obtain the first-party comparison data and the second-party comparison data;

[0117] S3, generating a first comparison element according to the first-party comparison data, generating a second comparison element according to the second-party comparison data, performing batch comparison on the first comparison element and the second comparison element, and storing the comparison elements of both parties that are different between the first comparison element and the second comparison element in a first result database;

[0118] S4, through the target comparison task, the newly read new comparison elements are compared with the corresponding comparison elements in the first result database one by one to obtain the comparison results.

[0119] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0120] In addition, in combination with the data comparison method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the data comparison method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the data comparison methods in the above embodiments is implemented.

[0121] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0123] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0124] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0125] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A data comparison method, characterized in that: include: Execute a target comparison task pre-configured for a target business scenario, so as to read the first-party target data and the second-party target data from a comparison database through the target comparison task; wherein the comparison database obtains the first-party target data from the first-party system and obtains the second-party target data from the second-party system; By means of the target comparison task, based on pre-configured target comparison task rules, the first-party target data and the second-party target data are subjected to rule filtering to obtain first-party comparison data and second-party comparison data; Generate a first comparison element according to the first comparison data, generate a second comparison element according to the second comparison data, perform batch comparison on the first comparison element and the second comparison element according to the difference between two sets of the open source tool library guava, and store the comparison elements of both parties with differences between the first comparison element and the second comparison element in a first result database; In the case where the reason for the inconsistency of the comparison elements is missing data on one side, if a new missing comparison element is received within a preset review period after the batch comparison is completed, the new comparison element is compared with the corresponding comparison element in the first result database one by one to obtain a comparison result; If the comparison result indicates that the corresponding comparison element is consistent with the new comparison element, removing the corresponding comparison element from the first result database; Through the target comparison task, the first result database is scanned according to a preset scanning cycle, and the comparison elements retained in the first result database are stored in the second result database; the interval time period of the scanning cycle is longer than the review time period.

2. The data comparison method according to claim 1, characterized in that: The method further comprises: The binary log messages of the first-party system and the second-party system are transmitted to the comparison database in real time through the message queue, and the first-party target data of the first-party system and the second-party target data of the second-party system are stored in the comparison database according to the binary log messages.

3. The data comparison method according to claim 2, characterized in that: According to the binary log message, the first-party target data of the first-party system and the second-party target data of the second-party system are stored in the comparison database, including: Normalizing the first-party initial data carried by the binary log message originating from the first-party system to obtain the first-party target data; Normalizing the second-party initial data carried by the binary log message originating from the second-party system to obtain the second-party target data; The first-party target data and the second-party target data are stored in the comparison database.

4. The data comparison method according to claim 1, characterized in that: The target comparison task rules include a first comparison rule and a second comparison rule; through the target comparison task, based on the pre-configured target comparison task rules, the first-party target data and the second-party target data are rule filtered to obtain the first-party comparison data and the second-party comparison data, including: Through the target comparison task, based on the first comparison rule and a preset comparison time cursor, the first party target data and the second party target data within a preset time range are read from the comparison database respectively; Through the target comparison task, based on the second comparison rule, the first-party comparison data whose business type matches the comparison requirements in the target business scenario is filtered out from the first-party target data within the preset time range, and the second-party comparison data whose business type matches the comparison requirements is filtered out from the second-party target data within the preset time range.

5. The data comparison method according to claim 1, characterized in that: The method further comprises: According to the data in the second result database, an alarm message is pushed to the associated business party.

6. A data comparison device, characterized in that: include: Data reading module, data filtering module, batch comparison module and item-by-item comparison module; among which: The data reading module is used to execute a target comparison task pre-configured for a target business scenario, so as to read the first-party target data and the second-party target data from the comparison database through the target comparison task; wherein the comparison database obtains the first-party target data from the first-party system and obtains the second-party target data from the second-party system; The data filtering module is used to perform rule filtering on the first-party target data and the second-party target data through the target comparison task based on pre-configured target comparison task rules to obtain first-party comparison data and second-party comparison data; The batch comparison module is used to generate a first comparison element according to the first comparison data, generate a second comparison element according to the second comparison data, perform batch comparison on the first comparison element and the second comparison element according to the difference between two sets of the open source tool library guava, and store the comparison elements of both parties with differences between the first comparison element and the second comparison element in the first result database; The item-by-item comparison module is used to compare the new comparison elements with the corresponding comparison elements in the first result database item by item to obtain a comparison result if the missing new comparison elements are received within a preset review time period after the batch comparison is completed due to the missing data of one party. If the comparison result indicates that the corresponding comparison elements are consistent with the new comparison elements, the corresponding comparison elements are removed from the first result database. Through the target comparison task, the first result database is scanned according to the preset scanning cycle, and the comparison elements retained in the first result database are stored in the second result database. The interval time period of the scanning cycle is longer than the review time period.

7. A business data comparison system, characterized in that: include: A comparison database, a first result database, and a server device; the server device is connected to the comparison database and the first result database respectively; The server device is used to execute the data comparison method according to any one of claims 1 to 5.

8. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the data comparison method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data comparison method according to any one of claims 1 to 5 are implemented.

Images