Cross-data-source data difference detection method and device, computer equipment and medium

Abstract

The invention relates to a cross-data-source data difference detection method and device, computer equipment and a medium. The method comprises the steps that configuration information of a current difference detection task is acquired; determining a plurality of data sources based on the data source information, and converting target data from the plurality of data sources into a unified data model according to a field mapping rule; performing difference detection on the unified data model according to the comparison logic to obtain a difference detection result; and if any comparison task detection abnormality occurs in the difference detection result, generating a difference detection report according to the target data of the comparison task detection abnormality. And if the comparison logic comprises a plurality of comparison tasks, performing difference detection on the unified data model in an asynchronous calculation mode. In this way, total number comparison, detail difference analysis and field value difference check can be carried out on data in different data sources, and the accuracy and efficiency of cross-data-source data difference detection are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of database technology, and in particular to a method, apparatus, computer equipment and medium for detecting data differences across data sources. Background Art

[0002] In today's information technology field, information system architecture generally faces the problem of decentralized data storage. These data are stored in different types of data sources, such as relational databases, NoSQL databases, and big data platforms. Since these data sources may be asynchronous when updating operations, and their respective data models are different, this leads to data inconsistency. Data inconsistency not only affects the accuracy of the data, but also in business scenarios involving capital flow, once data inconsistency occurs, it may cause serious product experience problems and property loss problems. This is a huge challenge for enterprises because it is not only related to user experience, but also to the economic interests and market reputation of enterprises. Therefore, when designing information system architecture, how to effectively manage and synchronize data in different data sources to ensure data consistency and accuracy has become a key issue that needs to be solved urgently. This is of great significance for maintaining the stability of enterprise operations and improving user satisfaction. To solve this problem, innovative technical solutions are needed to achieve real-time synchronization and consistency assurance of data, thereby reducing or avoiding various problems caused by data inconsistency. Summary of the invention

[0003] In response to the above-mentioned deficiencies or shortcomings, the present application provides a method, apparatus, computer equipment and medium for detecting data differences across data sources, which can realize total comparison, detailed difference analysis and field value difference check of data in different data sources, thereby improving the accuracy and efficiency of detecting data differences across data sources.

[0004] According to a first aspect, the present application provides a method for detecting data differences across data sources, the method comprising:

[0005] Get the configuration information of the current difference detection task, including data source information, field mapping rules, and comparison logic;

[0006] Determine multiple data sources based on the data source information, and convert target data from the multiple data sources into a unified data model according to field mapping rules;

[0007] Perform difference detection on the unified data model according to the comparison logic to obtain a difference detection result; the comparison logic includes one or more comparison tasks of total number comparison, detailed difference analysis and field value difference check of the target data in the unified data model;

[0008] If any comparison task detection anomaly appears in the difference detection result, a difference detection report is generated based on the target data of the comparison task detection anomaly;

[0009] The unified data model is tested for differences based on comparison logic, including:

[0010] If the comparison logic includes multiple comparison tasks, the difference detection is performed on the unified data model by asynchronously calculating the message queues of the multiple comparison tasks.

[0011] In some embodiments, obtaining configuration information of the current difference detection task includes:

[0012] In response to a difference detection start instruction from a user, executing a Redis SETNX command for the difference detection task to set a corresponding key-value pair to be used as a target task lock;

[0013] After determining that the target task lock has been successfully set, the target data is extracted from multiple data sources based on a predefined unified data access interface and data extraction method.

[0014] In some embodiments, the types of the multiple data sources include MySQL, NoSQL, and PostgreSQL; converting the target data from the multiple data sources into a unified data model according to the field mapping rule includes:

[0015] Generate multiple corresponding data tables based on the target data according to the field mapping rules;

[0016] Identify the data source type corresponding to each data table, use the Java Database Connectivity Driver (JDBC) to establish a connection with MySQL and PostgreSQL data tables, and use a specific Application Programming Interface (API) connector to establish a connection with NoSQL data tables;

[0017] Use SQL tools to extract data from data tables with data source types of MySQL and PostgreSQL, and use specific API calls to extract data from data tables with data source types of NoSQL;

[0018] Based on the plurality of first data extracted from data tables of various data source types, the plurality of first data and their corresponding field mapping rules are aggregated using a set ETL tool; wherein the types of the field mapping rules include MySQL enumeration type ENUM (Enumerated Type), NoSQL type and PostgreSQL TEXT type;

[0019] The aggregated multiple first data and their corresponding field mapping rules are loaded into the set open data processing service ODPS (Open Data Processing Service) to perform data mapping synchronization, conversion and obtain a unified data model.

[0020] In some embodiments, the unified data model is detected for differences by asynchronously computing message queues of multiple comparison tasks, including:

[0021] Use the API provided by ODPS to create multiple temporary tables according to the number of items in the comparison task;

[0022] Query and determine the data source type of each data table in the unified data model;

[0023] Write the column data in each data table into multiple temporary tables according to the data source type of each data table;

[0024] Based on the specified message queue system, encapsulate the column data written into multiple temporary tables into messages and send them to the message queue;

[0025] Use the specified asynchronous computing framework to perform corresponding comparison tasks on the messages from the same temporary table in the message queue in turn to obtain the difference detection results.

[0026] In some embodiments, the designated message queue system is any one of RabbitMQ, Kafka and RocketMQ, and the designated asynchronous computing framework is Java's CompletableFuture or Spring's @Async annotation; and the column data in each data table is written into multiple temporary tables according to the data source type of each data table, including:

[0027] For a type of data table whose data source type is MySQL or PostgreSQL, use SQL tools to write column data in the type of data table into multiple temporary tables;

[0028] And for another part of the second-category data tables whose data source type is NoSQL, use specific API calls to write the column data in the second-category data tables into multiple temporary tables.

[0029] In some embodiments, generating a difference detection report based on the target data detected abnormally by the comparison task includes:

[0030] According to the difference detection results, determine the temporary table for the comparison task to detect anomalies from various temporary tables;

[0031] Use the API provided by ODPS to pull the target data from the temporary table of the comparison task to detect anomalies, and generate a difference detection report based on the pulled target data.

[0032] In some embodiments, after generating a difference detection report according to the target data for detecting abnormalities in the comparison task, the method further includes:

[0033] Delete the corresponding key KEY under the SETNX command to release the target task lock.

[0034] According to a second aspect, the present application provides a data difference detection device across data sources, the device comprising:

[0035] The configuration information acquisition module is used to obtain the configuration information of the current difference detection task, which includes data source information, field mapping rules and comparison logic;

[0036] A data model conversion module, used to determine multiple data sources based on the data source information, and convert the target data from the multiple data sources into a unified data model according to the field mapping rules;

[0037] The data difference detection module is used to perform difference detection on the unified data model according to the comparison logic to obtain the difference detection result; the comparison logic includes one or more comparison tasks of the total number comparison, detailed difference analysis and field value difference check of the target data in the unified data model;

[0038] A detection report generation module is used to generate a difference detection report according to the target data of the comparison task detection anomaly if any comparison task detection anomaly appears in the difference detection result;

[0039] The data difference detection module is also used to:

[0040] If the comparison logic includes multiple comparison tasks, the difference detection is performed on the unified data model by asynchronously calculating the message queues of the multiple comparison tasks.

[0041] According to a third aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the process orchestration method in any one of the above embodiments are implemented.

[0042] According to a fourth aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the process orchestration methods in the above-mentioned embodiments when executed.

[0043] The data difference detection method of the above embodiment can be applied to a cloud platform. The cloud platform first obtains the configuration information of the current difference detection task, and the configuration information includes data source information, field mapping rules and comparison logic. Then, multiple data sources are determined based on the data source information, and the target data from multiple data sources are converted into a unified data model according to the field mapping rules. Then, the unified data model is subjected to difference detection according to the comparison logic to obtain a difference detection result. Among them, the comparison logic includes one or more comparison tasks in the total number comparison, detailed difference analysis and field value difference check of the target data in the unified data model. Finally, if any comparison task detection anomaly appears in the difference detection result, a difference detection report is generated according to the target data of the comparison task detection anomaly. If the comparison logic includes multiple comparison tasks, the unified data model is subjected to difference detection by asynchronously calculating the message queues of the multiple comparison tasks. Through the steps of the above method, the total number comparison, detailed difference analysis and field value difference check of the data in different data sources can be realized, thereby improving the accuracy and efficiency of data difference detection across data sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of an application environment of a data difference detection method across data sources in one or more embodiments of the present application;

[0045] Figure 2 A flow chart of a method for detecting data differences across data sources in one or more embodiments of the present application;

[0046] Figure 3 A flow chart of a method for setting a target task lock in one or more embodiments of the present application;

[0047] Figure 4 A flowchart of a method for converting target data from multiple data sources into a unified data model according to a field mapping rule in one or more embodiments of the present application;

[0048] Figure 5 A flowchart of a method for detecting differences in a unified data model by asynchronously calculating message queues of multiple comparison tasks in one or more embodiments of the present application;

[0049] Figure 6 A flowchart of a method for writing column data in each data table into multiple temporary tables according to the data source type of each data table in one or more embodiments of the present application;

[0050] Figure 7 A flow chart of a method for generating a difference detection report based on detecting abnormal target data in a comparison task in one or more embodiments of the present application;

[0051] Figure 8 This is a schematic diagram of a difference detection service architecture in one or more embodiments of the present application;

[0052] Fig. 9 This is an example diagram of a data difference detection process across data sources in another embodiment of the present application;

[0053] Fig.10 A structural diagram of a data difference detection device across data sources in one or more embodiments of the present application;

[0054] Fig.11 This is a diagram of the internal structure of a computer device in one or more embodiments of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The present application provides a method for detecting data differences across data sources, which can be applied to Figure 1 In the cloud environment shown, the user terminal (101) can first send a difference detection start instruction to the cloud server (102), so that the cloud server (102) starts the data difference detection process described in the present application. Then the cloud server (102) obtains data from different data sources from the database cluster (103) to perform data difference detection. Finally, the cloud server (102) generates a difference detection report based on the difference detection result and feeds it back to the user terminal (101).

[0057] In some exemplary embodiments of the present application, Figure 2 As shown, a method for detecting data differences across data sources is provided, which can be applied to Figure 1 The cloud server in the example is used as an example to illustrate the process, including the following steps:

[0058] Step 201: Obtain configuration information of the current difference detection task, where the configuration information includes data source information, field mapping rules, and comparison logic.

[0059] Specifically, the current difference detection task refers to a difference detection task that is triggered by a timer or a user actively triggers a difference detection start instruction. Each difference detection task has corresponding configuration information pre-configured by the user. Among them, the data source information is used to determine the database from which the user specifies to obtain data, and the field mapping rule refers to the process of converting data from one format or structure to another format or structure. The comparison logic includes one or more comparison tasks specified by the user.

[0060] For example, suppose the data source information includes MySQL and PostgreSQL, the field mapping rule is "eliminate value", and the comparison task of the comparison logic is "detailed difference analysis". It is known that the data obtained from the database MySQL is UTF-8, and the data obtained from PostgreSQL is ISO-8859-1. According to the above field mapping rules, UTF and ISO are converted. Obviously, UTF and ISO are inconsistent, and the comparison task of "detailed difference analysis" detects anomalies.

[0061] In some embodiments, the configuration information of the current difference detection task is obtained, such as Figure 3 As shown, including:

[0062] Step 301: In response to a difference detection start instruction from a user, a Redis SETNX command is executed for a difference detection task to set a corresponding key-value pair to be used as a target task lock.

[0063] The key-value pair can be key 1 and key 0. When the cloud server uses the SETNX command to try to set the lock, if the key-value pair key 1 is returned, it means that the target task lock is successfully set and the current difference detection task obtains the target task lock, which is used to prevent repeated execution of the difference detection task.

[0064] Step 302: After determining that the target task lock has been successfully set, extract target data from multiple data sources based on a predefined unified data access interface and data extraction method.

[0065] Among them, the pre-defined unified data access interface generally includes common data operation methods, such as addition, deletion, modification and query, etc. Multiple data sources can come from databases MySQL, NoSQL and PostgreSQL respectively. The data extraction method can be to set up MySQL adapter, MongoDB adapter and PostgreSQL adapter respectively to extract target data from these databases.

[0066] For example, the cloud server sets the MySQL adapter through the "import" command, extracts data from the database MySQL as UTF-8, and then converts UTF-8 to UTF according to the delete command included in the pre-defined unified data access interface to serve as the extracted target data.

[0067] The above data difference detection method ensures the uniqueness and consistency of the difference detection task by introducing a task lock mechanism, thereby avoiding resource waste and data errors caused by repeated execution.

[0068] Step 202: Determine multiple data sources based on the data source information, and convert target data from the multiple data sources into a unified data model according to field mapping rules.

[0069] Generally, the target data from multiple data sources are of similar type and are used for difference detection.

[0070] For example, the data source information may include identification information of three types of databases: MySQL, NoSQL, and PostgreSQL, indicating that the cloud server should extract target data from the databases MySQL, NoSQL, and PostgreSQL. Assuming that the data obtained from the MySQL database is UTF-8 and the data obtained from PostgreSQL is UTO-8859-1, after conversion according to the above field mapping rules, UTF and UTO are obtained. The cloud server then enters UTF and UTO into the unified data model to perform the next step of difference detection.

[0071] In some embodiments, the types of the multiple data sources include MySQL, NoSQL, and PostgreSQL; the target data from the multiple data sources are converted into a unified data model according to the field mapping rules, such as Figure 4 As shown, including:

[0072] Step 401: Generate multiple corresponding data tables based on the target data according to the field mapping rules.

[0073] Among them, each data table contains multiple data from the same data source. If the data comes from the database MySQL, the data source type of the data is MySQL; similarly, for data from the databases NoSQL and PostgreSQL, their data source types are NoSQL and PostgreSQL respectively.

[0074] For example, the cloud server extracts multiple target data PWD123, PWD456 and PWD789 from the MySQL database. These data actually constitute a user information table, that is, the above-mentioned data table A. The cloud server extracts multiple target data user123, user456 and user789 from the PostgreSQL database. These data actually constitute another user information table, that is, another data table B.

[0075] Step 402: Identify the data source type corresponding to each data table, use JDBC to establish a connection with a data table of MySQL or PostgreSQL type, and use a specific API connector to establish a connection with a data table of NoSQL type.

[0076] Specifically, JDBC is a Java API that can be used to connect and execute queries to databases such as MySQL and PostgreSQL.

[0077] For example, if data table A from MySQL, data table B from PostgreSQL, and data table C from NoSQL are known, the cloud server can use JDBC to establish connections with data table A and data table B using corresponding commands. Then, for data table C, the API connector can execute corresponding commands to establish a connection with data table C. Among them, the database MongoDB is a specific type of database NoSQL.

[0078] Step 403: Use SQL tools to extract data from data tables whose data source types are MySQL and PostgreSQL, and use specific API calls to extract data from data tables whose data source type is NoSQL.

[0079] For example, the cloud server can use SQL tools to extract data from data table A and data table B by executing corresponding commands. Then the cloud server can extract data from data table C by executing corresponding commands through specific API calls.

[0080] Step 404: Based on the multiple first data extracted from the data tables of various data source types, the multiple first data and their corresponding field mapping rules are aggregated using a set ETL tool.

[0081] Among them, the types of field mapping rules include ENUM and NoSQL types of MySQL and TEXT types of PostgreSQL. The field mapping rules will specify how to map the source field of the first data to the target field. The first data extracted from each data table is generally consistent with the above-mentioned target data.

[0082] For example, the value stored in the ENUM type field can be in string form, the field in the NoSQL document can be a JSON object, and the TEXT type field can store text data. Then, when the cloud server uses ETL, in addition to mapping the source fields of each first data to the target field according to the field mapping rules, it also needs to complete the aggregation and conversion of the ENUM type field through the corresponding command, and then execute another unified command to complete the aggregation and conversion of the NoSQL type data and the PostgreSQL TEXT type data respectively.

[0083] Step 405: Load the aggregated multiple first data and their corresponding field mapping rules into the set ODPS to perform data mapping synchronization, convert and obtain a unified data model.

[0084] For example, the cloud server may use MaxCompute SQL tasks to implement a loading operation of multiple first data and their corresponding field mapping rules, and then execute corresponding commands to complete a conversion operation of the multiple first data and their corresponding field mapping rules.

[0085] In the above-mentioned data difference detection method, since it can process various types of data sources including MySQL, NoSQL and PostgreSQL, it enhances the flexibility and scalability of the system, provides strong support for big data analysis, and improves the system's ability to process and analyze large-scale data.

[0086] Step 203: Perform difference detection on the unified data model according to the comparison logic to obtain a difference detection result; the comparison logic includes one or more comparison tasks of total number comparison, detailed difference analysis and field value difference check of the target data in the unified data model.

[0087] The difference detection result includes the comparison task results between the above-mentioned data tables.

[0088] For example, it is known that the price comparison logic includes total number comparison and detailed difference analysis. For the above-mentioned data table A including target data PWD123, PWD456 and PWD789, and data table B including target data user123, user456 and user789, after difference detection on the two data tables, it is found that the total number of data in the two tables is consistent, but there is an exception in the comparison task detection of "detailed difference analysis".

[0089] Step 204: If any comparison task detection anomaly appears in the difference detection result, a difference detection report is generated according to the target data of the comparison task detection anomaly.

[0090] The number of items for detecting anomalies in the comparison task can be multiple.

[0091] For example, assuming that the price comparison logic includes total comparison, detailed difference analysis, and field value difference check, for data table A including target data PWD123 and PWD456, and data table B including target data user123, user456, and user789, after performing difference detection on the two data tables, it is found that there are anomalies in the three comparison tasks of "total comparison", "detailed difference analysis", and "field value difference check". In this way, it is possible to compare the total number, analyze the detailed difference, and check the field value difference of data in different data sources, thereby improving the accuracy and efficiency of cross-data source data difference detection.

[0092] Step 205: If the comparison logic includes multiple comparison tasks, the unified data model is detected for differences by asynchronously computing the message queues of the multiple comparison tasks.

[0093] Specifically, only when the price comparison logic includes a single item among total comparison, detail difference analysis and field value difference check, the cloud server does not need to use asynchronous computing to perform difference detection on the unified data model.

[0094] In some embodiments, the unified data model is detected for differences by asynchronously computing message queues of multiple comparison tasks, such as Figure 5 As shown, including:

[0095] Step 501: Create multiple temporary tables using the API provided by ODPS according to the number of items in the comparison task.

[0096] The number of temporary tables can be the same as that of data tables, or a single temporary table can be composed of multiple data tables.

[0097] Step 502: Query and determine the data source type of each data table in the unified data model.

[0098] Among them, it can be known from the above that the data source type of each data table can be MySQL, NoSQL and PostgreSQL respectively.

[0099] Step 503: writing the column data in each data table into multiple temporary tables according to the data source type of each data table.

[0100] In some embodiments, the specified message queue system is any one of RabbitMQ, Kafka and RocketMQ, and the specified asynchronous computing framework is Java's CompletableFuture or Spring's @Async annotation; the column data in each data table is written into multiple temporary tables according to the data source type of each data table, such as Figure 6 As shown, including:

[0101] Step 601: For a type of data table whose data source type is MySQL or PostgreSQL, use an SQL tool to write column data in the type of data table into multiple temporary tables.

[0102] For example, the cloud server can use the SQL tool to write column data in a data table of a type whose data source type is MySQL or PostgreSQL into a corresponding temporary table by executing the same command.

[0103] Step 602: For another part of the second-category data tables whose data source type is NoSQL, use a specific API call to write the column data in the second-category data tables into multiple temporary tables.

[0104] For example, the cloud server may use a Java driver called by a specific API to write column data in a second-category data table whose data source type is NoSQL into a corresponding temporary table by executing corresponding commands.

[0105] Step 504: Based on the specified message queue system, encapsulate the column data written into the multiple temporary tables into a message and send it to the message queue.

[0106] Specifically, the cloud server can choose to use the message queue systems of RabbitMQ, Kafka and RocketMQ.

[0107] For example, assuming that the cloud server chooses to use RabbitMQ as the message queue system, the cloud server first initializes the RabbitMQ connection and channel, then reads data from the temporary table of MySQL, and encapsulates the first column of each row into a message and sends it to the RabbitMQ queue.

[0108] Step 505: Use the specified asynchronous computing framework to sequentially execute corresponding comparison tasks on the messages from the same temporary table in the message queue to obtain difference detection results.

[0109] For example, assuming that the cloud server chooses to use RabbitMQ as the message queue system, the cloud server can choose to use CompletableFuture to implement the asynchronous computing framework.

[0110] Specifically, the cloud server first initializes the RabbitMQ connection and channel, then defines an asynchronous comparison task DataComparisonTask, which uses CompletableFuture to asynchronously execute the comparison task. Finally, the cloud server creates a message consumer MessageConsumer, which listens to messages in the RabbitMQ queue and asynchronously executes the comparison task when a message is received.

[0111] The above data difference detection method, by combining asynchronous computing and message queue technology, provides an efficient, flexible and scalable method to detect differences in the unified data model of multiple data sources.

[0112] In some embodiments, a difference detection report is generated based on the target data detected abnormally by the comparison task, such as Figure 7 As shown, including:

[0113] Step 701: According to the difference detection result, determine the temporary table for detecting anomalies in the comparison task from various temporary tables.

[0114] Specifically, as can be seen from the above, the number of temporary tables can be consistent with the number of data tables, or a single temporary table can be composed of multiple data tables. The difference is that each temporary table is newly created through the ODPS API.

[0115] Assume that the price comparison logic includes total comparison, detailed difference analysis, and field value difference check. For temporary table A including target data PWD123 and PWD456, and temporary table B including target data user123, user456, and user789, difference detection is performed on temporary table A respectively. It is found that there are anomalies in the three comparison task detections of "total comparison", "detailed difference analysis", and "field value difference check".

[0116] Step 702: Use the API provided by ODPS to pull the target data from the temporary table of the comparison task anomaly detection, and generate a difference detection report based on the pulled target data.

[0117] The cloud server then uses the API provided by ODPS to pull target data PWD123, PWD456, user123, user456 and user789 from temporary tables A and B to generate a difference detection report.

[0118] The above data difference detection method provides an efficient, flexible and accurate data anomaly detection solution by introducing dynamic temporary table management, multi-dimensional difference detection logic and automatic report generation.

[0119] In some embodiments, after generating a difference detection report according to the target data for detecting abnormalities in the comparison task, the method further includes:

[0120] Delete the corresponding key KEY under the SETNX command to release the target task lock.

[0121] Specifically, as can be seen from the above, when the cloud server uses the SETNX command to try to set the lock, if the key-value pair key1 is returned, it means that the target task lock is successfully set. After the cloud server completes the comparison task and generates the difference detection report, it can delete the corresponding key 1 by calling the releaseLock method to release the target task lock. In this way, the uniqueness and consistency of the difference detection task are ensured, and resource waste and data errors caused by repeated execution are avoided.

[0122] It should be noted that, with respect to the various steps included in the data difference detection method across data sources provided in any of the above embodiments, unless otherwise clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of these steps can include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0123] In another embodiment of the present application, Figure 8 , 9 As shown, the data difference detection method across data sources can be applied to Figure 1 The cloud server 102 shown in FIG. Figure 8 In the difference detection business architecture shown, the user end 101 can first trigger the difference detection start instruction in the web end to instruct the cloud server 102 to start the data difference detection process described in this application. After determining that the task lock is set successfully, the cloud server 102 first queries the configuration information of the current difference detection task, which includes data source information, field mapping rules and comparison logic. Then, the data source abstract layer module and the data model mapping layer are successively used to determine multiple data sources based on the data source information, and the target data from multiple data sources are converted into a unified data model according to the field mapping rules. Then, the data comparison engine is used to perform difference detection on the unified data model according to the comparison logic to obtain the difference detection result. Among them, the comparison logic in the data comparison engine includes one or more comparison tasks in the total number comparison, detailed difference analysis and field value difference check of the target data in the unified data model. Finally, if any comparison task detection exception appears in the difference detection result, the report module is used to generate a difference detection report according to the target data of the comparison task detection exception, end the difference detection task and release the task lock. Through the steps of the above method, it is possible to compare the total number of data in different data sources, analyze the detailed differences and check the field value differences, thereby improving the accuracy and efficiency of data difference detection across data sources.

[0124] According to a second aspect, the present application provides a data difference detection device across data sources, such as Fig.10 As shown, the device comprises:

[0125] The configuration information acquisition module 110 is used to acquire the configuration information of the current difference detection task, and the configuration information includes data source information, field mapping rules and comparison logic.

[0126] The data model conversion module 120 is used to determine multiple data sources based on the data source information, and convert the target data from the multiple data sources into a unified data model according to the field mapping rules.

[0127] The data difference detection module 130 is used to perform difference detection on the unified data model according to the comparison logic to obtain the difference detection result; the comparison logic includes one or more comparison tasks of the total number comparison, detailed difference analysis and field value difference check of the target data in the unified data model.

[0128] The detection report generating module 140 is used to generate a difference detection report according to the target data of the comparison task detection anomaly if any comparison task detection anomaly appears in the difference detection result.

[0129] The data difference detection module 150 is further configured to detect differences in the unified data model by asynchronously calculating message queues of the plurality of comparison tasks if the comparison logic includes a plurality of comparison tasks.

[0130] In some embodiments, the configuration information acquisition module 110 is also used to respond to a difference detection start instruction from the user, execute the Redis SETNX command for the difference detection task to set the corresponding key-value pair to be used as the target task lock; after determining that the target task lock has been set successfully, extract the target data from multiple data sources based on a predefined unified data access interface and data extraction method.

[0131] In some embodiments, the types of multiple data sources include MySQL, NoSQL, and PostgreSQL. The data model conversion module 120 is also used to generate multiple corresponding data tables according to the field mapping rules for the target data; identify the data source type corresponding to each data table, use JDBC to establish a connection with the data table of MySQL and PostgreSQL types, and use a specific API connector to establish a connection with the data table of NoSQL type; use SQL tools to extract data from data tables with data source types of MySQL and PostgreSQL, and use specific API calls to extract data from data tables with data source types of NoSQL; based on the multiple first data extracted from the data tables of various data source types, use the set ETL tool to aggregate the multiple first data and their corresponding field mapping rules; wherein the types of field mapping rules include MySQL's ENUM type, NoSQL type, and PostgreSQL's TEXT type; load the aggregated multiple first data and their corresponding field mapping rules into the set ODPS for data mapping synchronization, conversion, and obtaining a unified data model.

[0132] In some embodiments, the data difference detection module 130 is also used to create multiple temporary tables using the API provided by ODPS according to the number of projects in the comparison task; query and determine the data source type of each data table in the unified data model; write the column data in each data table into multiple temporary tables according to the data source type of each data table; based on the specified message queue system, encapsulate the column data written into multiple temporary tables into messages and send them to the message queue; use the specified asynchronous computing framework to execute corresponding comparison tasks on the messages from the same temporary table in the message queue in turn to obtain difference detection results.

[0133] In some embodiments, the designated message queue system is any one of RabbitMQ, Kafka and RocketMQ, and the designated asynchronous computing framework is Java's CompletableFuture or Spring's @Async annotation; the data difference detection module 130 is also used for, for a type of data table whose data source type is MySQL or PostgreSQL, using SQL tools to write the column data in the type of data table into multiple temporary tables; and for another type of data table whose data source type is NoSQL, using specific API calls to write the column data in the type of data table into multiple temporary tables.

[0134] In some embodiments, the detection report generation module 140 is also used to determine the temporary table for detecting anomalies of the comparison task from various temporary tables based on the difference detection results; use the API provided by ODPS to pull target data from the temporary table for detecting anomalies of the comparison task, and generate a difference detection report based on the pulled target data.

[0135] In some embodiments, after generating a difference detection report based on the target data for detecting abnormalities in the comparison task, the configuration information acquisition module 110 is further used to delete the corresponding key KEY under the SETNX command to release the target task lock.

[0136] For the specific limitations of the data difference detection device applicable to cross-data sources, please refer to the limitations of the data difference detection method applicable to cross-data sources above, which will not be repeated here. Each module in the above-mentioned data difference detection device applicable to cross-data sources can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0137] According to a third aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the process orchestration method in any one of the above embodiments are implemented.

[0138] According to a fourth aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the process orchestration methods in the above-mentioned embodiments when executed.

[0139] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to data difference detection. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, any of the above-mentioned data difference detection methods across data sources is implemented.

[0140] Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (RamCUs), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

[0143] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for detecting data differences across data sources, characterized in that: The method comprises: Obtain configuration information of the current difference detection task, the configuration information including data source information, field mapping rules and comparison logic; Determine multiple data sources based on the data source information, and convert target data from the multiple data sources into a unified data model according to the field mapping rule; Performing difference detection on the unified data model according to the comparison logic to obtain a difference detection result; the comparison logic includes one or more comparison tasks of total number comparison, detailed difference analysis and field value difference check of target data in the unified data model; If any comparison task detection anomaly appears in the difference detection result, a difference detection report is generated according to the target data of the comparison task detection anomaly; The performing difference detection on the unified data model according to the comparison logic further includes: If the comparison logic includes multiple comparison tasks, the difference detection is performed on the unified data model by asynchronously calculating the message queues of the multiple comparison tasks.

2. The method according to claim 1, characterized in that The obtaining of configuration information of the current difference detection task includes: In response to a difference detection start instruction from a user, executing a Redis SETNX command for the difference detection task to set a corresponding key-value pair to be used as a target task lock; After determining that the target task lock has been successfully set, the target data is extracted from the multiple data sources based on a predefined unified data access interface and data extraction method.

3. The method according to claim 2, characterized in that The types of the multiple data sources include MySQL, NoSQL and PostgreSQL; and converting the target data from the multiple data sources into a unified data model according to the field mapping rule includes: Generate a plurality of corresponding data tables according to the field mapping rules for the target data; Identify the data source type corresponding to each of the data tables, use a Java database connection driver to establish a connection with a MySQL or PostgreSQL type data table, and use a specific application programming interface connector to establish a connection with a NoSQL type data table; Use SQL tools to extract data from tables with data sources of MySQL and PostgreSQL, and use specific application programming interface calls to extract data from tables with data sources of NoSQL; Based on the plurality of first data extracted from data tables of various data source types, the plurality of first data and their corresponding field mapping rules are aggregated using a set ETL tool; wherein the types of the field mapping rules include an enumeration type of MySQL, a NoSQL type, and a TEXT type of PostgreSQL; The aggregated multiple first data and their corresponding field mapping rules are loaded into the set open data processing service to perform data mapping synchronization, conversion and obtain the unified data model.

4. The method according to claim 3, characterized in that The detecting differences of the unified data model by asynchronously calculating the message queues of the plurality of comparison tasks includes: According to the number of items in the comparison task, a plurality of temporary tables are created using an application programming interface provided by the open data processing service; Query and determine the data source type of each of the data tables in the unified data model; Writing the column data in each of the data tables into the multiple temporary tables according to the data source type of each of the data tables; Based on a specified message queue system, encapsulate the column data written into the multiple temporary tables into messages and send them to the message queue; Use the specified asynchronous computing framework to perform corresponding comparison tasks on the messages from the same temporary table in the message queue in turn to obtain the difference detection results.

5. The method according to claim 4, characterized in that The specified message queue system is any one of RabbitMQ, Kafka and RocketMQ, and the specified asynchronous computing framework is Java's CompletableFuture or Spring's @Async annotation; Writing the column data in each of the data tables into the multiple temporary tables according to the data source type of each of the data tables includes: For a data table of type MySQL or PostgreSQL, using an SQL tool to write column data in the data table of type MySQL or PostgreSQL into the multiple temporary tables; And for another part of the second-category data tables whose data source type is NoSQL, a specific application programming interface call is used to write the column data in the second-category data tables into the multiple temporary tables.

6. The method according to claim 4, characterized in that The generating of a difference detection report according to the target data detected abnormally by the comparison task includes: According to the difference detection result, determining a temporary table for detecting anomalies in the comparison task from each of the temporary tables; The target data is pulled from the temporary table of the comparison task detection anomaly using an application programming interface provided by the open data processing service, and a difference detection report is generated according to the pulled target data.

7. The method according to claim 2, characterized in that After generating a difference detection report based on the target data for detecting abnormalities in the comparison task, the method further includes: Delete the corresponding key under the SETNX command to release the target task lock.

8. A data difference detection device across data sources, characterized in that: The device includes: A configuration information acquisition module is used to acquire the configuration information of the current difference detection task, wherein the configuration information includes data source information, field mapping rules and comparison logic; A data model conversion module, used to determine multiple data sources based on the data source information, and convert the target data from the multiple data sources into a unified data model according to the field mapping rule; A data difference detection module, used to perform difference detection on the unified data model according to the comparison logic to obtain a difference detection result; the comparison logic includes one or more comparison tasks of total number comparison, detailed difference analysis and field value difference check of target data in the unified data model; A detection report generating module, configured to generate a difference detection report according to target data of the comparison task detection anomaly if any comparison task detection anomaly appears in the difference detection result; The data difference detection module is further used for: If the comparison logic includes multiple comparison tasks, the difference detection is performed on the unified data model by asynchronously calculating the message queues of the multiple comparison tasks.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images