Data comparison method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of data analysis, and relates to a data comparison method and device, computer equipment and a storage medium, and the method comprises the steps: receiving a data comparison request sent by a request terminal; constructing a data comparison task corresponding to the data source information according to a pre-configured comparison rule file; performing data conversion operation on the data source information according to a data conversion mode corresponding to the type of the data source information to obtain standard data source information; performing data preprocessing operation on the standard data source information to obtain preprocessed data source information; performing data comparison operation on the preprocessed data source information according to the metadata description to obtain a data comparison result; and carrying out difference data identification operation on the data comparison result to obtain a difference data identification result. According to the method, the user or the system administrator is effectively helped to identify and analyze the difference between the data, so that the accuracy and consistency of the data are ensured.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, applicable to the financial or medical fields, and in particular to a data comparison method, device, computer equipment and storage medium. Background Art

[0002] In the existing data comparison technology solutions, usually two sets of data information that need to be compared on the same benchmark are obtained by executing SQL statements in the EXCEL configuration file. Then the results of the comparison with differences are output to the configured file. Or after the data after complex processing logic is exported to the file, the data files are compared.

[0003] However, the applicant found that the current implementation method, although it can be compared after some logical processing through SQL, its main disadvantage is that it can only compare relatively intuitive data. If it involves large fields (i.e., text content with complex meanings), there is no way to compare through SQL. Or the large field content is parsed into a specified data file through a hard-coded program and then compared, which is not flexible enough. Once there is a change, the processing logic needs to be adjusted again through hard coding. These two forms are not flexible enough for comparison of complex scenarios and specialized business scenarios, and their applicability is not high. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a data comparison method, apparatus, computer equipment and storage medium to solve the problem that traditional data comparison methods cannot guarantee the accuracy and consistency of data when applied in complex scenarios.

[0005] In order to solve the above technical problems, the present application embodiment provides a data comparison method, which adopts the following technical solution:

[0006] Receiving a data comparison request sent by a requesting terminal, wherein the data comparison request includes data source information to be compared;

[0007] Constructing a data comparison task corresponding to the data source information according to a pre-configured comparison rule file, wherein the data comparison task includes a metadata description;

[0008] Performing a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information;

[0009] Performing data preprocessing operation on the standard data source information to obtain preprocessed data source information;

[0010] Performing a data comparison operation on the pre-processed data source information according to the metadata description to obtain a data comparison result;

[0011] Performing a difference data identification operation on the data comparison result to obtain a difference data identification result;

[0012] The difference data identification result is output to the requesting terminal.

[0013] Furthermore, the step of performing a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information specifically includes the following steps:

[0014] Constructing a data conversion mapping table according to the data characteristics of the data source information;

[0015] According to the type of the data source information, a corresponding parser is used to perform structural conversion to obtain converted data source information;

[0016] The converted data source information is processed into a unified field type according to the data conversion mapping table to obtain the standard data source information.

[0017] Furthermore, the step of performing a difference data identification operation on the data comparison result to obtain a difference data identification result specifically includes the following steps:

[0018] Performing a similarity calculation operation on the data comparison result according to a cluster analysis algorithm to obtain a similarity calculation result;

[0019] Classify the data comparison results according to the similarity calculation results and a preset similarity threshold to obtain similar comparison data;

[0020] A field difference evaluation operation is performed on the same type of comparison data of the same category according to the minimum edit distance algorithm to obtain the difference data identification result.

[0021] Furthermore, after the step of performing a difference data identification operation on the data comparison result to obtain a difference data identification result, the following step is also included:

[0022] If there is difference data in the difference data identification result, a verification operation is performed on the difference data according to a preset verification rule to obtain a verification result;

[0023] If the verification result does not contain abnormal data, confirming the difference data as normal data;

[0024] If there are abnormal data in the verification results, the data comparison results and the difference data are classified, summarized and statistically analyzed to generate a data comparison report;

[0025] Outputting the data comparison report to the requesting terminal.

[0026] Furthermore, if the verification result contains logically abnormal data, the data comparison result and the difference data are classified, summarized and statistically analyzed to generate a data comparison report, which specifically includes the following steps:

[0027] A data correction operation is performed on the abnormal data according to the data comparison result, the verification result and a preset processing rule.

[0028] Furthermore, if there is difference data in the difference data identification result, the step of performing a verification operation on the difference data according to a preset verification rule to obtain a verification result specifically includes the following steps:

[0029] A logic check operation is performed on the difference data according to a preset logic check rule to obtain the check result.

[0030] Furthermore, if there is difference data in the difference data identification result, the step of performing a verification operation on the difference data according to a preset verification rule to obtain a verification result specifically includes the following steps:

[0031] A business verification operation is performed on the difference data according to a preset business verification rule to obtain the verification result.

[0032] In order to solve the above technical problems, the embodiment of the present application further provides a data comparison device, which adopts the following technical solution:

[0033] A request receiving module, used to receive a data comparison request sent by a request terminal, wherein the data comparison request includes data source information to be compared;

[0034] A task construction module, used to construct a data comparison task corresponding to the data source information according to a pre-configured comparison rule file, wherein the data comparison task includes a metadata description;

[0035] A data conversion module, used to perform a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information;

[0036] A preprocessing module, used to perform data preprocessing operations on the standard data source information to obtain preprocessed data source information;

[0037] A data comparison module, used to perform a data comparison operation on the pre-processed data source information according to the metadata description to obtain a data comparison result;

[0038] A difference data identification module is used to perform a difference data identification operation on the data comparison result to obtain a difference data identification result;

[0039] A result output module is used to output the difference data identification result to the request terminal.

[0040] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0041] It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data comparison method as described above when executing the computer-readable instructions.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0043] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the data comparison method described above are implemented.

[0044] The present application provides a data comparison method, comprising: receiving a data comparison request sent by a request terminal, wherein the data comparison request includes data source information to be compared; constructing a data comparison task corresponding to the data source information according to a pre-configured comparison rule file, wherein the data comparison task includes a metadata description; performing a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information; performing a data preprocessing operation on the standard data source information to obtain preprocessed data source information; performing a data comparison operation on the preprocessed data source information according to the metadata description to obtain a data comparison result; performing a difference data identification operation on the data comparison result to obtain a difference data identification result; and outputting the difference data identification result to the request terminal. Compared with the prior art, the present application effectively helps users or system administrators identify and analyze differences between data, thereby ensuring the accuracy and consistency of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0047] Figure 2 is a flow chart of the implementation of the data comparison method provided in the embodiment of the present application;

[0048] Figure 3 is a schematic diagram of the structure of a data comparison device provided in an embodiment of the present application;

[0049] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0051] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0053] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0054] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0055] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0056] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0057] It should be noted that the data comparison method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data comparison device is generally set in the server / terminal device.

[0058] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0059] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data comparison method according to the present application. The data comparison method comprises: step S201, step S202, step S203, step S204, step S205, step S206 and step S207.

[0060] In step S201, a data comparison request sent by a requesting terminal is received, wherein the data comparison request includes data source information to be compared.

[0061] In the embodiment of the present application, the system first receives a data comparison request from a requesting terminal (eg, a user device, a server or other system component). The request includes the data source information to be compared, that is, the data that the user wants the system to compare and analyze.

[0062] In the embodiment of the present application, the data source information may include key information such as the type, format, storage location, etc. of the data, which is crucial for subsequent data processing.

[0063] In the embodiment of the present application, the user inputs a question or inquiry through their terminal device (such as a mobile phone, computer, etc.), and this input is received by the system. This data source information is the specific content that the user wants the system to compare. Specifically, the data source information can be "transaction data or payment data or business data or purchase data" related to a financial institution (such as a bank, etc.). The data source information can also be medical data related to medical scenarios, such as personal health records, prescriptions, examination reports, etc. It should be understood that the examples of data source information here are only for the convenience of understanding and are not used to limit this application.

[0064] In step S202, a data comparison task corresponding to the data source information is constructed according to a pre-configured comparison rule file, wherein the data comparison task includes a metadata description.

[0065] In an embodiment of the present application, the system constructs a data comparison task corresponding to the received data source information according to a pre-configured comparison rule file (these rules may be formulated by the user or system administrator according to actual needs).

[0066] In the embodiment of the present application, the comparison rule file contains specific standards, methods, parameters, etc. for data comparison, and these rules will guide the system on how to perform the comparison operation.

[0067] In an embodiment of the present application, the data comparison task also includes metadata description, that is, descriptive information about the data structure and content, which is crucial for subsequent data processing and comparison.

[0068] In step S203, a data conversion operation is performed on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information.

[0069] In the embodiment of the present application, since different data sources may have different formats and types, the system needs to select a corresponding data conversion method according to the type of data source information.

[0070] In the embodiment of the present application, the data conversion operation aims to convert the data source information into standard data source information that can be understood and processed by the system, including operations such as format conversion, encoding conversion, and data cleaning.

[0071] In step S204, a data preprocessing operation is performed on the standard data source information to obtain preprocessed data source information.

[0072] In the embodiment of the present application, after the data conversion, the system may need to perform further preprocessing operations on the data to ensure the quality and consistency of the data.

[0073] In an embodiment of the present application, data preprocessing may include steps such as data deduplication, missing value processing, and outlier detection. These operations help to improve the accuracy and reliability of data comparison.

[0074] In step S205, a data comparison operation is performed on the pre-processed data source information according to the metadata description to obtain a data comparison result.

[0075] In an embodiment of the present application, after completing data preprocessing, the system performs a data comparison operation on the preprocessed data source information according to the previously constructed metadata description.

[0076] In the embodiment of the present application, the data comparison operation involves item-by-item comparison of data, pattern matching, statistical analysis, etc., to find out the differences or similarities between the data.

[0077] In step S206, a difference data identification operation is performed on the data comparison result to obtain a difference data identification result.

[0078] In the embodiment of the present application, after obtaining the data comparison result, the system needs to further perform a difference data identification operation on the result.

[0079] In the embodiment of the present application, the difference data identification is intended to identify which data items or data records have differences, and these differences are caused by data errors, data updates or data inconsistencies.

[0080] In step S207, the difference data identification result is output to the requesting terminal.

[0081] In the embodiment of the present application, the system outputs the difference data identification result to the requesting terminal.

[0082] In an embodiment of the present application, these results can be presented in the form of reports, charts, lists, etc., to help users or system administrators understand the differences between the data and make corresponding decisions or adjustments accordingly.

[0083] In an embodiment of the present application, a data comparison method is provided, comprising: receiving a data comparison request sent by a request terminal, wherein the data comparison request includes data source information to be compared; constructing a data comparison task corresponding to the data source information according to a pre-configured comparison rule file, wherein the data comparison task includes a metadata description; performing a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information; performing a data preprocessing operation on the standard data source information to obtain preprocessed data source information; performing a data comparison operation on the preprocessed data source information according to the metadata description to obtain a data comparison result; performing a difference data identification operation on the data comparison result to obtain a difference data identification result; and outputting the difference data identification result to the request terminal. Compared with the prior art, the present application effectively helps users or system administrators identify and analyze differences between data, thereby ensuring the accuracy and consistency of the data.

[0084] In some optional implementations of the embodiments of the present application, the step of performing a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information specifically includes the following steps:

[0085] Construct a data conversion mapping table based on the data characteristics of the data source information;

[0086] According to the type of data source information, a corresponding parser is used to perform structural conversion to obtain converted data source information;

[0087] The converted data source information is processed into a unified field type according to the data conversion mapping table to obtain standard data source information.

[0088] In the embodiment of the present application, the system first needs to conduct an in-depth analysis of the data source information to understand its data characteristics. Data characteristics include data structure (such as table, tree, graph, etc.), data type (such as string, number, date, etc.), data format (such as CSV, JSON, XML, etc.), and the relationship between data.

[0089] In the embodiment of the present application, based on these data features, the system will construct a data conversion mapping table. This mapping table is a detailed guide for instructing how to convert data source information into a standard format that the system can process. The mapping table includes key information such as source field name, source field type, target field name, target field type, conversion rules, etc.

[0090] In the embodiment of the present application, the process of constructing the mapping table can be manually participated to ensure the accuracy and completeness of the mapping. In particular, when the data source information is relatively complex or there are special requirements, manual review and adjustment of the mapping table is particularly important.

[0091] In the embodiment of the present application, after determining the type of data source information, the system will select a suitable parser for structured conversion, wherein the parser is mainly used for tools or libraries for parsing and converting specific types of data. As an example, if the data source information is in JSON format, the system will select a parser that can parse JSON; if the data source information is in CSV format, the system will select a parser that can parse CSV.

[0092] In the embodiment of the present application, the parser converts the data source information into a more easily processed structured format, such as an internal data structure, a database table, etc., according to its structure and content. This conversion process involves data format conversion, data type conversion, data structure adjustment, etc.

[0093] In an embodiment of the present application, after obtaining the converted data source information, the system also needs to unify the field types according to the previously constructed data conversion mapping table, wherein the field type unification processing refers to adjusting the field types in the converted data source information to standard types that the system can handle. This includes converting string type fields to numeric types, converting date and time type fields to a unified date and time format, converting date and time fields of different formats to a unified time zone, etc. Field type unification processing is an important step to ensure data consistency and accuracy. Through this step, the system can more easily compare, analyze and process data.

[0094] In the embodiment of the present application, the data source information after the field type is unified is called standard data source information. This standard data source information can be used by the system for subsequent data processing, analysis, comparison and other operations.

[0095] In an embodiment of the present application, the data source information is converted into a standard format that can be processed by the system to ensure the accuracy and consistency of the data.

[0096] In some optional implementations of the embodiments of the present application, the above step of performing a difference data identification operation on the data comparison result to obtain a difference data identification result specifically includes the following steps:

[0097] Performing a similarity calculation operation on the data comparison result according to the cluster analysis algorithm to obtain a similarity calculation result;

[0098] Classify the data comparison results according to the similarity calculation results and the preset similarity threshold to obtain the same type of comparison data;

[0099] The field difference evaluation operation is performed on the same category of similar comparison data according to the minimum edit distance algorithm to obtain the difference data identification result.

[0100] In the embodiment of the present application, the present application first uses a cluster analysis algorithm to process the data comparison result, wherein the cluster analysis is mainly used to classify similar data points or objects into the same category.

[0101] In the embodiment of the present application, the data comparison result may include the matching degree or difference information between multiple groups of data. The clustering analysis algorithm will divide these data into different groups according to the similarities or differences between them.

[0102] In the embodiments of the present application, similarity calculation is the core part of cluster analysis. It involves using specific metrics (such as Euclidean distance, cosine similarity, Jaccard similarity, etc.) to calculate the similarity between data points. These metrics can be selected according to the specific characteristics and requirements of the data.

[0103] In the embodiment of the present application, the cluster analysis algorithm (such as K-means, hierarchical clustering, DBSCAN, etc.) will divide the data comparison results into different clusters or groups based on the similarity calculation results. The data points in each cluster have a high similarity, while the data points between different clusters have a low similarity.

[0104] In the embodiment of the present application, a similarity calculation result is obtained, that is, the cluster information to which each data point or object belongs, and the similarity value or score within the cluster.

[0105] In the embodiment of the present application, after obtaining the similarity calculation result, it is necessary to perform further classification operation on the data comparison result according to a preset similarity threshold.

[0106] In the embodiment of the present application, the preset similarity threshold is a pre-set value used to determine whether the similarity between data points is high enough so that they can be considered as part of the same category or group. According to this threshold, the data comparison results are divided into different categories or groups. The similarity value between data points belonging to the same category must be higher than or equal to the preset threshold.

[0107] In the embodiment of the present application, the result of the classification operation is the same type of comparison data, that is, the data points or objects classified into the same category. These same type of comparison data have similar characteristics or attributes and can be regarded as a whole for subsequent processing.

[0108] In the embodiment of the present application, after obtaining the same type of comparison data, it is necessary to perform a field difference evaluation operation on these data to identify the differences or differences between them.

[0109] In an embodiment of the present application, the minimum edit distance algorithm is mainly used to calculate the minimum number of transformations (such as insertion, deletion, replacement, etc.) between two strings to make them equal. The algorithm can be used to evaluate the degree of difference between data fields. Specifically, the minimum edit distance algorithm is applied to fields of similar comparison data of the same category. For each field, we calculate its minimum edit distance between different data points to quantify the degree of difference between them. By comparing the minimum edit distance values ​​of different fields, it is possible to identify which fields have differences and determine the specific content and degree of these differences.

[0110] In the embodiment of the present application, a difference data identification result is obtained, that is, a detailed report including all fields with differences and their degree of difference. This report can be used for subsequent data correction, cleaning or integration operations.

[0111] In the embodiment of the present application, similarities and differences between data are identified and processed through cluster analysis, classification operations and field difference evaluation.

[0112] In some optional implementations of the embodiments of the present application, after performing a difference data identification operation on the data comparison result to obtain a difference data identification result, the following steps are also included:

[0113] If there is difference data in the difference data identification result, the difference data is verified according to the preset verification rules to obtain the verification result;

[0114] If there is no abnormal data in the verification result, the difference data is confirmed as normal data;

[0115] If there are abnormal data in the verification results, the data comparison results and difference data will be classified, summarized and statistically analyzed to generate a data comparison report;

[0116] Output data comparison report to the requesting terminal.

[0117] In the embodiment of the present application, in the difference data identification stage, some differences between the data have been found. These differences may be caused by data input errors, data updates, inconsistent data formats, etc.

[0118] In an embodiment of the present application, after the difference data is identified, it is necessary to perform further verification operations on the data according to preset verification rules, wherein the verification rules are usually formulated based on the business logic of the data, data format requirements, data range restrictions, etc.

[0119] In an embodiment of the present application, the verification operation may include checking the integrity of the data (such as whether necessary fields are missing), accuracy (such as whether the value is within a reasonable range), consistency (such as whether the data between different data sources are consistent), etc.

[0120] In the embodiment of the present application, by applying the verification rules, the difference data can be checked item by item to determine whether they meet the expected data standards or requirements.

[0121] In the embodiment of the present application, a verification result is obtained, that is, an identification of whether each difference data has passed the verification. These verification results will be used for subsequent data processing decisions.

[0122] In an embodiment of the present application, if abnormal data is found during the verification process, that is, certain difference data fails to pass the inspection of the verification rules, then these abnormal data need to be further classified, summarized and statistically analyzed, where the classification and summary involves grouping and organizing the abnormal data according to specific dimensions (such as data type, data source, error type, etc.) in order to more clearly understand the distribution and characteristics of the abnormal data; statistical analysis includes calculating indicators such as the number, proportion, and distribution of abnormal data, as well as analyzing the possible causes and impacts of the abnormal data.

[0123] In the embodiment of the present application, a detailed data comparison report can be generated based on the results of classification summary and statistical analysis. This report will contain information such as comparison results, difference data, abnormal data and its classification summary and statistical analysis, providing data management personnel with comprehensive data quality assessment and improvement suggestions.

[0124] In the embodiment of the present application, the quality and accuracy of the data are improved by identifying and processing the differences between the data.

[0125] In some optional implementations of the embodiments of the present application, if there is difference data in the difference data identification result, the step of performing a verification operation on the difference data according to a preset verification rule to obtain a verification result specifically includes the following steps:

[0126] Perform a logic check operation on the difference data according to the preset logic check rules to obtain the check result.

[0127] In the embodiment of the present application, the compared difference data is subjected to a logic check according to a preset check rule, wherein the logic check mainly checks the integrity, uniqueness, value range, etc. of the data to ensure the basic correctness of the data.

[0128] In some optional implementations of the embodiments of the present application, if there is difference data in the difference data identification result, the step of performing a verification operation on the difference data according to a preset verification rule to obtain a verification result specifically includes the following steps:

[0129] Perform business verification operations on the difference data according to the preset business verification rules to obtain the verification results.

[0130] In the embodiment of the present application, the compared difference data is subjected to business verification according to preset verification rules, wherein the business verification determines whether the data conforms to the business logic and finds potential business anomalies according to the business rules of a specific field.

[0131] The comparison results and difference data are classified, summarized and statistically analyzed to generate a multi-dimensional data comparison report. Through data visualization technology, the distribution of comparison results, the type and quantity of difference data, etc. are intuitively displayed in the form of charts and reports, so that users can quickly understand and grasp the data quality status.

[0132] In some optional implementations of the embodiments of the present application, if there is logical abnormal data in the verification result, the step of classifying and summarizing the data comparison results and the difference data and performing statistical analysis to generate a data comparison report specifically includes the following steps:

[0133] Perform data correction operations on abnormal data based on data comparison results, verification results and preset processing rules.

[0134] In the embodiment of the present application, the difference data is automatically or manually processed according to the comparison result and the verification result. For the difference that can be automatically corrected, the system corrects and synchronizes the data according to the preset processing rules.

[0135] In some optional implementations of the present application, for differences that require manual confirmation, the system records the difference data and notifies relevant personnel to conduct verification and processing, and tracks the processing progress and results. At the same time, the comparison and processing logs are recorded in the database for audit and problem tracing.

[0136] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0137] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0139] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0140] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data comparison device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0141] like Figure 3 As shown, the data comparison device 200 of the embodiment of the present application includes:

[0142] The request receiving module 210 is used to receive a data comparison request sent by a request terminal, wherein the data comparison request includes data source information to be compared;

[0143] A task construction module 220, configured to construct a data comparison task corresponding to the data source information according to a pre-configured comparison rule file, wherein the data comparison task includes a metadata description;

[0144] The data conversion module 230 is used to perform a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information;

[0145] The preprocessing module 240 is used to perform data preprocessing operations on the standard data source information to obtain preprocessed data source information;

[0146] The data comparison module 250 is used to perform a data comparison operation on the pre-processed data source information according to the metadata description to obtain a data comparison result;

[0147] The difference data identification module 260 is used to perform a difference data identification operation on the data comparison result to obtain a difference data identification result;

[0148] The result output module 270 is used to output the difference data identification result to the requesting terminal.

[0149] In an embodiment of the present application, a data comparison device 200 is provided, comprising: a request receiving module 210, for receiving a data comparison request sent by a request terminal, wherein the data comparison request includes data source information to be compared; a task construction module 220, for constructing a data comparison task corresponding to the data source information according to a pre-configured comparison rule file, wherein the data comparison task includes a metadata description; a data conversion module 230, for performing a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information, to obtain standard data source information; a preprocessing module 240, for performing a data preprocessing operation on the standard data source information, to obtain preprocessed data source information; a data comparison module 250, for performing a data comparison operation on the preprocessed data source information according to the metadata description, to obtain a data comparison result; a difference data identification module 260, for performing a difference data identification operation on the data comparison result, to obtain a difference data identification result; and a result output module 270, for outputting the difference data identification result to the request terminal. Compared with the prior art, the present application effectively helps users or system administrators identify and analyze differences between data, thereby ensuring the accuracy and consistency of data.

[0150] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0151] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0152] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0153] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the data comparison method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0154] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as computer-readable instructions for running the data comparison method.

[0155] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0156] The computer device provided by this application effectively helps users or system administrators identify and analyze differences between data, thereby ensuring the accuracy and consistency of the data.

[0157] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data comparison method as described above.

[0158] The computer-readable storage medium provided in this application effectively helps users or system administrators identify and analyze differences between data, thereby ensuring the accuracy and consistency of the data.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0160] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data comparison method, characterized in that: The steps include: Receiving a data comparison request sent by a requesting terminal, wherein the data comparison request includes data source information to be compared; Constructing a data comparison task corresponding to the data source information according to a pre-configured comparison rule file, wherein the data comparison task includes a metadata description; Performing a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information; Performing data preprocessing operation on the standard data source information to obtain preprocessed data source information; Performing a data comparison operation on the pre-processed data source information according to the metadata description to obtain a data comparison result; Performing a difference data identification operation on the data comparison result to obtain a difference data identification result; The difference data identification result is output to the requesting terminal.

2. The data comparison method according to claim 1, characterized in that: The step of performing a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information specifically includes the following steps: Constructing a data conversion mapping table according to the data characteristics of the data source information; According to the type of the data source information, a corresponding parser is used to perform structural conversion to obtain converted data source information; The converted data source information is processed into a unified field type according to the data conversion mapping table to obtain the standard data source information.

3. The data comparison method according to claim 1, characterized in that: The step of performing a difference data identification operation on the data comparison result to obtain a difference data identification result specifically includes the following steps: Performing a similarity calculation operation on the data comparison result according to a cluster analysis algorithm to obtain a similarity calculation result; Classify the data comparison results according to the similarity calculation results and a preset similarity threshold to obtain similar comparison data; A field difference evaluation operation is performed on the same type of comparison data of the same category according to the minimum edit distance algorithm to obtain the difference data identification result.

4. The data comparison method according to claim 1, characterized in that: After the step of performing a difference data identification operation on the data comparison result to obtain a difference data identification result, the following steps are also included: If there is difference data in the difference data identification result, a verification operation is performed on the difference data according to a preset verification rule to obtain a verification result; If the verification result does not contain abnormal data, confirming the difference data as normal data; If there are abnormal data in the verification results, the data comparison results and the difference data are classified, summarized and statistically analyzed to generate a data comparison report; Outputting the data comparison report to the requesting terminal.

5. The data comparison method according to claim 4, characterized in that: If the verification result contains logically abnormal data, the step of classifying and summarizing the data comparison results and the difference data and performing statistical analysis to generate a data comparison report specifically includes the following steps: A data correction operation is performed on the abnormal data according to the data comparison result, the verification result and a preset processing rule.

6. The data comparison method according to claim 4, characterized in that: If there is difference data in the difference data identification result, the step of performing a verification operation on the difference data according to a preset verification rule to obtain a verification result specifically includes the following steps: A logic check operation is performed on the difference data according to a preset logic check rule to obtain the check result.

7. The data comparison method according to claim 4, characterized in that: If there is difference data in the difference data identification result, the step of performing a verification operation on the difference data according to a preset verification rule to obtain a verification result specifically includes the following steps: A business verification operation is performed on the difference data according to a preset business verification rule to obtain the verification result.

8. A data comparison device, characterized in that: include: A request receiving module, used to receive a data comparison request sent by a request terminal, wherein the data comparison request includes data source information to be compared; A task construction module, used to construct a data comparison task corresponding to the data source information according to a pre-configured comparison rule file, wherein the data comparison task includes a metadata description; A data conversion module, used to perform a data conversion operation on the data source information according to a data conversion method corresponding to the type of the data source information to obtain standard data source information; A preprocessing module, used to perform data preprocessing operations on the standard data source information to obtain preprocessed data source information; A data comparison module, used to perform a data comparison operation on the pre-processed data source information according to the metadata description to obtain a data comparison result; A difference data identification module is used to perform a difference data identification operation on the data comparison result to obtain a difference data identification result; A result output module is used to output the difference data identification result to the request terminal.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the data comparison method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data comparison method according to any one of claims 1 to 7 are implemented.

Images