Multi-data-source data standardization processing method and device supporting differential comparison

Abstract

The invention relates to the technical field of data processing, and particularly provides a multi-data-source data standardization processing method and device supporting differential comparison, firstly, a data association configuration module provides a data source mapping relation for a data filling module, and the data filling module calls a data query and selection module to realize automatic data filling; the data filling module transmits filled data to the data difference comparison module, the data difference comparison module carries out difference analysis and standardization processing on the filled data, the data objection feedback module feeds back the updated data to the data filling module, and the data filling module carries out filling again according to the updated data. Compared with the prior art, the method has the advantages that the difference between the system data and the standardized data can be efficiently and accurately identified, and the difference data is subjected to standardized processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and specifically provides a method and device for standardizing data from multiple data sources that supports differentiated comparison. Background Art

[0002] In the field of data processing, system data often originates from diverse data sources, potentially with significant discrepancies in format, structure, and content. To ensure data consistency and usability, it is often necessary to compare system data with standardized data and standardize any discrepancies. However, existing technologies often rely on manual, item-by-item verification, which is inefficient and prone to errors. Furthermore, they lack support for cross-data source querying, making data reporting and standardization cumbersome and time-consuming.

[0003] Therefore, a new technical solution is urgently needed to solve the problem of data comparison and processing with standardized data. Summary of the Invention

[0004] The present invention aims to address the above-mentioned deficiencies in the prior art and provides a highly practical method for standardizing data from multiple data sources that supports differentiated comparison.

[0005] A further technical task of the present invention is to provide a rationally designed, safe and applicable multi-data source data standardization processing device that supports differentiated comparison.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A data standardization processing method for multiple data sources that supports differential comparison. First, a data association configuration module provides a data source mapping relationship for a data reporting module, and the data reporting module calls a data query and selection module to realize automatic data filling.

[0008] The data reporting module passes the reported data to the data difference comparison module, which performs difference analysis and standardization on the reported data. The data objection feedback module feeds back the updated data to the data reporting module, which then re-reports based on the updated data.

[0009] Furthermore, the data association configuration module and the data reporting module include the following steps:

[0010] A1. The ledger creator configures the data source information and the basic information of the ledger reporting fields according to business requirements;

[0011] A2. Create a mapping relationship between the ledger's personnel configuration fields and the data center's source data. Bind the data center data by selecting the ledger fields and the data center's source data fields. The form is associated with the data source form. A form field can only be mapped to one data source field, but a data source field can be referenced by multiple form fields.

[0012] A3. Configure data access permissions;

[0013] A4. The user enters the data reporting page, enters the primary key field information, and clicks the search button next to the associated primary key.

[0014] A5. The system queries the data in the data source form based on the configured data association information. Since data is not unique, multiple different data may exist for the same field. The system then determines whether there are multiple entries.

[0015] A6. When there is only one piece of data, it is directly filled into the form when it is returned to the user. When there are multiple pieces of data, the system generates a form for the user to select the data they need.

[0016] A7. Select the data and save it. The data will then enter the review process.

[0017] Furthermore, in the data difference comparison module, according to the form information set by the user, the following steps are performed:

[0018] B1. Select the data to be compared from the system. You can select one or more data items, or choose to directly compare all data. Use data cleaning tools to remove invalid and duplicate data.

[0019] B2. Based on the primary key of the selected comparison data, the associated standard data in the data source is queried, a comparison record is generated, and then a difference comparison is performed asynchronously. A hash algorithm is used to compare the system data and the standardized data one by one, identifying differences in format, structure, and content. For text data, a text similarity algorithm is used for comparison to generate a difference report.

[0020] B3. Automatically standardize discrepant data based on the specifications of the standardized data set. For ID card information, extract birthday and gender information and automatically fill in the information. For data that cannot be automatically processed, provide a user interface for manual adjustment by the user.

[0021] B4. After the user modifies the data, it is reviewed and stored in the database. The difference report and standardization processing results are stored in the database, and a visual report is generated for the user to view.

[0022] Furthermore, in step B2, the data comparison method mentioned above first performs preliminary standardization on the data before comparison. The system generates a hash value for each field of each data item. The hash algorithm converts the data content into a unique hash value of fixed length.

[0023] The MD5 hash algorithm is used to generate a hash value for each field of each data, and the similarity of the text data is calculated using the cosine similarity algorithm. The similarity threshold is set to 0.8, and data below this threshold is marked as different data.

[0024] Furthermore, in step B3, the difference data found during the comparison process are annotated, and the annotation content includes the difference field, difference type and difference value. The system generates a difference report based on the annotated difference data. The difference report is displayed in a table format, listing detailed information for each difference data, which is convenient for users to view and analyze.

[0025] Furthermore, a combination of automation and manual intervention is adopted. For regularized data, automated processing is used; for complex or non-regularized data, a manual intervention interface is provided.

[0026] A dynamic standardization processing method of dynamic mapping and completion is adopted to automatically standardize the difference data according to the specifications of the standardized data set.

[0027] Furthermore, in the data objection feedback module, the following steps are included:

[0028] C1. Select the data to which you are disputing, fill in the basic information of the data dispute, enter the reason for submitting the dispute, and support uploading attachments;

[0029] C2. The user submits objection information. The administrator reviews the objection submitted by the user and conducts a data review. After the review is passed, the administrator modifies the data in the data source to complete the data repair.

[0030] C3. After the administrator completes the data modification, the system notifies the user who raised the objection of the result of the data review. The user then returns the correct data based on auxiliary reporting or difference comparison to complete the data standardization process.

[0031] A multi-data source data standardization processing device supporting differential comparison includes: at least one memory and at least one processor;

[0032] The at least one memory is configured to store a machine-readable program;

[0033] The at least one processor is configured to call the machine-readable program to execute a multi-data source data standardization processing method that supports differential comparison.

[0034] Compared with the prior art, the method and device for processing data standardization from multiple data sources supporting differential comparison of the present invention have the following outstanding beneficial effects:

[0035] (1) Improve data reporting efficiency and ensure data accuracy: By configuring data association information, the system can automatically query and fill in relevant data from multiple data sources, reducing the workload of users' manual input and significantly improving data reporting efficiency. At the same time, the system supports users to manually select and modify data, ensuring data accuracy and flexibility.

[0036] (2) Support historical data difference comparison: The system can compare historical data and generate difference reports to help users quickly identify inconsistencies in the data. Through hashing algorithms and text similarity algorithms (such as cosine similarity), the system can efficiently compare differences in format, structure, and content to ensure data standardization and consistency.

[0037] (3) Data objection handling mechanism and data standardization: The system provides a data objection handling function. Users can submit feedback on the data with which they have objections, and the administrator can modify the data in the standard table after review. This mechanism ensures the normalization and standardization of data, while enhancing the user's sense of participation in data management. The system can automatically standardize the difference data (such as unifying the date format, currency unit, etc.) according to the specifications of the standardized data set, reducing the need for manual intervention. For data that cannot be processed automatically, the system provides a user interface for manual adjustment to ensure data standardization.

[0038] (4) The present invention can be widely used in various data reporting and comparison scenarios, such as enterprise data management, government data statistics, financial data review, etc., and has broad application prospects and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 It is a flowchart of a method for standardizing data from multiple data sources that supports differential comparison;

[0041] Figure 2 It is a schematic diagram of the auxiliary reporting process in a multi-data source data standardization processing method that supports differentiated comparison;

[0042] Figure 3It is a schematic diagram of the difference comparison process in a multi-data source data standardization processing method that supports differential comparison;

[0043] Figure 4 It is a schematic diagram of the data objection process in a multi-data source data standardization processing method that supports differential comparison. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention will be further described in detail below in conjunction with specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0045] A best embodiment is given below:

[0046] like Figure 1 As shown, in this embodiment, a multi-data source data standardization method supporting differential comparison is described. The method comprises a data association configuration module, a data reporting module, a data query and selection module, a data difference comparison module, and a data objection feedback module. The data association configuration module provides data source mapping relationships for the data reporting module, which then calls the data query and selection module to implement automated data filling. The data reporting module passes the reported data to the data difference comparison module, which then performs difference analysis and standardization on the reported data. The data objection feedback module feeds back updated data to the data reporting module, which then re-reports the data based on the updated data.

[0047] The data association configuration module and data reporting module include the following steps:

[0048] A1. The person creating the ledger configures the data source information, including the physical path, logical path, library table name, data access permissions, etc. of the specified data source; and configures the basic information of the ledger reporting fields according to business requirements.

[0049] A2. Create a mapping relationship between the ledger's personnel configuration fields and the data center's source data. By selecting the ledger fields and the data center's source data fields, you can quickly bind the data center data. Forms are associated with data source forms. A form field can only be mapped to one data source field, but a data source field can be referenced by multiple form fields.

[0050] A3. Configure data access permissions to ensure data security.

[0051] A4. The user enters the data filling page, enters the primary key field information, and clicks the search button next to the associated primary key.

[0052] A5. The system queries the data on the data source form based on the configured data association information. Since data is not unique, multiple different data may exist for the same field. The system determines whether there are multiple data entries.

[0053] A6. When there is only one piece of data, it is directly filled into the form when it is returned to the user. When there are multiple pieces of data, the system generates a form for the user to select the data they need.

[0054] A7. Select the data and save it. The data will then enter the review process.

[0055] For the data difference comparison module, according to the form information set by the user, the following steps are included:

[0056] B1. Select the data to be compared from the system. You can select one or more data, or choose to directly compare all data and use data cleaning tools to remove invalid and duplicate data.

[0057] B2. Based on the primary key of the selected comparison data, the associated standard data in the data source is queried, a comparison record is generated, and then a difference comparison is performed asynchronously. A hash algorithm is used to compare the system data and the standardized data line by line, identifying differences in format, structure, and content. For text data, a text similarity algorithm (such as cosine similarity) is used for comparison and a difference report is generated.

[0058] Before comparison, the system performs preliminary standardization on the data, for example, standardizing the date format to "YYYY-MM-DD" to ensure data consistency. The system generates a hash value for each field in each piece of data. Hash algorithms (such as MD5 and SHA-256) convert data content into a unique hash value of fixed length. Even slight changes in the data content will result in a significantly different hash value.

[0059] The MD5 hash algorithm is used to generate a hash value for each field of each data, and the similarity of the text data is calculated using the cosine similarity algorithm. The similarity threshold is set to 0.8, and data below this threshold is marked as different data.

[0060] B3. Automatically standardize discrepant data according to the specifications of the standardized dataset. For example, standardize the date format to "YYYY-MM-DD" and the currency unit to "USD." For ID card information, extract birthday and gender information and automatically fill in the data. For data that cannot be automatically processed, provide a user interface for manual adjustment.

[0061] The system annotates discrepancies found during the comparison process, including the discrepancy field, discrepancy type, and discrepancy value (comparison between system data and standardized data). Based on the annotated discrepancy data, the system generates a discrepancy report. The discrepancy report is presented in a table format, listing detailed information about each discrepancy for easy review and analysis.

[0062] A combination of automation and manual intervention is employed. For data that can be regularized (such as date formats and currency units), automated processing is used; for complex or non-regularized data, a manual intervention interface is provided. This approach improves efficiency while ensuring flexibility in data processing.

[0063] A dynamic standardization approach with dynamic mapping and completion is adopted. For example, non-standard fields are mapped to standard fields, or missing data is completed based on the context.

[0064] B4. After the user modifies the data, it is reviewed and stored in the database. The difference report and standardization processing results are stored in the database, and a visual report is generated for the user to view.

[0065] The data objection feedback module includes the following steps:

[0066] C1. Select the data that is disputed and fill in the basic information of the data dispute, including the form name, the field information that is disputed, the result that should be corrected, etc., fill in the reason for submitting the objection, and support uploading attachment information.

[0067] C2. The user submits objection information. The administrator can view the objection submitted by the user and review the data. After the review is passed, the administrator can modify the data in the data source to complete the data repair.

[0068] C3. After the administrator completes the data modification, the system notifies the user who raised the objection of the data review result. The user can then return the correct data based on the auxiliary reporting or difference comparison, completing the data standardization process.

[0069] Based on the above method, a data standardization processing device for multiple data sources supporting differential comparison in this embodiment includes: at least one memory and at least one processor;

[0070] The at least one memory is configured to store a machine-readable program;

[0071] The at least one processor is configured to call the machine-readable program to execute a multi-data source data standardization processing method that supports differential comparison.

[0072] The above-mentioned specific implementation methods are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementation methods. Any technical solutions that conform to the above-mentioned specific implementation methods of the present invention and any appropriate changes or substitutions made thereto by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0073] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A data standardization processing method for multiple data sources supporting differential comparison, characterized in that: First, the data association configuration module provides the data source mapping relationship for the data reporting module, and the data reporting module calls the data query and selection module to realize automatic data filling; The data reporting module passes the reported data to the data difference comparison module, which performs difference analysis and standardization on the reported data. The data objection feedback module feeds back the updated data to the data reporting module, which then re-reports based on the updated data.

2. A method for standardizing data from multiple data sources supporting differential comparison according to claim 1, characterized in that: The data association configuration module and the data reporting module include the following steps: A1. The ledger creator configures the data source information and the basic information of the ledger reporting fields according to business requirements; A2. Create a mapping relationship between the ledger's personnel configuration fields and the data center's source data. Bind the data center data by selecting the ledger fields and the data center's source data fields. The form is associated with the data source form. A form field can only be mapped to one data source field, but a data source field can be referenced by multiple form fields. A3. Configure data access permissions; A4. The user enters the data reporting page, enters the primary key field information, and clicks the search button next to the associated primary key. A5. The system queries the data in the data source form based on the configured data association information. Since data is not unique, multiple different data may exist for the same field. The system then determines whether there are multiple entries. A6. When there is only one piece of data, it is directly filled into the form when it is returned to the user. When there are multiple pieces of data, the system generates a form for the user to select the data they need. A7. Select the data and save it. The data will then enter the review process.

3. The method for standardizing data from multiple data sources supporting differential comparison according to claim 2, characterized in that: In the data difference comparison module, according to the form information set by the user, there are the following steps: B1. Select the data to be compared from the system. You can select one or more data items, or choose to directly compare all data. Use data cleaning tools to remove invalid and duplicate data. B2. Based on the primary key of the selected comparison data, the associated standard data in the data source is queried, a comparison record is generated, and then a difference comparison is performed asynchronously. A hash algorithm is used to compare the system data and the standardized data one by one, identifying differences in format, structure, and content. For text data, a text similarity algorithm is used for comparison to generate a difference report. B3. Automatically standardize discrepant data based on the specifications of the standardized data set. For ID card information, extract birthday and gender information and automatically fill in the information. For data that cannot be automatically processed, provide a user interface for manual adjustment by the user. B4. After the user modifies the data, it is reviewed and stored in the database. The difference report and standardization processing results are stored in the database, and a visual report is generated for the user to view.

4. The method for standardizing data from multiple data sources supporting differential comparison according to claim 3, characterized in that: In step B2, the data comparison method mentioned above first performs preliminary standardization on the data before comparison. The system generates a hash value for each field of each data item. The hash algorithm converts the data content into a unique hash value of fixed length. The MD5 hash algorithm is used to generate a hash value for each field of each data, and the similarity of the text data is calculated using the cosine similarity algorithm. The similarity threshold is set to 0.8, and data below this threshold is marked as different data.

5. The method for standardizing data from multiple data sources supporting differential comparison according to claim 4, characterized in that: In step B3, the difference data found during the comparison process are annotated. The annotation content includes the difference field, difference type, and difference value. The system generates a difference report based on the annotated difference data. The difference report is displayed in a table format, listing the detailed information of each difference data, which is convenient for users to view and analyze.

6. The method for standardizing data from multiple data sources supporting differential comparison according to claim 5, characterized in that: A combination of automation and manual intervention is used. For regularized data, automated processing is used; for complex or non-regularized data, a manual intervention interface is provided. A dynamic standardization processing method of dynamic mapping and completion is adopted to automatically standardize the difference data according to the specifications of the standardized data set.

7. The method for standardizing data from multiple data sources supporting differential comparison according to claim 6, characterized in that: In the data objection feedback module, there are the following steps: C1. Select the data to which you are disputing, fill in the basic information of the data dispute, enter the reason for submitting the dispute, and support uploading attachments; C2. The user submits objection information. The administrator reviews the objection submitted by the user and conducts a data review. After the review is passed, the administrator modifies the data in the data source to complete the data repair. C3. After the administrator completes the data modification, the system notifies the user who raised the objection of the result of the data review. The user then returns the correct data based on auxiliary reporting or difference comparison to complete the data standardization process.

8. A data standardization processing device for multiple data sources supporting differential comparison, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 7.

Images