Data cleaning and multiplexing method and system

Abstract

The invention discloses a data cleaning and multiplexing method and system. The method comprises the following steps: S1, data acquisition: collecting and integrating data from various data sources; s2, data cleaning: detecting the data to identify abnormal data in the data, performing data correction processing on the abnormal data, performing data standardization processing on the corrected data, performing verification processing on the standardized data, integrating the verified data into a unified storage, and storing the unified storage in a database; the abnormal data comprises a missing value, an abnormal value and a duplicate value; and S3, data multiplexing: according to a new application scene, carrying out adaptation and conversion on the verified data, integrating the data subjected to adaptation and conversion into a unified storage, and publishing the data subjected to adaptation and conversion to different platforms or applications for other users or systems to use.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data cleaning and reuse method and system. Background Art

[0002] In the daily operation of an enterprise, it is often necessary to process various types of documents, such as invoices, orders, warehouse receipts, sales orders, warehouse delivery orders, etc., and there are often problems with inconsistent data in various documents. Inconsistent data sources may also be caused by data quality issues, such as missing data, outliers, incorrect input, etc. These problems may lead to inconsistencies in data quantity, quality, and accuracy.

[0003] In order to avoid such problems, enterprises need to add multi-level auditing devices to repeatedly audit the consistency of data. However, when there are many data sources, a lot of manpower and material resources need to be invested in verification to ensure the accuracy of source data. In a business process, multiple departments need to enter information on the same electronic flow multiple times, and each department needs to enter related data, and the data in each link needs to form a correlation relationship. Summary of the invention

[0004] In view of the problems and shortcomings of the prior art, the present invention provides a data cleaning and reuse method and system.

[0005] The present invention solves the above technical problems through the following technical solutions:

[0006] The present invention provides a data cleaning and reuse method, which is characterized in that it comprises the following steps:

[0007] S1. Data acquisition: Collect and integrate data from various data sources;

[0008] S2. Data cleaning: Detect the data to identify abnormal data in the data, correct the abnormal data, standardize the corrected data, verify the standardized data, and integrate the verified data into a unified storage. The abnormal data includes missing values, abnormal values ​​and duplicate values.

[0009] S3. Data reuse: Adapt and convert the verified data according to the new application scenario, integrate the adapted and converted data into a unified storage, and publish the adapted and converted data to different platforms or applications for use by other users or systems.

[0010] Preferably, the data cleaning and reuse method further comprises the following steps:

[0011] Data management: Record and manage the source, structure, cleaning and reuse rule information of data, continuously monitor the quality of data, and provide data quality reports and alerts to ensure the security and privacy of data during the cleaning and reuse process.

[0012] Preferably, in step S2, the data is detected to identify missing values ​​in the data, rows or columns with serious missing values ​​are deleted, and missing values ​​in rows or columns with less serious missing values ​​are filled by comparing historical data, wherein for missing values ​​in time series data, the missing values ​​are filled by using time series interpolation methods; SQL statements are used to detect in the database through keyword screening to identify abnormal values ​​in the data, and the abnormal values ​​are corrected to normal values; SQL statements are used to detect in the database to identify duplicate values, and only the first or last duplicate data is retained according to requirements; the standardized data is verified to verify the correlation between the data, check whether the data conforms to the expected data type, and check whether the data conforms to logical rules.

[0013] This embodiment also provides a data cleaning and multiplexing system, which is characterized in that it includes a data acquisition module, a data cleaning module and a data multiplexing module, the data cleaning module includes a data detection unit, a data correction unit, a data standardization unit and a data verification unit, and the data multiplexing module includes a data adaptation unit, a data integration unit and a data publishing unit;

[0014] The data acquisition module is used to collect and integrate data from various data sources;

[0015] The data detection unit is used to detect the data to identify abnormal data in the data, and the abnormal data includes missing values, abnormal values ​​and repeated values;

[0016] The data correction unit is used to perform data correction processing on abnormal data;

[0017] The data standardization unit is used to perform data standardization processing on the corrected data;

[0018] The data verification unit is used to verify the standardized data, and the verified data is integrated into a unified storage;

[0019] The data adaptation unit is used to adapt and convert the verified data according to the new application scenario;

[0020] The data integration unit is used to integrate the adapted and converted data into a unified storage;

[0021] The data publishing unit is used to publish the adapted and converted data to different platforms or applications for use by other users or systems.

[0022] Preferably, the data cleaning and reuse system further comprises a data management module, wherein the data management module comprises a metadata management unit, a data quality management unit and a data security management unit;

[0023] The metadata management unit is used to record and manage the source, structure, cleaning and reuse rule information of data;

[0024] The data quality management unit is used to continuously monitor the quality of data and provide data quality reports and alarms;

[0025] The data security management unit is used to ensure the security and privacy of data during the cleaning and reuse process.

[0026] Preferably, the data detection unit is used to detect the data to identify missing values ​​in the data, detect by keyword screening in the database using SQL statements to identify abnormal values ​​in the data, and detect in the database using SQL statements to identify duplicate values ​​in the data;

[0027] The data correction unit is used to delete rows or columns with serious missing values, and fill in missing values ​​in rows or columns with less serious missing values ​​by comparing historical data. For missing values ​​in time series data, the missing values ​​are filled by using a time series interpolation method, abnormal values ​​are corrected to normal values, and for duplicate values, only the first or last duplicate data is retained according to requirements;

[0028] The data verification unit is used to perform verification processing on the standardized data to verify the correlation between the data, check whether the data conforms to the expected data type, and check whether the data conforms to the logical rules.

[0029] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present invention.

[0030] The positive and progressive effects of the present invention are:

[0031] 1) Resolved data inconsistencies and conflicts:

[0032] Data source consistency ensures that data from different sources are unified in format, definition, and classification standards. This avoids inconsistencies and conflicts between data, making data analysis more accurate and reliable.

[0033] 2) Improved data quality and accuracy:

[0034] A consistent data source means that data follows the same standards and specifications during collection, processing, and analysis, thereby reducing the possibility of data errors, missing data, and outliers. This improves the overall quality and accuracy of the data and provides a more solid foundation for decision making.

[0035] 3) Simplified data integration and processing:

[0036] When data sources are consistent, data integration and processing become simpler and more efficient. There is no need to spend a lot of time and resources to clean, transform and integrate data from different sources, thus reducing the complexity and cost of data processing.

[0037] 4) Enhanced data credibility and reliability:

[0038] Consistent data sources provide data with greater credibility and reliability, which makes the application of data in business decision-making, strategic planning and other key areas more trusted and valued.

[0039] 5) Promoted cross-departmental and cross-platform collaboration:

[0040] When different departments and platforms use the same data source, they can more easily share, exchange and use data. This promotes cross-departmental and cross-platform collaboration and communication, and improves the efficiency and responsiveness of the entire organization.

[0041] In summary, consistency of data sources is of great significance in ensuring data quality, improving analysis accuracy, simplifying data processing procedures, enhancing data credibility, and promoting organizational collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The figure is a flow chart of a data cleaning and reuse method according to a preferred embodiment of the present invention.

[0043] Figure 2 The structure block diagram of the data cleaning and reuse system of the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, this embodiment provides a data cleaning and reuse method for ensuring the quality, accuracy and reusability of data, which includes the following steps:

[0046] Step 101, data acquisition: collect and integrate data from various data sources.

[0047] Step 102, data cleaning: Use statistical methods, rule engines or other technologies to detect data to identify abnormal data in the data, perform data correction processing on the abnormal data, and perform data standardization on the corrected data to ensure that the data follows a unified format and standard, such as data coding, date format, etc., verify the standardized data, and integrate the verified data into a unified storage. Abnormal data includes missing values, outliers and duplicate values.

[0048] In step 102, the data is tested to identify missing values ​​in the data, rows or columns with serious missing values ​​are deleted, and missing values ​​in rows or columns with less serious missing values ​​are filled by comparing historical data, wherein the missing values ​​in time series data are filled by time series interpolation method; SQL statements are used to detect in the database through keyword screening to identify abnormal values ​​in the data, and the abnormal values ​​are corrected to normal values; SQL statements are used to detect in the database to identify duplicate values ​​in the data, and only the first or last duplicate data is retained according to requirements; the standardized data is verified to verify the correlation between the data, check whether the data conforms to the expected data type, and check whether the data conforms to logical rules, such as date format, value range, etc. Integrate data from different sources, resolve conflicts and redundancy issues, and ensure field matching between different data sources.

[0049] Step 103, data reuse: Adapt and convert the verified data according to the new application scenario, integrate the adapted and converted data into a unified storage, and publish the adapted and converted data to different platforms or applications for use by other users or systems.

[0050] In addition, the data cleaning and reuse method also includes the following steps:

[0051] Data management: Record and manage the source, structure, cleaning and reuse rule information of data, continuously monitor the quality of data, and provide data quality reports and alerts to ensure the security and privacy of data during the cleaning and reuse process.

[0052] This method provides a complete process from raw data collection, cleaning, to reuse, aiming to ensure that data maintains high quality and reusability throughout its life cycle. At the same time, through the data management layer, effective management and monitoring of data can be achieved to meet the needs of different application scenarios.

[0053] like Figure 2As shown, this embodiment also provides a data cleaning and reuse system, which includes a data acquisition module 10, a data cleaning module 20, a data reuse module 30 and a data management module 40, the data cleaning module 20 includes a data detection unit 21, a data correction unit 22, a data standardization unit 23 and a data verification unit 24, the data reuse module 30 includes a data adaptation unit 31, a data integration unit 32 and a data publishing unit 33, and the data management module 40 includes a metadata management unit 41, a data quality management unit 42 and a data security management unit 43.

[0054] The data acquisition module 10 is used to collect and integrate data from various data sources.

[0055] The data detection unit 21 is used to detect the data to identify abnormal data in the data, and the abnormal data includes missing values, abnormal values ​​and duplicate values. Specifically, the data detection unit 21 is used to detect the data to identify missing values ​​in the data, detect by keyword screening in the database using SQL statements to identify abnormal values ​​in the data, and detect in the database using SQL statements to identify duplicate values ​​in the data.

[0056] The data correction unit 22 is used to perform data correction processing on abnormal data. Specifically, the data correction unit 22 is used to delete rows or columns with serious missing values, and fill the missing values ​​in rows or columns with less serious missing values ​​by comparing historical data. For missing values ​​in time series data, the time series interpolation method is used to fill the missing values, abnormal values ​​are corrected to normal values, and only the first or last piece of repeated data is retained according to needs.

[0057] The data standardization unit 23 is used to perform data standardization processing on the corrected data.

[0058] The data verification unit 24 is used to verify the standardized data, and the verified data is integrated into a unified storage. Specifically, the data verification unit 24 is used to verify the standardized data to verify the correlation between the data, check whether the data conforms to the expected data type, and check whether the data conforms to the logical rules.

[0059] The data adaptation unit 31 is used to adapt and convert the verified data according to the new application scenario.

[0060] The data integration unit 32 is used to integrate the adapted and converted data into a unified storage.

[0061] The data publishing unit 33 is used to publish the adapted and converted data to different platforms or applications for use by other users or systems.

[0062] The metadata management unit 41 is used to record and manage the source, structure, cleaning and reuse rule information of the data.

[0063] The data quality management unit 42 is used to continuously monitor the quality of the data and provide data quality reports and alarms.

[0064] The data security management unit 43 is used to ensure the security and privacy of data during the cleaning and reuse process.

[0065] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that these are only examples, and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A data cleaning and reuse method, characterized in that: It includes the following steps: S1. Data acquisition: Collect and integrate data from various data sources; S2. Data cleaning: Detect the data to identify abnormal data in the data, correct the abnormal data, standardize the corrected data, verify the standardized data, and integrate the verified data into a unified storage. The abnormal data includes missing values, abnormal values ​​and duplicate values. S3. Data reuse: Adapt and convert the verified data according to the new application scenario, integrate the adapted and converted data into a unified storage, and publish the adapted and converted data to different platforms or applications for use by other users or systems.

2. The data cleaning and reuse method according to claim 1, characterized in that: The data cleaning and reuse method further comprises the following steps: Data management: Record and manage the source, structure, cleaning and reuse rule information of data, continuously monitor the quality of data, and provide data quality reports and alerts to ensure the security and privacy of data during the cleaning and reuse process.

3. The data cleaning and reuse method according to claim 1, characterized in that: In step S2, the data is tested to identify missing values ​​in the data, rows or columns with serious missing values ​​are deleted, and missing values ​​in rows or columns with less serious missing values ​​are filled by comparing with historical data. For missing values ​​in time series data, the time series interpolation method is used to fill the missing values; SQL statements are used to detect keywords in the database to identify abnormal values ​​in the data, and the abnormal values ​​are corrected to normal values; Use SQL statements to detect in the database to identify duplicate values ​​in the data, and retain only the first or last piece of duplicate data as required; verify the standardized data to verify the correlation between the data, check whether the data conforms to the expected data type, and check whether the data conforms to logical rules.

4. A data cleaning and reuse system, characterized in that: It includes a data acquisition module, a data cleaning module and a data multiplexing module, the data cleaning module includes a data detection unit, a data correction unit, a data standardization unit and a data verification unit, and the data multiplexing module includes a data adaptation unit, a data integration unit and a data publishing unit; The data acquisition module is used to collect and integrate data from various data sources; The data detection unit is used to detect the data to identify abnormal data in the data, and the abnormal data includes missing values, abnormal values ​​and repeated values; The data correction unit is used to perform data correction processing on abnormal data; The data standardization unit is used to perform data standardization processing on the corrected data; The data verification unit is used to verify the standardized data, and the verified data is integrated into a unified storage; The data adaptation unit is used to adapt and convert the verified data according to the new application scenario; The data integration unit is used to integrate the adapted and converted data into a unified storage; The data publishing unit is used to publish the adapted and converted data to different platforms or applications for use by other users or systems.

5. The data cleaning and reuse system according to claim 4, characterized in that: The data cleaning and reuse system also includes a data management module, which includes a metadata management unit, a data quality management unit and a data security management unit; The metadata management unit is used to record and manage the source, structure, cleaning and reuse rule information of data; The data quality management unit is used to continuously monitor the quality of data and provide data quality reports and alarms; The data security management unit is used to ensure the security and privacy of data during the cleaning and reuse process.

6. The data cleaning and reuse system according to claim 4, characterized in that: The data detection unit is used to detect the data to identify missing values ​​in the data, detect by keyword screening in the database using SQL statements to identify abnormal values ​​in the data, and detect in the database using SQL statements to identify duplicate values ​​in the data; The data correction unit is used to delete rows or columns with serious missing values, and fill in missing values ​​in rows or columns with less serious missing values ​​by comparing historical data. For missing values ​​in time series data, the missing values ​​are filled by using a time series interpolation method, abnormal values ​​are corrected to normal values, and for duplicate values, only the first or last duplicate data is retained according to requirements; The data verification unit is used to perform verification processing on the standardized data to verify the correlation between the data, check whether the data conforms to the expected data type, and check whether the data conforms to the logical rules.

Images