Processing method for cleaning, governance and analysis of commercial data

Through the cleaning, governance and analysis and processing methods of commercial data, data pollution and quality problems are solved, data accuracy and consistency are achieved, data efficiency and security are ensured, and reliable technical support is provided for commercial data.

CN119066054AInactive Publication Date: 2024-12-03JINZHONG (WUXI) TECHNOLOGY CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411086150.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Commercial data is easily hijacked and polluted or has quality problems when stored, such as errors, inconsistencies, missing, redundant, mutually exclusive, etc., which affects subsequent data analysis and mining work.

Method used

Adopt commercial data cleaning, governance and analysis processing methods, including data copying and backup, type classification and labeling, filtering, deep cleaning, integration and verification, and ensure data accuracy through a combination of multiple data processing algorithms and strategies. , integrity and consistency, and provide real-time monitoring and optimization capabilities through visual models.

Benefits of technology

It improves the quality of data cleaning, ensures the efficiency, security, accuracy and consistency of data, and provides reliable technical support for commercial data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066054A_ABST
    Figure CN119066054A_ABST
Patent Text Reader

Abstract

The invention relates to a processing method for cleaning, managing and analyzing commercial data. The processing method comprises the following steps: S1, inputting original commercial data; inputting original business data as source data of a data processing flow; s2, copying and backing up data; and the input original data is comprehensively copied, and a copy is stored in a backup system, so that the security and traceability of the data are ensured. According to the processing method for cleaning, managing and analyzing the commercial data, multiple data processing algorithms are combined, accurate classification marking and flexible screening and filtering are carried out, deep cleaning is carried out according to data types, after error data are corrected, the data are standardized, integrated and output, the data are verified, and feedback information is generated; meanwhile, the data processing flow is continuously optimized according to feedback, the data cleaning quality is improved, the high efficiency, safety, accuracy, integrity and consistency of data cleaning are ensured, and reliable technical support is provided for business data cleaning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a processing method for cleaning, managing and analyzing commercial data. Background Art

[0002] Business data refers to the collection of historical and real-time information of each important link in an industry value chain, including but not limited to internal data of commercial enterprises, distribution channel data, consumer market data, etc.

[0003] In the era of big data, this type of data has become an important asset in all walks of life. Precisely because of the importance of data, the original data is easily hijacked and contaminated during storage, or the original data itself has various quality problems, such as errors, inconsistencies, omissions, redundancies, mutual exclusions, etc. These problems will have a serious impact on subsequent data analysis and mining work.

[0004] Therefore, data cleaning, governance and analysis become an indispensable and important step before the data processing process. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a processing method for cleaning, managing and analyzing business data, which can realize the cleaning of business data.

[0006] The technical solution of the present invention to solve the above technical problems is as follows: A processing method for cleaning, managing and analyzing business data, comprising the following steps:

[0007] S1: original business data input;

[0008] Input original business data as source data for data processing flow;

[0009] S2: Data replication and backup;

[0010] Fully copy the input original data and store the copy in the backup system to ensure data security and traceability;

[0011] S3: Data type classification and labeling

[0012] Use data processing models to classify raw data, differentiate them according to data types, and label each type of data;

[0013] S4: Data screening and filtering;

[0014] Based on preset business rules or data quality standards, the classified data is accurately screened to eliminate invalid, duplicate or abnormal data;

[0015] S5: Data cleaning and correction;

[0016] The screened data are cleaned in depth to ensure the accuracy, completeness and consistency of the data;

[0017] S6: Data integration output;

[0018] Integrate the cleaned data according to industry standards to generate new data set output;

[0019] S7: Data verification and optimization;

[0020] Perform data verification on newly generated data, evaluate data quality through comparison and verification, and iteratively optimize processing steps, rules or algorithms based on the verification results to improve data processing effectiveness.

[0021] Preferably, the data type classification in step S3 includes text data, numerical data, date and time data, image data, video data and audio data;

[0022] The marking method includes automatic marking and manual marking. Automatic marking identifies data features based on preset rules of the data processing model, and manual marking is used to supplement and correct errors in automatic marking or to supplement descriptive information of complex data.

[0023] Preferably, the step of classifying the input raw data by data type by the data processing model in step S3 specifically includes:

[0024] S31: Identify content in business data;

[0025] S32: judging whether the identified content has a standardized expression, and performing standardized conversion on the data that has a standardized expression but is not a standardized expression;

[0026] S33: Adjust classification rules and labeling strategies based on historical data processing experience to adapt to commercial data from different sources, different structures or different quality requirements.

[0027] Preferably, the preset business rules and data quality standards in step S4 allow for flexible configuration, and support dynamic adjustment of screening conditions according to specific business needs.

[0028] Preferably, the data cleaning in step S5 adopts a method combining multiple algorithms and strategies to cope with different types of data cleaning requirements.

[0029] Preferably, the integration method in step S6 includes merging the cleaned data according to industry standards to form a data set and / or performing customized integration according to user-specific needs.

[0030] Preferably, the data verification in step S7 includes data integrity verification, data consistency verification, and business logic verification. According to the verification results, the cleaning process is optimized and adjusted, including adjusting the screening rules, optimizing the cleaning algorithm, and correcting the data source, so as to achieve continuous improvement in data cleaning quality.

[0031] Preferably, a visualization model is also included to provide a visualization interface and real-time monitoring function for the data processing process, so that users can intuitively understand the data processing progress, quality status and potential problems, and support instant alarm and processing of abnormal situations.

[0032] The beneficial effects of the present invention are as follows: the present invention combines a variety of data processing algorithms to perform precise classification and labeling, flexible screening and filtering, and deep cleaning by data type. After correcting erroneous data, the data is integrated and output in a standardized manner, and the data is verified to generate feedback information. At the same time, the data processing process is continuously optimized based on the feedback, thereby improving the quality of data cleaning, ensuring the efficiency, security, accuracy, completeness and consistency of data cleaning, and providing reliable technical support for commercial data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a processing method for cleaning, managing and analyzing business data of the present invention;

[0034] Figure 2 A specific flow chart of step 3 of a processing method for cleaning, managing and analyzing business data of the present invention;

[0035] Figure 3 A schematic diagram of a cleaning process of a processing method for cleaning, managing and analyzing business data of the present invention;

[0036] Figure 4 A schematic diagram of data classification steps of a processing method for cleaning, managing and analyzing commercial data of the present invention. DETAILED DESCRIPTION

[0037] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0038] like Figure 1 , Figure 2 , Figure 3 and Figure 4 As shown, a method for cleaning, managing and analyzing business data includes the following steps:

[0039] S1: original business data input;

[0040] Input original business data as source data for data processing flow;

[0041] S2: Data replication and backup;

[0042] Fully copy the input original data and store the copy in the backup system to ensure data security and traceability;

[0043] S3: Data type classification and labeling

[0044] Use data processing models to classify raw data, differentiate them according to data types, and label each type of data;

[0045] S4: Data screening and filtering;

[0046] Based on preset business rules or data quality standards, the classified data is accurately screened to eliminate invalid, duplicate or abnormal data;

[0047] S5: Data cleaning and correction;

[0048] The screened data are cleaned in depth to ensure the accuracy, completeness and consistency of the data;

[0049] S6: Data integration output;

[0050] Integrate the cleaned data according to industry standards to generate new data set output;

[0051] S7: Data verification and optimization;

[0052] Perform data verification on newly generated data, evaluate data quality through comparison and verification, and iteratively optimize processing steps, rules or algorithms based on the verification results to improve data processing effectiveness.

[0053] Specifically, the original data is stored in various channels, such as databases, Excel files, CSV files, etc., and is read using Python's Pandas library. In order to ensure the security and traceability of the data, Python's file operation function is used to store a copy of the data in the backup system. The Pandas library is combined with custom data processing logic to classify and mark the data. Based on preset business rules or data quality standards, Pandas' conditional screening function is used to accurately screen the data. For different types of data, functions and algorithms in libraries such as Pandas, NumPy, and SciPy are used for in-depth cleaning. The cleaned data is integrated according to industry specifications and standards to generate new data set outputs. Data quality is evaluated through comparison and verification, and processing steps are optimized based on the verification results. In the data cleaning process, in order to provide intuitive data processing process monitoring, Python's Matplotlib, Seaborn library, or R's ggplot2 library can be used for visualization, and the visualization interface and real-time monitoring functions provided by third-party tools such as Trifacto and DataRobot can be combined.

[0054] In an embodiment, the data type classification in step S3 includes text data, numerical data, date and time data, image data, video data and audio data;

[0055] The marking method includes automatic marking and manual marking. Automatic marking identifies data features based on preset rules of the data processing model, and manual marking is used to supplement and correct errors in automatic marking or to supplement descriptive information of complex data.

[0056] Specifically, in step S3, the data types are subdivided into text data, numerical data, date and time data, image data, video data, and audio data. Classification is the basis of data processing, because different types of data require different processing methods and strategies. Text data needs to be processed by word segmentation, stop word removal, and stem extraction; while numerical data may need to be normalized, standardized, or outlier detected through algorithms; date and time data needs to ensure the uniformity and correctness of the format in order to perform time series analysis or date-related calculations; image, video, and audio data may need to be processed by advanced processing such as format conversion, compression, and feature extraction.

[0057] Automatic labeling relies on pre-trained data processing models that automatically assign data type labels by identifying features in the data. Automatic labeling by data processing models greatly improves processing efficiency, but there may also be misjudgments or omissions. Therefore, manual labeling is used as a supplement to correct errors in automatic labeling and provide more accurate description information for complex or ambiguous data.

[0058] In an embodiment, the step of classifying the input raw data by data type by the data processing model in step S3 specifically includes:

[0059] S31: Identify the content in the business data;

[0060] S32: judging whether the identified content has a standardized expression, and performing standardized conversion on the data that has a standardized expression but is not a standardized expression;

[0061] S33: Adjust classification rules and labeling strategies based on historical data processing experience to adapt to commercial data from different sources, different structures or different quality requirements.

[0062] In the embodiment, the preset business rules and data quality standards in step S4 allow for flexible configuration, and support dynamic adjustment of screening conditions according to specific business needs.

[0063] Specifically, the preset business rules and data quality standards are the basis for data screening and filtering. The preset business rules and data quality standards can be flexibly configured and adjusted according to specific business needs; for example, in the financial industry, strict outlier detection rules may need to be set to ensure data accuracy; while in the retail industry, more attention may be paid to data integrity and consistency.

[0064] In an embodiment, the data cleaning in step S5 adopts a method combining multiple algorithms and strategies to cope with different types of data cleaning requirements.

[0065] Specifically, data cleaning is performed by using a combination of multiple algorithms and strategies to clean the multiple different types of data classified in step S3.

[0066] For text data, natural language processing algorithms are used to correct spelling errors and remove stop words and punctuation marks. For numerical data, Excel statistical methods, Pandas, and NumPy are used to process missing values ​​or outliers, and perform operations such as mean replacement and median replacement. The comprehensive use of algorithms and strategies can significantly improve the efficiency and accuracy of data cleaning.

[0067] In an embodiment, the integration method in step S6 includes merging the cleaned data according to industry standards to form a data set and / or performing customized integration according to user-specific needs.

[0068] Specifically, the cleaned data needs to be integrated according to industry standards to form a data set. These standards usually include but are not limited to data format, encoding, naming conventions and other aspects.

[0069] At the same time, customized integration can also be performed to meet specific user needs. For example, in a data analysis project, you may need to merge multiple related data tables into a wide table for cross-table analysis; in a machine learning project, you may need to split the data into training sets, validation sets, and test sets for model training and evaluation. By using SQL to provide a flexible integration method, data integration and output can be easily achieved.

[0070] In an embodiment, the data verification in step S7 includes data integrity verification, data consistency verification, and business logic verification. According to the verification results, the cleaning process is optimized and adjusted, including adjusting the screening rules, optimizing the cleaning algorithm, and correcting the data source, so as to achieve continuous improvement in data cleaning quality.

[0071] Specifically, data verification includes data integrity verification, data consistency verification and business logic verification, which aims to ensure the accuracy and reliability of the data. After evaluating the data quality through comparison and verification, the reference data for comparison and verification can be a backup or standard value based on the original data. The cleaning process is optimized and adjusted according to the verification results, including adjusting the screening rules to eliminate more invalid data, optimizing the cleaning algorithm to improve cleaning efficiency and quality, and correcting the data source to eliminate data errors.

[0072] In the embodiment, a visualization model is also included to provide a visualization interface and real-time monitoring function for the data processing process, so that users can intuitively understand the data processing progress, quality status and potential problems, and support instant alarm and processing of abnormal situations.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for cleaning, managing and analyzing business data, characterized in that: The following steps are involved: S1: original business data input; Input original business data as source data for data processing flow; S2: Data replication and backup; Fully copy the input original data and store the copy in the backup system to ensure data security and traceability; S3: Data type classification and labeling Use data processing models to classify raw data, differentiate them according to data types, and label each type of data; S4: Data screening and filtering; Based on preset business rules or data quality standards, the classified data is accurately screened to eliminate invalid, duplicate or abnormal data; S5: Data cleaning and correction; The screened data are cleaned in depth to ensure the accuracy, completeness and consistency of the data; S6: Data integration output; Integrate the cleaned data according to industry standards to generate new data set output; S7: Data verification and optimization; Perform data verification on newly generated data, evaluate data quality through comparison and verification, and iteratively optimize processing steps, rules or algorithms based on the verification results to improve data processing effectiveness.

2. A method for cleaning, managing and analyzing business data according to claim 1, characterized in that: The data type classification in step S3 includes text data, numerical data, date and time data, image data, video data and audio data; The marking method includes automatic marking and manual marking. Automatic marking identifies data features based on preset rules of the data processing model, and manual marking is used to supplement and correct errors in automatic marking or to supplement descriptive information of complex data.

3. A method for cleaning, managing and analyzing business data according to claim 2, characterized in that: The step of classifying the input raw data by data type by the data processing model in step S3 specifically includes: S31: Identify the content in the business data; S32: judging whether the identified content has a standardized expression, and performing standardized conversion on the data that has a standardized expression but is not a standardized expression; S33: Adjust classification rules and labeling strategies based on historical data processing experience to adapt to commercial data from different sources, different structures or different quality requirements.

4. The method for cleaning, managing and analyzing business data according to claim 1, characterized in that: The preset business rules and data quality standards in step S4 allow for flexible configuration and support dynamic adjustment of screening conditions according to specific business needs.

5. The method for cleaning, managing and analyzing business data according to claim 1, characterized in that: The data cleaning in step S5 adopts a method combining multiple algorithms and strategies to cope with different types of data cleaning requirements.

6. The method for cleaning, managing and analyzing business data according to claim 1, characterized in that: The integration method in step S6 includes merging the cleaned data according to industry standards to form a data set and / or performing customized integration according to user-specific needs.

7. A method for cleaning, managing and analyzing business data according to claim 1, characterized in that: The data verification in step S7 includes data integrity verification, data consistency verification, and business logic verification. According to the verification results, the cleaning process is optimized and adjusted, including adjusting the screening rules, optimizing the cleaning algorithm, and correcting the data source, so as to achieve continuous improvement in data cleaning quality.

8. A method for cleaning, managing and analyzing business data according to any one of claims 1 to 7, characterized in that: It also includes a visualization model, which is used to provide a visualization interface and real-time monitoring function for the data processing process, so that users can intuitively understand the data processing progress, quality status and potential problems, and support instant alarm and processing of abnormal situations.

Citation Information

Cited By

  • Real-time data stream intelligent cleaning and integrating method

    CN121301739A

  • Real-time data stream intelligent cleaning and integration method

    CN121301739B