Processing method for cleaning, governance and analysis of commercial data
Through the cleaning, governance and analysis and processing methods of commercial data, data pollution and quality problems are solved, data accuracy and consistency are achieved, data efficiency and security are ensured, and reliable technical support is provided for commercial data.
Patent Information
- Application Number
- CN202411086150.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Commercial data is easily hijacked and polluted or has quality problems when stored, such as errors, inconsistencies, missing, redundant, mutually exclusive, etc., which affects subsequent data analysis and mining work.
Adopt commercial data cleaning, governance and analysis processing methods, including data copying and backup, type classification and labeling, filtering, deep cleaning, integration and verification, and ensure data accuracy through a combination of multiple data processing algorithms and strategies. , integrity and consistency, and provide real-time monitoring and optimization capabilities through visual models.
It improves the quality of data cleaning, ensures the efficiency, security, accuracy and consistency of data, and provides reliable technical support for commercial data.
Smart Images

Figure CN119066054A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a processing method for cleaning, managing and analyzing commercial data. Background Art
[0002] Business data refers to the collection of historical and real-time information of each important link in an industry value chain, including but not limited to internal data of commercial enterprises, distribution channel data, consumer market data, etc.
[0003] In the era of big data, this type of data has become an important asset in all walks of life. Precisely because of the importance of data, the original data is easily hijacked and contaminated during storage, or the original data itself has various quality problems, such as errors, inconsistencies, omissions, redundancies, mutual exclusions, etc. These problems will have a serious impact on subsequent data analysis and mining work.
[0004] Therefore, data cleaning, governance and analysis become an indispensable and important step before the data processing process. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a processing method for cleaning, managing and analyzing business data, which can realize the cleaning of business data.
[0006] The technical solution of the present invention to solve the above technical problems is as follows: A processing method for cleaning, managing and analyzing business data, comprising the following steps:
[0007] S1: original business data input;
[0008] Input original business data as source data for data processing flow;
[0009] S2: Data replication and backup;
[0010] Fully copy the input original data and store the copy in the backup system to ensure data security and traceability;
[0011] S3: Data type classification and labeling
[0012] Use data processing models to classify raw data, differentiate them according to data types, and label each type of data;
[0013] S4: Data screening and filtering;
[0014] Based on preset business rules or data quality standards, the classified data is accurately screened to eliminate invalid, duplicate or abnormal data;
[0015] S5: Data cleaning and correction;
[0016] The screened data are cleaned in depth to ensure the accuracy, completeness and consistency of the data;
[0017] S6: Data integration output;
[0018] Integrate the cleaned data according to industry standards to generate new data set output;
[0019] S7: Data verification and optimization;
[0020] Perform data verification on newly generated data, evaluate data quality through comparison and verification, and iteratively optimize processing steps, rules or algorithms based on the verification results to improve data processing effectiveness.
[0021] Preferably, the data type classification in step S3 includes text data, numerical data, date and time data, image data, video data and audio data;
[0022] The marking method includes automatic marking and manual marking. Automatic marking identifies data features based on preset rules of the data processing model, and manual marking is used to supplement and correct errors in automatic marking or to supplement descriptive information of complex data.
[0023] Preferably, the step of classifying the input raw data by data type by the data processing model in step S3 specifically includes:
[0024] S31: Identify content in business data;
[0025] S32: judging whether the identified content has a standardized expression, and performing standardized conversion on the data that has a standardized expression but is not a standardized expression;
[0026] S33: Adjust classification rules and labeling strategies based on historical data processing experience to adapt to commercial data from different sources, different structures or different quality requirements.
[0027] Preferably, the preset business rules and data quality standards in step S4 allow for flexible configuration, and support dynamic adjustment of screening conditions according to specific business needs.
[0028] Preferably, the data cleaning in step S5 adopts a method combining multiple algorithms and strategies to cope with different types of data cleaning requirements.
[0029] Preferably, the integration method in step S6 includes merging the cleaned data according to industry standards to form a data set and / or performing customized integration according to user-specific needs.
[0030] Preferably, the data verification in step S7 includes data integrity verification, data consistency verification, and business logic verification. According to the verification results, the cleaning process is optimized and adjusted, including adjusting the screening rules, optimizing the cleaning algorithm, and correcting the data source, so as to achieve continuous improvement in data cleaning quality.
[0031] Preferably, a visualization model is also included to provide a visualization interface and real-time monitoring function for the data processing process, so that users can intuitively understand the data processing progress, quality status and potential problems, and support instant alarm and processing of abnormal situations.
[0032] The beneficial effects of the present invention are as follows: the present invention combines a variety of data processing algorithms to perform precise classification and labeling, flexible screening and filtering, and deep cleaning by data type. After correcting erroneous data, the data is integrated and output in a standardized manner, and the data is verified to generate feedback information. At the same time, the data processing process is continuously optimized based on the feedback, thereby improving the quality of data cleaning, ensuring the efficiency, security, accuracy, completeness and consistency of data cleaning, and providing reliable technical support for commercial data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A flowchart of a processing method for cleaning, managing and analyzing business data of the present invention;
[0034] Figure 2 A specific flow chart of step 3 of a processing method for cleaning, managing and analyzing business data of the present invention;
[0035] Figure 3 A schematic diagram of a cleaning process of a processing method for cleaning, managing and analyzing business data of the present invention;
[0036] Figure 4 A schematic diagram of data classification steps of a processing method for cleaning, managing and analyzing commercial data of the present invention. DETAILED DESCRIPTION
[0037] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.
[0038] like Figure 1 , Figure 2 , Figure 3 and Figure 4 As shown, a method for cleaning, managing and analyzing business data includes the following steps:
[0039] S1: original business data input;
[0040] Input original business data as source data for data processing flow;
[0041] S2: Data replication and backup;
[0042] Fully copy the input original data and store the copy in the backup system to ensure data security and traceability;
[0043] S3: Data type classification and labeling
[0044] Use data processing models to classify raw data, differentiate them according to data types, and label each type of data;
[0045] S4: Data screening and filtering;
[0046] Based on preset business rules or data quality standards, the classified data is accurately screened to eliminate invalid, duplicate or abnormal data;
[0047] S5: Data cleaning and correction;
[0048] The screened data are cleaned in depth to ensure the accuracy, completeness and consistency of the data;
[0049] S6: Data integration output;
[0050] Integrate the cleaned data according to industry standards to generate new data set output;
[0051] S7: Data verification and optimization;
[0052] Perform data verification on newly generated data, evaluate data quality through comparison and verification, and iteratively optimize processing steps, rules or algorithms based on the verification results to improve data processing effectiveness.
[0053] Specifically, the original data is stored in various channels, such as databases, Excel files, CSV files, etc., and is read using Python's Pandas library. In order to ensure the security and traceability of the data, Python's file operation function is used to store a copy of the data in the backup system. The Pandas library is combined with custom data processing logic to classify and mark the data. Based on preset business rules or data quality standards, Pandas' conditional screening function is used to accurately screen the data. For different types of data, functions and algorithms in libraries such as Pandas, NumPy, and SciPy are used for in-depth cleaning. The cleaned data is integrated according to industry specifications and standards to generate new data set outputs. Data quality is evaluated through comparison and verification, and processing steps are optimized based on the verification results. In the data cleaning process, in order to provide intuitive data processing process monitoring, Python's Matplotlib, Seaborn library, or R's ggplot2 library can be used for visualization, and the visualization interface and real-time monitoring functions provided by third-party tools such as Trifacto and DataRobot can be combined.
[0054] In an embodiment, the data type classification in step S3 includes text data, numerical data, date and time data, image data, video data and audio data;
[0055] The marking method includes automatic marking and manual marking. Automatic marking identifies data features based on preset rules of the data processing model, and manual marking is used to supplement and correct errors in automatic marking or to supplement descriptive information of complex data.
[0056] Specifically, in step S3, the data types are subdivided into text data, numerical data, date and time data, image data, video data, and audio data. Classification is the basis of data processing, because different types of data require different processing methods and strategies. Text data needs to be processed by word segmentation, stop word removal, and stem extraction; while numerical data may need to be normalized, standardized, or outlier detected through algorithms; date and time data needs to ensure the uniformity and correctness of the format in order to perform time series analysis or date-related calculations; image, video, and audio data may need to be processed by advanced processing such as format conversion, compression, and feature extraction.
[0057] Automatic labeling relies on pre-trained data processing models that automatically assign data type labels by identifying features in the data. Automatic labeling by data processing models greatly improves processing efficiency, but there may also be misjudgments or omissions. Therefore, manual labeling is used as a supplement to correct errors in automatic labeling and provide more accurate description information for complex or ambiguous data.
[0058] In an embodiment, the step of classifying the input raw data by data type by the data processing model in step S3 specifically includes:
[0059] S31: Identify the content in the business data;
[0060] S32: judging whether the identified content has a standardized expression, and performing standardized conversion on the data that has a standardized expression but is not a standardized expression;
[0061] S33: Adjust classification rules and labeling strategies based on historical data processing experience to adapt to commercial data from different sources, different structures or different quality requirements.
[0062] In the embodiment, the preset business rules and data quality standards in step S4 allow for flexible configuration, and support dynamic adjustment of screening conditions according to specific business needs.
[0063] Specifically, the preset business rules and data quality standards are the basis for data screening and filtering. The preset business rules and data quality standards can be flexibly configured and adjusted according to specific business needs; for example, in the financial industry, strict outlier detection rules may need to be set to ensure data accuracy; while in the retail industry, more attention may be paid to data integrity and consistency.
[0064] In an embodiment, the data cleaning in step S5 adopts a method combining multiple algorithms and strategies to cope with different types of data cleaning requirements.
[0065] Specifically, data cleaning is performed by using a combination of multiple algorithms and strategies to clean the multiple different types of data classified in step S3.
[0066] For text data, natural language processing algorithms are used to correct spelling errors and remove stop words and punctuation marks. For numerical data, Excel statistical methods, Pandas, and NumPy are used to process missing values or outliers, and perform operations such as mean replacement and median replacement. The comprehensive use of algorithms and strategies can significantly improve the efficiency and accuracy of data cleaning.
[0067] In an embodiment, the integration method in step S6 includes merging the cleaned data according to industry standards to form a data set and / or performing customized integration according to user-specific needs.
[0068] Specifically, the cleaned data needs to be integrated according to industry standards to form a data set. These standards usually include but are not limited to data format, encoding, naming conventions and other aspects.
[0069] At the same time, customized integration can also be performed to meet specific user needs. For example, in a data analysis project, you may need to merge multiple related data tables into a wide table for cross-table analysis; in a machine learning project, you may need to split the data into training sets, validation sets, and test sets for model training and evaluation. By using SQL to provide a flexible integration method, data integration and output can be easily achieved.
[0070] In an embodiment, the data verification in step S7 includes data integrity verification, data consistency verification, and business logic verification. According to the verification results, the cleaning process is optimized and adjusted, including adjusting the screening rules, optimizing the cleaning algorithm, and correcting the data source, so as to achieve continuous improvement in data cleaning quality.
[0071] Specifically, data verification includes data integrity verification, data consistency verification and business logic verification, which aims to ensure the accuracy and reliability of the data. After evaluating the data quality through comparison and verification, the reference data for comparison and verification can be a backup or standard value based on the original data. The cleaning process is optimized and adjusted according to the verification results, including adjusting the screening rules to eliminate more invalid data, optimizing the cleaning algorithm to improve cleaning efficiency and quality, and correcting the data source to eliminate data errors.
[0072] In the embodiment, a visualization model is also included to provide a visualization interface and real-time monitoring function for the data processing process, so that users can intuitively understand the data processing progress, quality status and potential problems, and support instant alarm and processing of abnormal situations.
[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for cleaning, managing and analyzing business data, characterized in that: The following steps are involved: S1: original business data input; Input original business data as source data for data processing flow; S2: Data replication and backup; Fully copy the input original data and store the copy in the backup system to ensure data security and traceability; S3: Data type classification and labeling Use data processing models to classify raw data, differentiate them according to data types, and label each type of data; S4: Data screening and filtering; Based on preset business rules or data quality standards, the classified data is accurately screened to eliminate invalid, duplicate or abnormal data; S5: Data cleaning and correction; The screened data are cleaned in depth to ensure the accuracy, completeness and consistency of the data; S6: Data integration output; Integrate the cleaned data according to industry standards to generate new data set output; S7: Data verification and optimization; Perform data verification on newly generated data, evaluate data quality through comparison and verification, and iteratively optimize processing steps, rules or algorithms based on the verification results to improve data processing effectiveness.
2. A method for cleaning, managing and analyzing business data according to claim 1, characterized in that: The data type classification in step S3 includes text data, numerical data, date and time data, image data, video data and audio data; The marking method includes automatic marking and manual marking. Automatic marking identifies data features based on preset rules of the data processing model, and manual marking is used to supplement and correct errors in automatic marking or to supplement descriptive information of complex data.
3. A method for cleaning, managing and analyzing business data according to claim 2, characterized in that: The step of classifying the input raw data by data type by the data processing model in step S3 specifically includes: S31: Identify the content in the business data; S32: judging whether the identified content has a standardized expression, and performing standardized conversion on the data that has a standardized expression but is not a standardized expression; S33: Adjust classification rules and labeling strategies based on historical data processing experience to adapt to commercial data from different sources, different structures or different quality requirements.
4. The method for cleaning, managing and analyzing business data according to claim 1, characterized in that: The preset business rules and data quality standards in step S4 allow for flexible configuration and support dynamic adjustment of screening conditions according to specific business needs.
5. The method for cleaning, managing and analyzing business data according to claim 1, characterized in that: The data cleaning in step S5 adopts a method combining multiple algorithms and strategies to cope with different types of data cleaning requirements.
6. The method for cleaning, managing and analyzing business data according to claim 1, characterized in that: The integration method in step S6 includes merging the cleaned data according to industry standards to form a data set and / or performing customized integration according to user-specific needs.
7. A method for cleaning, managing and analyzing business data according to claim 1, characterized in that: The data verification in step S7 includes data integrity verification, data consistency verification, and business logic verification. According to the verification results, the cleaning process is optimized and adjusted, including adjusting the screening rules, optimizing the cleaning algorithm, and correcting the data source, so as to achieve continuous improvement in data cleaning quality.
8. A method for cleaning, managing and analyzing business data according to any one of claims 1 to 7, characterized in that: It also includes a visualization model, which is used to provide a visualization interface and real-time monitoring function for the data processing process, so that users can intuitively understand the data processing progress, quality status and potential problems, and support instant alarm and processing of abnormal situations.
Citation Information
Cited By
Real-time data stream intelligent cleaning and integrating method
CN121301739A
Real-time data stream intelligent cleaning and integration method
CN121301739B
