Intelligent data cleaning method in big data environment

Abstract

The invention provides an intelligent data cleaning method in a big data environment, and belongs to the technical field of big data processing, and the intelligent data cleaning method in the big data environment comprises the steps of data acquisition and preprocessing, intelligent detection and identification, intelligent cleaning strategy making, cleaning execution and verification, continuous optimization and monitoring and the like. Data are collected through a database connection tool, a file reading interface and a web crawler, and format conversion and standardization processing are carried out. Abnormal values and outliers are detected through a machine learning algorithm and statistical analysis, a cleaning strategy is made, and verification is executed. A feedback mechanism is established, and the data cleaning process is monitored in real time. The problems that a traditional data cleaning method depends on manpower, is low in efficiency and lacks flexibility and adaptability are solved, the data cleaning efficiency, accuracy and reliability are improved, the method adapts to big data diversity and dynamics, the data quality is improved, the decision risk is reduced, and the actual application requirement is met.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular relates to an intelligent data cleaning method in a big data environment. Background Art

[0002] In today's big data era, the scale of data is growing explosively. Various enterprises and institutions are constantly accumulating massive amounts of data in their daily operations. These data come from different channels and systems and have multiple formats and structures. However, there are often various noises, errors and inconsistencies in large amounts of data. For example, in the e-commerce field, the address entered by the user may contain typos, be incomplete or have irregular formats; in the financial field, transaction data may have incorrect values ​​due to system failures or human errors; in the medical field, patients' medical records may be missing or inconsistent. These problematic data will not only affect the accuracy and reliability of data analysis, but may also lead to wrong decisions, causing huge losses to enterprises and society.

[0003] Traditional data cleaning methods are inefficient when faced with massive amounts of data and are unable to meet the needs of a big data environment. Traditional methods usually rely on manual rules and operations, which are not only time-consuming and labor-intensive, but also prone to omissions and errors. In addition, traditional methods lack flexibility and adaptability and are unable to cope with the diversity and dynamism of big data.

[0004] Therefore, based on the above technical problems, it is necessary to design intelligent data cleaning methods in big data environments. Summary of the invention

[0005] The purpose of the present invention is to provide an intelligent data cleaning method in a big data environment, aiming to solve the problem that traditional methods in the prior art usually rely on artificial rules and manual operations, which are not only time-consuming and labor-intensive, but also prone to omissions and errors. In addition, traditional methods lack flexibility and adaptability and cannot cope with the diversity and dynamics of big data.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The intelligent data cleaning method in a big data environment is characterized by comprising the following steps:

[0008] Step 1: Data collection and preprocessing: Collect big data from various data sources, convert and standardize the data to ensure data integrity and consistency;

[0009] Step 2, intelligent detection and identification: use machine learning algorithms and statistical analysis methods to detect abnormal values ​​and outliers in the data. Parameters include the learning rate of the classifier is set to 0.1, the number of iterations is 1000, and the indicators include the proportion of abnormal values ​​not exceeding 5%, the number of outliers is less than 3% of the total data, etc.

[0010] Step 3: Intelligent cleaning strategy formulation: formulate corresponding cleaning strategies according to the problem type, including the processing of missing values, error values ​​and inconsistent data, while considering the context information and business rules of the data;

[0011] Step 4, cleaning execution and verification: perform data cleaning operations according to the cleaning strategy, and verify through machine learning model evaluation or comparison with high-quality data. The indicators include accuracy reaching more than 90%, recall reaching more than 85%, F1 value reaching more than 88%, similarity greater than 95%, difference less than 5%, etc.;

[0012] Step 5: Continuous optimization and monitoring: Establish a feedback mechanism to continuously optimize cleaning strategies and algorithms, while monitoring the data cleaning process in real time.

[0013] As a preferred solution of the present invention, in step 1, the data in the database are collected in batches using a database connection tool, the data in the file system are collected through a file reading interface, and the network data are collected using a network crawler technology.

[0014] As a preferred solution of the present invention, in step 1, format conversion includes converting data in different formats into a unified format, and standardization processing includes standardizing the field name, data type, encoding method, etc. of the data.

[0015] As a preferred solution of the present invention, in step 2, a supervised learning algorithm and an unsupervised learning algorithm are used to train a classifier to distinguish normal data from abnormal data.

[0016] As a preferred solution of the present invention, in step 2, by calculating the mean, standard deviation, median, quartiles and other statistics of the data, outliers are judged according to the set threshold value, and outliers are detected using visualization methods such as box plots and scatter plots or distance-based methods.

[0017] As a preferred solution of the present invention, in step 3, missing values ​​are filled by using mean filling, median filling or interpolation methods, erroneous values ​​are corrected by data validation rules or machine learning models, and inconsistent data are aligned and standardized.

[0018] As a preferred solution of the present invention, in step 3, context information and business rules of the data are considered to ensure that the cleaned data meets actual application requirements.

[0019] As a preferred solution of the present invention, in step 4, a data processing tool and a programming language are used to perform a cleaning operation, paying attention to the security and privacy of the data.

[0020] As a preferred solution of the present invention, in step 4, the cleaning strategy and parameters are readjusted by re-evaluating using the machine learning model or comparing with known high-quality data for verification if the requirements are not met.

[0021] As a preferred solution of the present invention, in step 5, a feedback mechanism is established through user feedback, data analysis reports, automatic monitoring, etc., and tools such as Grafana and Prometheus are used to monitor data, and alarm rules and notification methods are set.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. In this solution, the efficiency of data cleaning is improved: by using database connection tools, file reading interfaces and web crawler technology for data collection, and professional data processing tools and programming languages ​​for cleaning operations, large-scale data can be processed quickly, reducing the time and errors of manual operations, and greatly improving the efficiency of data cleaning. For example, using Python's Pandas library, data can be easily filtered, sorted, merged, and other operations can be performed to speed up data cleaning;

[0024] Enhance the accuracy and reliability of data cleaning: Use machine learning algorithms and statistical analysis methods to detect abnormal values ​​and outliers in the data, and formulate corresponding cleaning strategies according to the problem type, so as to more accurately identify and process problematic data. At the same time, by re-evaluating the machine learning model or verifying it by comparing it with high-quality data, it can ensure that the quality of the data is effectively improved. For example, by training classifiers to distinguish between normal data and abnormal data, it can effectively identify newly emerging fraudulent transactions.

[0025] Adapting to the diversity and dynamics of big data: Format conversion and standardization of data in different formats are performed so that they can use the same processing flow and algorithm, which improves the versatility and scalability of the method. At the same time, continuous optimization and monitoring mechanisms can continuously adjust the cleaning strategies and algorithms according to changes in data and the development of business needs to ensure that the method can adapt to the dynamics of big data. For example, after converting data in different formats into a unified format, only one set of code is needed for processing, which is convenient and fast.

[0026] 2. In this solution, data quality is improved and decision-making risks are reduced: cleaned data has higher accuracy, completeness and consistency, which can provide a more reliable basis for data analysis and decision-making, and reduce the risk of wrong decisions caused by data problems; for example, in the financial field, cleaned transaction data can more accurately reflect market conditions and provide strong support for investment decisions;

[0027] Meeting actual application needs: Consider the contextual information and business rules of the data for cleaning to ensure that the cleaned data meets the actual application needs; this can improve the availability and readability of the data and enhance users' trust and satisfaction with data products; for example, in social network data, considering the user's social context information for data cleaning can more reasonably fill in missing data or correct erroneous data, thereby improving the quality and practicality of the data. DETAILED DESCRIPTION

[0028] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Example 1

[0030] The present invention provides the following technical solutions:

[0031] The intelligent data cleaning method in a big data environment is characterized by comprising the following steps:

[0032] Step 1: Data collection and preprocessing: Collect big data from various data sources, convert and standardize the data to ensure data integrity and consistency;

[0033] Step 2, intelligent detection and identification: use machine learning algorithms and statistical analysis methods to detect abnormal values ​​and outliers in the data. Parameters include the learning rate of the classifier is set to 0.1, the number of iterations is 1000, and the indicators include the proportion of abnormal values ​​not exceeding 5%, the number of outliers is less than 3% of the total data, etc.

[0034] Step 3: Intelligent cleaning strategy formulation: formulate corresponding cleaning strategies according to the problem type, including the processing of missing values, error values ​​and inconsistent data, while considering the context information and business rules of the data;

[0035] Step 4, cleaning execution and verification: perform data cleaning operations according to the cleaning strategy, and verify through machine learning model evaluation or comparison with high-quality data. The indicators include accuracy reaching more than 90%, recall reaching more than 85%, F1 value reaching more than 88%, similarity greater than 95%, difference less than 5%, etc.;

[0036] Step 5: Continuous optimization and monitoring: Establish a feedback mechanism to continuously optimize cleaning strategies and algorithms, while monitoring the data cleaning process in real time.

[0037] In a specific embodiment of the present invention, step 1, data collection and preprocessing: collect big data from various data sources, including databases, file systems, network data, etc. During the collection process, the integrity and accuracy of the data must be ensured to avoid data loss or damage.

[0038] For data in the database, you can use the database connection tool for batch collection; for data in the file system, you can collect it through the file reading interface; for network data, you can use network crawler technology to collect it.

[0039] The collected data is initially formatted and standardized for subsequent cleaning operations.

[0040] Format conversion includes converting data in different formats into a unified format, such as converting CSV files into JSON format or database table format. Standardization processing includes standardizing the field names, data types, encoding methods, etc. of the data to ensure data consistency.

[0041] Step 2: Intelligent detection and identification: Use machine learning algorithms to extract features from data and identify data patterns that may be problematic.

[0042] For example, a classifier is trained to distinguish between normal data and abnormal data. Supervised learning algorithms such as support vector machines, decision trees, random forests, etc. can be used, or unsupervised learning algorithms such as clustering algorithms and principal component analysis can be used. When training a classifier, it is necessary to select appropriate features and labels, as well as appropriate algorithm parameters to improve the accuracy and generalization ability of the classifier. The parameters may include setting the learning rate of the classifier to 0.1, the number of iterations to 1000, etc.

[0043] Statistical analysis methods are used to detect abnormal values ​​and outliers in the data.

[0044] Calculate the mean, standard deviation, median, quartiles and other statistics of the data, and judge outliers according to the set threshold. For example, data outside the range of the mean plus or minus three times the standard deviation can be considered outliers. For outliers, you can use visualization methods such as box plots and scatter plots to detect them, or you can use distance-based methods such as K nearest neighbor algorithms and local outlier factor algorithms to detect them. Indicators may include the proportion of outliers not exceeding 5%, the number of outliers less than 3% of the total data, etc.

[0045] Step 3: Develop intelligent cleaning strategies: Develop corresponding cleaning strategies based on the type of problems detected.

[0046] For missing values, you can use mean filling, median filling, or interpolation methods to fill them. If the data has time series characteristics, you can use time series prediction models to fill them. For example, for missing values ​​in stock price data, you can use historical price data for interpolation filling, or use machine learning models to predict future prices for filling.

[0047] For incorrect values, you can correct them through data validation rules, or use machine learning models for prediction correction. Data validation rules can be formulated according to business needs and data characteristics. For example, for ID card numbers, you can verify that their length is 18 digits, the format complies with specific rules, and the checksum is correct. For machine learning models, you can use regression models, classification models, etc. to make prediction corrections. For example, for incorrect values ​​in sales data, you can use historical sales data and related factors for regression analysis to predict the correct sales.

[0048] For inconsistent data, data alignment and standardization can be performed. Data alignment includes matching and integrating data from different data sources to eliminate duplicate and contradictory data. Standardization includes standardizing the data unit, precision, encoding method, etc. to ensure data consistency. For example, temperature data from different regions can be uniformly converted to degrees Celsius with precision retained to one decimal place.

[0049] Step 4: Cleaning execution and verification: Clean the data according to the established cleaning strategy.

[0050] Cleaning operations can be implemented using data processing tools and programming languages, such as using Python's Pandas library and R's data.table package to clean data. During the cleaning process, attention should be paid to data security and privacy to avoid data leakage and abuse.

[0051] Verify the cleaned data to ensure that the data quality is effectively improved.

[0052] You can evaluate by using the machine learning model again, or by comparing it with known high-quality data. For example, you can use a classifier to evaluate the cleaned data, and calculate indicators such as accuracy of more than 90%, recall of more than 85%, and F1 value of more than 88%. You can also compare the cleaned data with the data from the authoritative data source, and calculate indicators such as similarity greater than 95% and difference less than 5%. If the verification results do not meet the requirements, you need to readjust the cleaning strategy and parameters, and perform another cleaning and verification.

[0053] Step 5. Continuous optimization and monitoring: Establish a feedback mechanism to continuously optimize the cleaning strategy and algorithm based on the analysis results of the cleaned data.

[0054] Feedback mechanisms can include user feedback, data analysis reports, automatic monitoring, etc. User feedback can be obtained through questionnaires, user evaluations, etc. to obtain user satisfaction and suggestions on the cleaned data; data analysis reports can be used to perform statistical analysis and visual display of the cleaned data to discover problems and trends in the data and provide a basis for optimizing the cleaning strategy; automatic monitoring can set monitoring indicators and thresholds to monitor the performance and quality of the data cleaning process in real time, and promptly discover and handle problems.

[0055] Specifically, in step 1, the data in the database is collected in batches using a database connection tool, the data in the file system is collected through a file reading interface, and the network data is collected using a network crawler technology.

[0056] In this embodiment: Database data batch collection:

[0057] Efficiency: The database connection tool can interact directly with the database and quickly read large amounts of data in batches, avoiding the tedious and inefficient manual extraction of data one by one, greatly improving the speed and efficiency of data collection.

[0058] Accuracy: It can ensure the integrity and accuracy of data because it is collected directly from the database source, reducing possible errors and losses during data transmission.

[0059] Scalability: It can adapt to different types of database systems, whether relational or non-relational, and can collect data through corresponding connection tools to meet the needs of different application scenarios.

[0060] Specifically, in step 1, format conversion includes converting data in different formats into a unified format, and standardization processing includes standardizing the field name, data type, encoding method, etc. of the data.

[0061] In this embodiment, different tools and methods are required for analyzing and processing data in different formats. After converting them into a unified format, the same processing flow and algorithm can be used, greatly improving the efficiency of data processing. For example, it was originally necessary to write processing codes for different formats such as CSV, JSON, and database tables, but only one set of codes is needed after the unified format.

[0062] Specifically, in step 2, a supervised learning algorithm and an unsupervised learning algorithm are used to train a classifier to distinguish normal data from abnormal data.

[0063] In this embodiment: the supervised learning algorithm is trained by labeled normal data and abnormal data, and can learn clear patterns and features, so that it can classify new data more accurately and distinguish normal data from abnormal data. For example, in financial transaction data, by marking some known fraudulent transactions and normal transaction data to train the classifier, newly emerging fraudulent transactions can be effectively identified.

[0064] Specifically, in step 2, by calculating the mean, standard deviation, median, quartiles and other statistics of the data, outliers are determined according to the set thresholds, and outliers are detected using visualization methods such as box plots and scatter plots or distance-based methods.

[0065] In this embodiment: objective quantification: by calculating statistics such as mean, standard deviation, median and quartile, an objective quantitative standard can be provided for judging outliers. These statistics reflect the central tendency and dispersion degree of the data, making the judgment of outliers more accurate and reliable. For example, by setting thresholds based on the mean and standard deviation, values ​​that are obviously deviated from the data center can be quickly identified.

[0066] Strong versatility: These statistics are basic indicators commonly used in data analysis and are applicable to all types of data. Whether it is numerical data or categorical data, you can use appropriate statistics to determine abnormal situations. For example, for continuous temperature data, you can use the mean and standard deviation to determine abnormal high or low temperature values; for discrete product category data, you can find abnormal category distributions by counting the frequency of each category.

[0067] Specifically, in step 3, missing values ​​are filled by using mean filling, median filling or interpolation methods, erroneous values ​​are corrected by using data validation rules or machine learning models, and inconsistent data are aligned and standardized.

[0068] In this embodiment: Maintaining data integrity: By filling missing values ​​with the mean, each variable in the data set can have a value, avoiding incomplete data due to missing values. This is very important for subsequent data analysis and modeling, ensuring the availability of data.

[0069] Reflect the overall trend: The mean represents the overall average level of the data. Using mean filling can reflect the central trend of the data to a certain extent, making the filled data set closer to the actual situation.

[0070] Simple and efficient: The mean filling method is relatively simple and fast in calculation, which is suitable for fast processing of large-scale data sets.

[0071] Specifically, in step 3, the context information and business rules of the data are considered to ensure that the cleaned data meets the actual application requirements.

[0072] In this embodiment: Enhanced data accuracy: The contextual information of the data can provide more background and meaning about the data. By considering the contextual information for data cleaning, it is possible to more accurately determine whether there is a problem with the data and how to deal with it. For example, in time series data, data anomalies at a certain point in time may be judged as reasonable values ​​after considering the data trends at the previous and next time points, rather than simply treating them as erroneous values.

[0073] Improve data relevance: Contextual information helps discover the associations between data. In the cleaning process, these associations can be used to better handle missing values, inconsistencies, and other issues. For example, in social network data, a user's behavior may be affected by his or her social circle. By considering the user's social context information, missing user behavior data can be more reasonably filled or erroneous data can be corrected.

[0074] Specifically, in step 4, data processing tools and programming languages ​​are used to perform cleaning operations, paying attention to the security and privacy of the data.

[0075] In this embodiment: Improve efficiency: Professional data processing tools and programming languages ​​usually have efficient data processing capabilities and rich functional modules. They can quickly process large-scale data, reduce the time and errors of manual operations, and improve the efficiency of data cleaning. For example, using Python's Pandas library, data filtering, sorting, merging and other operations can be easily performed, greatly speeding up the speed of data cleaning.

[0076] Flexibility and customizability: Data processing tools and programming languages ​​allow users to customize data cleaning operations according to specific needs. Specific code logic can be written to handle complex data problems and meet the requirements of different data cleaning tasks. For example, using R language, you can write custom functions according to specific business rules to accurately clean and transform data.

[0077] Specifically, in step 4, the machine learning model is used again for evaluation or compared with known high-quality data for verification, and the cleaning strategy and parameters are readjusted if the requirements are not met.

[0078] In this embodiment: Objective accuracy: The machine learning model can be evaluated based on the characteristics and patterns of the data, without relying on subjective judgment, and can provide objective and accurate evaluation results. For example, by training a classification model to determine whether the cleaned data is accurately classified, the output of the model can clearly indicate the quality status of the data.

[0079] Adapting to complex data: Machine learning models have powerful processing capabilities for complex data structures and relationships in big data environments. They can capture nonlinear relationships and potential patterns in the data, thereby more comprehensively evaluating the effectiveness of data cleaning. For example, in image data cleaning, machine learning models can identify specific features in the image and evaluate whether the quality of the cleaned image meets the requirements.

[0080] Specifically, in step 5, a feedback mechanism is established through user feedback, data analysis reports, automatic monitoring, etc., data monitoring is performed using tools such as Grafana and Prometheus, and alarm rules and notification methods are set.

[0081] In this embodiment: Close to actual needs: Users are the final users of data, and their feedback directly reflects the effect of cleaned data in actual applications. By collecting user feedback, we can understand whether the data really meets the needs of users and which aspects need further improvement.

[0082] Discover potential problems: Users may discover some problems that are not easily detected at the technical level, such as problems with data readability and usability. These feedbacks can help the data cleaning team adjust strategies in a timely manner and improve data quality.

[0083] Improve user satisfaction: Actively responding to user feedback and continuously improving data quality can enhance user trust and satisfaction with data products.

[0084] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent data cleaning method in a big data environment, characterized in that: Including the following steps: Step 1: Data collection and preprocessing: Collect big data from various data sources, convert and standardize the data to ensure data integrity and consistency; Step 2, intelligent detection and identification: use machine learning algorithms and statistical analysis methods to detect abnormal values ​​and outliers in the data. Parameters include the learning rate of the classifier is set to 0.1, the number of iterations is 1000, and the indicators include the proportion of abnormal values ​​not exceeding 5%, the number of outliers is less than 3% of the total data, etc. Step 3: Intelligent cleaning strategy formulation: formulate corresponding cleaning strategies according to the problem type, including the processing of missing values, error values ​​and inconsistent data, while considering the context information and business rules of the data; Step 4, cleaning execution and verification: perform data cleaning operations according to the cleaning strategy, and verify through machine learning model evaluation or comparison with high-quality data. The indicators include accuracy reaching more than 90%, recall reaching more than 85%, F1 value reaching more than 88%, similarity greater than 95%, difference less than 5%, etc.; Step 5: Continuous optimization and monitoring: Establish a feedback mechanism to continuously optimize cleaning strategies and algorithms, while monitoring the data cleaning process in real time.

2. The intelligent data cleaning method in a big data environment according to claim 1, characterized in that: In step 1, the data in the database is collected in batches using a database connection tool, the data in the file system is collected through a file reading interface, and the network data is collected using a network crawler technology.

3. The intelligent data cleaning method in a big data environment according to claim 1, characterized in that: In step 1, format conversion includes converting data in different formats into a unified format, and standardization processing includes standardizing the field name, data type, encoding method, etc. of the data.

4. The intelligent data cleaning method in a big data environment according to claim 1, characterized in that: In step 2, a classifier is trained using supervised learning algorithms and unsupervised learning algorithms to distinguish between normal data and abnormal data.

5. The intelligent data cleaning method in a big data environment according to claim 1, characterized in that: In step 2, by calculating the mean, standard deviation, median, quartiles and other statistics of the data, outliers are determined according to the set thresholds, and outliers are detected using visualization methods such as box plots and scatter plots or distance-based methods.

6. The intelligent data cleaning method in a big data environment according to claim 1, characterized in that: In step 3, missing values ​​are filled by using mean filling, median filling or interpolation methods, erroneous values ​​are corrected by using data validation rules or machine learning models, and inconsistent data are aligned and standardized.

7. The intelligent data cleaning method in a big data environment according to claim 1, characterized in that: In step 3, the context information and business rules of the data are considered to ensure that the cleaned data meets the actual application requirements.

8. The intelligent data cleaning method in a big data environment according to claim 1, characterized in that: In step 4, data cleaning operations are performed using data processing tools and programming languages, paying attention to data security and privacy.

9. The intelligent data cleaning method in a big data environment according to claim 1, characterized in that: In step 4, the cleaning strategy and parameters are readjusted if the requirements are not met by re-evaluating the machine learning model or comparing it with known high-quality data for verification.

10. The intelligent data cleaning method in a big data environment according to claim 1, characterized in that: In step 5, a feedback mechanism is established through user feedback, data analysis reports, automatic monitoring, etc., data monitoring is performed using tools such as Grafana and Prometheus, and alarm rules and notification methods are set.