Big data cleaning processing method based on artificial intelligence
By employing an AI-based big data cleaning method, utilizing intelligent agents and a real-time data stream processing framework, and combining reinforcement learning and deep learning for multimodal data cleaning, this approach solves the problems of high cost and long cycle in traditional methods. It achieves efficient and automated data cleaning and management, ensuring data consistency and privacy protection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-11
- Publication Date
- 2026-04-07
AI Technical Summary
Traditional big data cleaning methods are costly and time-consuming when dealing with massive amounts of data. They cannot adapt to changes in data distribution and require continuous manual updates to rules, which affects the accuracy and efficiency of data analysis.
This paper adopts an AI-based big data cleaning method, which uses intelligent agents to automatically identify data sources, integrates a real-time data stream processing framework, combines reinforcement learning and deep learning algorithms to perform multimodal data fusion and cleaning, applies federated learning to protect data privacy, constructs a knowledge graph to achieve semantic-level integration, and manages data through a distributed storage system.
It achieves efficient and automated big data cleaning, ensuring data consistency and accuracy, protecting data privacy, providing a transparent cleaning process, supporting real-time data access and optimization models, and adapting to different data quality standards.
Smart Images
Figure CN121807820A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of big data processing technology, specifically relating to a big data cleaning and processing method based on artificial intelligence. Background Technology
[0002] With the advent of the big data era, the amount of data has exploded. However, raw data often contains a lot of noise, inconsistencies, incompleteness, and duplicate information, which seriously affects the accuracy and efficiency of subsequent data analysis.
[0003] Traditional data cleaning methods mainly rely on rule engines and manual intervention. However, when faced with massive amounts of data, manual cleaning is costly and time-consuming, making it unable to adapt to changes in data distribution and requiring continuous manual updates to the rules. Therefore, those skilled in the art have provided an artificial intelligence-based big data cleaning and processing method to solve the problems mentioned in the background. Summary of the Invention
[0004] The purpose of this invention is to provide a simple and rationally designed big data cleaning and processing method based on artificial intelligence in order to solve the above problems.
[0005] The present invention achieves the above objectives through the following technical solutions:
[0006] A big data cleaning and processing method based on artificial intelligence includes the following steps:
[0007] S1: Data collection and import. Collect data from various data sources, use intelligent agents to automatically identify and collect relevant data sources, reduce manual intervention, and import the collected data into the big data processing platform. At the same time, integrate real-time data stream processing frameworks such as Apache Kafka or Apache Flink to realize real-time data collection and import.
[0008] S2: Data preprocessing adopts a self-learning cleaning model. Through reinforcement learning algorithms, the model can automatically adjust the cleaning strategy according to the changes in data and the cleaning effect. In addition, for multimodal data, we introduce multimodal data fusion technology and use deep learning models to perform cross-modal data cleaning to ensure the consistency and accuracy of different types of data.
[0009] S3: Data transformation, which transforms data into a unified format and scale using normalization or standardization, converts categorical data into numerical form, automatically extracts features using deep learning models, and realizes dynamic data mapping and transformation based on graph neural networks;
[0010] S4.1: Data integration. By applying federated learning technology, multi-source data can be fused without exposing the original data. Federated learning protects data privacy by training models locally and sharing only model parameters. At the same time, it constructs knowledge graphs to transform data into semantic networks, achieving semantic-level data integration.
[0011] S4.2: After data integration, data verification and quality assessment are required. Meta-learning technology is used to enable the model to quickly adapt to different data quality assessment standards. Interpretable AI technologies, such as LIME or SHAP, are introduced to provide transparency and interpretability in the data cleaning process.
[0012] S5: Data storage and output. The cleaned data is stored in a data warehouse or data lake for subsequent analysis and application. At the same time, a distributed data lake is built, using distributed storage systems such as HDFS or Amazon S3 to realize the storage and management of massive amounts of data. Data-as-a-service interfaces are provided to facilitate data access and use through APIs or Web services, meeting the needs of different users.
[0013] As a further optimization of the present invention, intelligent data collectors are introduced in step S1. These collectors can automatically identify and connect to various data sources, including traditional databases, NoSQL databases, API interfaces and IoT devices. Through machine learning algorithms, the collectors can dynamically adjust the collection strategy and prioritize the collection of high-value data.
[0014] As a further optimization of the present invention, step S1 integrates a real-time data stream processing framework such as Apache Kafka or Apache Flink, which can realize real-time data acquisition and import, and ensure the timeliness of data processing.
[0015] As a further optimization of the present invention, in step S3, the StandardScaler and MinMax Scaler of Scikit-learn are used to standardize and normalize the data, and a deep learning model is used to automatically extract features, while an autoencoder is used for unsupervised feature extraction.
[0016] As a further optimization of the present invention, in step S5, after data storage and output, it is necessary to monitor data quality in real time, identify new data problems, and continuously optimize the data cleaning model based on the monitoring results to improve the cleaning effect.
[0017] As a further optimization of the present invention, step S3 requires data normalization and standardization, wherein the formulas for Min-Max normalization and Z-score standardization are as follows:
[0018]
[0019] Where, x ′ The values represent the normalized or standardized values, x represents the original data points, μ is the mean of the data, σ is the standard deviation of the data, min(x) is the minimum value of the data, and max(x) is the maximum value of the data.
[0020] As a further optimization of the present invention, step S1 requires processing missing values of the data, using the mean to impute numerical data, using the median to impute non-numerical data or data with outliers, and using similarity calculation to delete duplicate data, which requires the following formula:
[0021]
[0022] Where J(A,B) is the similarity between A and B, A∩B is the size of the intersection of A and B, and A∪B is the size of the union of A and B.
[0023] Cosine similarity:
[0024]
[0025] Where cosine(A,B) is the cosine similarity between A and B, A·B is the point set of A and B, ‖A‖ is the norm of vector A, and ‖B‖ is the norm of vector B.
[0026] As a further optimization of the present invention, in step S4.1, federated learning technology is used in the data cleaning process to protect data privacy. Combined with interpretable AI technology, the data cleaning model is explained and visualized to help users understand the model's decision-making process and improve transparency and trust.
[0027] The beneficial effects of this invention are as follows:
[0028] 1. In this invention, intelligent agents and collectors are used to automatically identify and connect multi-source data. Apache Kafka / Flink is integrated to achieve real-time data stream acquisition and processing. The acquisition strategy is dynamically adjusted through machine learning, prioritizing the acquisition of high-value data. At the same time, reinforcement learning algorithms are introduced so that the cleaning model can automatically adjust the cleaning strategy according to data changes and cleaning effects. Furthermore, multimodal data fusion technology is adopted, and deep learning is used for cross-modal data cleaning to ensure data consistency and accuracy.
[0029] 2. In this invention, normalization and standardization are used to unify the data scale. Deep learning models and autoencoders are combined to perform unsupervised feature extraction. Graph neural networks are introduced to realize dynamic data mapping and transformation. Federated learning technology is applied to achieve multi-source data fusion without exposing the original data, thus protecting data privacy. A knowledge graph is constructed to transform the data into a semantic network and achieve semantic-level data integration.
[0030] 3. In this invention, a distributed data lake is constructed using HDFS / Amazon S3, which supports massive data storage, provides a data-as-a-service interface for easy data access and application, implements real-time data quality monitoring, and continuously optimizes the cleaning model. Attached Figure Description
[0031] Figure 1 This is a schematic diagram of the overall process of the present invention;
[0032] Figure 2 This is a schematic diagram of the overall process of data collection and import in this invention;
[0033] Figure 3 This is a schematic diagram of the overall data conversion process of the present invention;
[0034] Figure 4 This is a schematic diagram of the overall process of data storage and output in this invention. Detailed Implementation
[0035] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.
[0036] Example: Figure 1 - Figure 4 As shown, an artificial intelligence-based big data cleaning and processing method includes the following steps:
[0037] S1: Data collection and import. Collect data from various data sources, use intelligent agents to automatically identify and collect relevant data sources, reduce manual intervention, and import the collected data into the big data processing platform. At the same time, integrate real-time data stream processing frameworks such as Apache Kafka or Apache Flow to realize real-time data collection and import. Data sources come from databases, file systems, APIs, etc. The big data processing platform can use Hadoop, Spark, etc.
[0038] S2: Data preprocessing adopts a self-learning cleaning model. Through reinforcement learning algorithms, the model can automatically adjust the cleaning strategy according to the changes in data and the cleaning effect. In addition, for multimodal data, we introduce multimodal data fusion technology and use deep learning models to perform cross-modal data cleaning to ensure the consistency and accuracy of different types of data.
[0039] S3: Data transformation, which transforms data into a unified format and scale using normalization or standardization, converts categorical data into numerical form, and automatically extracts features using deep learning models. At the same time, it realizes dynamic data mapping and transformation based on graph neural networks. Among them, the transformation of categorical data into numerical form is such as one-hot encoding and label encoding.
[0040] S4.1: Data integration. By applying federated learning technology, multi-source data can be fused without exposing the original data. Federated learning protects data privacy by training models locally and sharing only model parameters. At the same time, it constructs knowledge graphs to transform data into semantic networks, achieving semantic-level data integration.
[0041] S4.2: After data integration, data verification and quality assessment are required. Meta-learning technology is used to enable the model to quickly adapt to different data quality assessment standards. Interpretable AI technologies, such as LIME or SHAP, are introduced to provide transparency and interpretability of the data cleaning process. Quality assessment evaluates the effectiveness of data cleaning through indicators such as accuracy, recall, F1-score, etc.
[0042] S5: Data storage and output. The cleaned data is stored in a data warehouse or data lake for subsequent analysis and application. At the same time, a distributed data lake is built, using distributed storage systems such as HDFS or Amazon S3 to realize the storage and management of massive amounts of data. Data-as-a-service interfaces are provided to facilitate data access and use through APIs or Web services, meeting the needs of different users.
[0043] like Figure 1 - Figure 4 As shown, step S1 introduces intelligent data collectors. These collectors can automatically identify and connect to various data sources, including traditional databases, NoSQL databases, API interfaces, and IoT devices. Through machine learning algorithms, the collectors can dynamically adjust their collection strategies and prioritize the collection of high-value data.
[0044] like Figure 1 - Figure 4As shown, step S1 integrates a real-time data stream processing framework such as Apache Kafka or Apache Flink, which can realize real-time data acquisition and import, ensuring the timeliness of data processing.
[0045] like Figure 1 - Figure 4 As shown, in step S3, the Standard Scaler and MinMaxScaler of Scikit-learn are used to standardize and normalize the data, and a deep learning model is used to automatically extract features. At the same time, an autoencoder is used for unsupervised feature extraction.
[0046] like Figure 1 - Figure 4 As shown, in step S5, after data storage and output, it is necessary to monitor data quality in real time, identify new data problems, and continuously optimize the data cleaning model based on the monitoring results to improve the cleaning effect.
[0047] like Figure 1 - Figure 4 As shown, step S3 requires data normalization and standardization. The formulas for Min-Max normalization and Z-score standardization are as follows:
[0048]
[0049] Where, x ′ The values represent the normalized or standardized values, x represents the original data points, μ is the mean of the data, σ is the standard deviation of the data, min(x) is the minimum value of the data, and max(x) is the maximum value of the data.
[0050] like Figure 1 - Figure 4 As shown, step S1 requires handling missing values in the data. Mean imputation is used for numerical data, and median imputation is used for non-numerical data or data with outliers. Simultaneously, similarity calculation is used to delete duplicate data. The following formula is required:
[0051]
[0052] Where J(A,B) is the similarity between A and B, A∩B is the size of the intersection of A and B, and A∪B is the size of the union of A and B.
[0053] Cosine similarity:
[0054]
[0055] Where cosine(A,B) is the cosine similarity between A and B, A·B is the point set of A and B, ‖A‖ is the norm of vector A, and ‖B‖ is the norm of vector B.
[0056] like Figure 1 - Figure 4 As shown, in step S4.1, federated learning technology is used in the data cleaning process to protect data privacy. Combined with interpretable AI technology, it provides explanations and visualizations of the data cleaning model, helping users understand the model's decision-making process and improving transparency and trust.
[0057] It should be noted that this AI-based big data cleaning and processing method involves data collection and import from various data sources. Intelligent agents automatically identify and collect relevant data sources, reducing manual intervention. The collected data is then imported into a big data processing platform, integrating Apache Kafka or Apache... Real-time data stream processing frameworks such as Flink are used to achieve real-time data acquisition and import. A self-learning cleaning model is employed, using reinforcement learning algorithms to automatically adjust the cleaning strategy based on data changes and cleaning effectiveness. Furthermore, for multimodal data, multimodal data fusion technology is introduced, utilizing deep learning models for cross-modal data cleaning to ensure consistency and accuracy across different data types. Data is normalized or standardized to a unified format and scale, categorical data is converted to numerical form, and features are automatically extracted using deep learning models. Dynamic data mapping and transformation are achieved based on graph neural networks. Federated learning technology is applied to achieve multi-source data fusion without exposing the original data. Federated learning protects data privacy by training models locally and sharing only model parameters. A knowledge graph is constructed to transform data into a semantic network, achieving semantic-level data integration. Meta-learning technology is used to enable the model to quickly adapt to different data quality assessment standards. Interpretable AI technologies, such as LIME or SHAP, are introduced to provide transparency and interpretability in the data cleaning process. Cleaned data is stored in a data warehouse or data lake for subsequent analysis and application. A distributed data lake is also constructed, utilizing HDFS or Amazon. Distributed storage systems such as S3 enable the storage and management of massive amounts of data, provide data-as-a-service interfaces, and facilitate data access and use through APIs or web services to meet the needs of different users.
[0058] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
Claims
1. A big data cleaning and processing method based on artificial intelligence, characterized in that: Includes the following steps: S1: Data collection and import. Collect data from various data sources, use intelligent agents to automatically identify and collect relevant data sources, reduce manual intervention, and import the collected data into the big data processing platform. At the same time, integrate real-time data stream processing frameworks such as Apache Kafka or Apache Filter to realize real-time data collection and import. S2: Data preprocessing adopts a self-learning cleaning model. Through reinforcement learning algorithms, the model can automatically adjust the cleaning strategy according to the changes in data and the cleaning effect. In addition, for multimodal data, we introduce multimodal data fusion technology and use deep learning models to perform cross-modal data cleaning to ensure the consistency and accuracy of different types of data. S3: Data transformation, which transforms data into a unified format and scale using normalization or standardization, converts categorical data into numerical form, automatically extracts features using deep learning models, and realizes dynamic data mapping and transformation based on graph neural networks; S4.1: Data integration. By applying federated learning technology, multi-source data can be fused without exposing the original data. Federated learning protects data privacy by training models locally and sharing only model parameters. At the same time, it constructs knowledge graphs to transform data into semantic networks, achieving semantic-level data integration. S4.2: After data integration, data verification and quality assessment are required. Meta-learning technology is used to enable the model to quickly adapt to different data quality assessment standards. Interpretable AI technologies, such as LIME or SHAP, are introduced to provide transparency and interpretability in the data cleaning process. S5: Data storage and output. The cleaned data is stored in a data warehouse or data lake for subsequent analysis and application. At the same time, a distributed data lake is built, using distributed storage systems such as HDFS or Amazon S3 to realize the storage and management of massive amounts of data. Data-as-a-service interfaces are provided to facilitate data access and use through APIs or Web services, meeting the needs of different users.
2. The big data cleaning and processing method based on artificial intelligence according to claim 1, characterized in that: In step S1, intelligent data collectors are introduced. These collectors can automatically identify and connect to various data sources, including traditional databases, NoSQL databases, API interfaces, and IoT devices. Through machine learning algorithms, the collectors can dynamically adjust their collection strategies and prioritize the collection of high-value data.
3. The big data cleaning and processing method based on artificial intelligence according to claim 1, characterized in that: Step S1 integrates a real-time data stream processing framework such as Apache Kafka or Apache Flink, which can realize real-time data acquisition and import, ensuring the timeliness of data processing.
4. The big data cleaning and processing method based on artificial intelligence according to claim 1, characterized in that: In step S3, the Standard Scaler and MinMax Scaler of Scikit-learn are used to standardize and normalize the data, and a deep learning model is used to automatically extract features. At the same time, an autoencoder is used for unsupervised feature extraction.
5. The big data cleaning and processing method based on artificial intelligence according to claim 1, characterized in that: In step S5, after data storage and output, it is necessary to monitor data quality in real time, identify new data problems, and continuously optimize the data cleaning model based on the monitoring results to improve the cleaning effect.
6. The big data cleaning and processing method based on artificial intelligence according to claim 1, characterized in that: In step S3, the data needs to be normalized and standardized. The formulas for Min-Max normalization and Z-score normalization are as follows: Where, x ′ The values represent the normalized or standardized values, x represents the original data points, μ is the mean of the data, σ is the standard deviation of the data, min(x) is the minimum value of the data, and max(x) is the maximum value of the data.
7. The big data cleaning and processing method based on artificial intelligence according to claim 1, characterized in that: In step S1, missing values in the data need to be handled. Mean imputation is used for numerical data, and median imputation is used for non-numerical data or data with outliers. Simultaneously, similarity calculation is used to delete duplicate data, requiring the following formula: Where J(A,B) is the similarity between A and B, A∩B is the size of the intersection of A and B, and A∪B is the size of the union of A and B. Cosine similarity: Where cosine(A,B) is the cosine similarity between A and B, A·B is the point set of A and B, ‖A‖ is the norm of vector A, and ‖B‖ is the norm of vector B.
8. The big data cleaning and processing method based on artificial intelligence according to claim 1, characterized in that: In step S4.1, federated learning technology is used during the data cleaning process to protect data privacy. Combined with interpretable AI technology, it provides explanations and visualizations of the data cleaning model, helping users understand the model's decision-making process and improving transparency and trust.