Data information processing method and system based on cloud computing
Through the cloud-based data information processing system, the problem of performance bottlenecks and insufficient intelligent analysis capabilities of traditional data processing systems is solved, real-time processing and efficient storage of large-scale data is realized, and data analysis capabilities and system availability are improved.
Patent Information
- Application Number
- CN202410908003.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-08
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Traditional data processing systems have performance bottlenecks when processing large-scale data sets, which are difficult to meet the needs of real-time analysis and processing, and lack the ability to learn in-depth data and intelligent analysis.
Adopt cloud computing-based data information processing system, including data acquisition module, data preprocessing module, data storage module, data analysis module and task scheduling module, through data stream processing algorithms, ETL processes, distributed file systems, NoSQL databases, and multiple This data processing framework, machine learning and optimization algorithms can realize real-time data collection, cleaning, storage, analysis and resource scheduling.
Real-time processing and efficient storage of large-scale data is realized, data processing speed and analysis capabilities are improved, real-time analysis needs are met, and the intelligence level of data analysis and high system availability are improved through intelligent algorithms.
Smart Images

Figure CN119025508A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data information processing, and in particular to a data information processing method and system based on cloud computing. Background Art
[0002] With the rapid development of information technology, especially the widespread application of the Internet and the Internet of Things, the generation, collection and analysis of data have become increasingly important. Enterprises and individuals need to extract valuable information from massive amounts of data to support decision-making, trend forecasting, market analysis and other business needs. However, traditional data processing systems often have performance bottlenecks when processing large-scale data sets, making it difficult to meet the needs of real-time analysis and processing.
[0003] After searching, the Chinese patent application number 201510698834.9 discloses a data information processing system including an electronic tag generation module, which is used to generate electronic tags that record data information; an electronic tag exchange module, which is used for users to obtain or exchange electronic tags; an electronic tag identification module, which is used to identify and decode electronic tags into the data information; an information card management module, which is used to perform custom management of information cards, and the custom management includes one, two or more combinations of the following: adding information cards, deleting information cards, modifying information cards, classifying information cards, rating information cards, setting information card priorities, and authenticating information card information. The electronic tag generation module in the above patent has the following shortcomings: it mainly relies on the generation and identification technology of electronic tags, lacks the ability to conduct in-depth learning and intelligent analysis of data, and needs to be improved. Summary of the invention
[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a data information processing method and system based on cloud computing.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] A data information processing system based on cloud computing, comprising:
[0007] Data acquisition module: responsible for collecting raw data from multiple sources; using data stream processing algorithms to ensure real-time data collection and transmission;
[0008] Data preprocessing module: cleans, formats and transforms the collected raw data; uses ETL process and data quality scoring mechanism to improve data accuracy and availability;
[0009] Data storage module: uses distributed file system and NoSQL database to support efficient storage and fast retrieval of big data; at the same time, applies data sharding and redundancy strategies to ensure data security and reliability;
[0010] Data analysis module: Integrates multiple data processing frameworks to perform complex data processing tasks, including data mining, machine learning, and statistical analysis; introduces the MapReduce programming model to optimize data processing procedures and improve processing speed;
[0011] Task scheduling module: through dynamic resource allocation algorithm, reasonably allocate computing resources and tasks to optimize system performance; use load balancing technology to avoid single point failure and ensure high availability of the system;
[0012] User interface module: provides a graphical interface that allows users to easily configure data processing parameters, monitor task progress, and display results; the application of RESTful API supports the integration of third-party services and applications.
[0013] As a preferred embodiment of the present invention, the data information processing method of the system comprises the following steps:
[0014] S1: The system first acquires data through the data acquisition module;
[0015] S2: Cleaned and formatted by the pre-processing module;
[0016] S3: Data is stored in a distributed file system;
[0017] S4: The data analysis module and the intelligent algorithm module process the data according to the preset analysis model and algorithm;
[0018] S5: Optimize resource usage through task scheduling module;
[0019] S6: The analysis results are displayed to the user through the user interface module.
[0020] As a preferred embodiment of the present invention, the data acquisition module includes:
[0021] Real-time data stream receiving unit: uses Kafka or Flume to receive real-time data streams;
[0022] Multi-source data integration unit: Integrates data from sensors, social media, and enterprise databases;
[0023] Data verification and monitoring unit: verify data integrity and monitor the quality of data flow;
[0024] The data preprocessing module comprises:
[0025] Data cleaning unit: remove duplicate and erroneous data;
[0026] Data conversion unit: convert data into a unified format and structure;
[0027] Data enrichment unit: adding additional information to enhance the data;
[0028] The data storage module comprises:
[0029] Distributed storage unit: Use HDFS for distributed storage of data;
[0030] Cache unit: Use memory cache system to improve data access speed;
[0031] Data backup and recovery unit: ensure data security and support disaster recovery.
[0032] As a preferred embodiment of the present invention: the data analysis module includes:
[0033] Batch processing unit: Use Hadoop for big data batch processing;
[0034] Real-time analysis unit: Use Spark for real-time data calculation and analysis;
[0035] Model building unit: builds prediction models through machine learning libraries;
[0036] The data analysis module uses a linear regression model to construct a linear relationship between feature X and output y; the formula is:
[0037] y=β0+β1X1+β2X2+…+β n X n +∈
[0038] y is the response variable, that is, the value you want to predict;
[0039] X1,X2,…,X n is a characteristic variable used to predict different factors of y;
[0040] β0 is the intercept term, the expected value of y when all characteristic variables are 0;
[0041] β1,β2,…,β n is the coefficient of the corresponding feature, indicating the average impact of each unit increase of the feature on y;
[0042] ∈ is the error term, which represents the random variability that cannot be explained by the model.
[0043] As a preferred embodiment of the present invention, the task scheduling module includes: a resource management unit: using YARN to allocate and manage resources; a task queue unit: optimizing the task execution sequence and managing the data pipeline; a fault recovery unit: realizing the automatic recovery and retry mechanism of the task:
[0044]
[0045] x j ∈{0,1}
[0046] m: is the total number of resources;
[0047] n: is the total number of tasks;
[0048] c ij : is the cost of assigning task j to resource i;
[0049] x ij : If task j is assigned to resource i, it is 1, otherwise it is 0; r j : The amount of resources required for task j;
[0050] R i : The total amount of resource i.
[0051] As a preferred embodiment of the present invention, the user interface module includes:
[0052] Configuration interface unit: allows users to configure data processing parameters;
[0053] Monitoring dashboard unit: displays system status and processing progress in real time; result visualization unit: displays analysis results in the form of charts and reports.
[0054] As a preferred embodiment of the present invention: it also includes:
[0055] Machine Learning Module: Apply supervised and unsupervised learning algorithms to predict data trends and pattern recognition; use random forest, support vector machine or neural network algorithms to train models based on historical data to intelligently predict and classify new data;
[0056] Natural Language Processing Module: For text data, it uses NLP technology to understand, parse and generate human language, and supports in-depth analysis of social media and customer feedback data;
[0057] Optimization algorithm module: Use genetic algorithms, simulated annealing and ant colony optimization techniques to solve optimization problems in resource allocation, task scheduling and data processing, and improve system operation efficiency and resource utilization;
[0058] Anomaly Detection Module: Combines cluster analysis and outlier detection algorithms to identify anomalies or outliers in a data set for use in fraud detection and system health monitoring scenarios.
[0059] As a preferred embodiment of the present invention, the machine learning module includes:
[0060] Predictive Analysis Unit: Use random forest and support vector machine for trend prediction;
[0061] Cluster analysis unit: Apply K-means and hierarchical clustering to group data.
[0062] The machine learning module uses K-means to group data. The goal of the K-means algorithm is to minimize the sum of the distances from each point to its cluster center. The optimization problem of the algorithm is expressed as:
[0063]
[0064] in:
[0065] k is the number of clusters;
[0066] x j is a data point;
[0067] S i is the set of data points assigned to cluster i;
[0068] μi is the center of cluster i.
[0069] As a preferred embodiment of the present invention: the natural language processing module includes:
[0070] Text understanding unit: realizes sentiment analysis and intent recognition;
[0071] Language Production Unit: Produce reports and summaries;
[0072] The optimization algorithm module comprises:
[0073] Resource optimization unit: optimize resource usage through genetic algorithms;
[0074] Task Scheduling Optimization Unit: Improving task scheduling strategies using simulated annealing.
[0075] As a preferred embodiment of the present invention: the abnormality detection module includes:
[0076] Outlier detection unit: Use DBSCAN to identify abnormal data;
[0077] Abnormal pattern analysis unit: Analyze the patterns and causes of abnormal data;
[0078] Among them, the anomaly detection module divides the area with sufficiently high density into clusters through DBSCAN, and finds abnormal points in the low-density area. The neighborhood of the point is defined as all points in the area with point p as the center and Eps as the radius, which is specifically expressed as:
[0079] kN(p)={q∈D|dist(p,q)≤Eps}
[0080] in:
[0081] D is the data set;
[0082] dist(p,q) is the distance between point p and point q;
[0083] Eps is the given neighborhood radius.
[0084] The beneficial effects of the present invention are:
[0085] 1. The present invention has good flexibility and scalability. Through modular design, the system can flexibly add or remove functional modules to adapt to different business needs. At the same time, cloud computing infrastructure provides easily scalable resources to support the processing of large-scale data sets.
[0086] 2. The present invention is capable of real-time data processing. The data acquisition module uses a data stream processing algorithm to ensure real-time data collection and transmission to meet the needs of real-time analysis. The data preprocessing module improves the accuracy and availability of data through the ETL process and data quality scoring mechanism, providing a clean and reliable data foundation for subsequent analysis.
[0087] 3. The data storage module of the present invention adopts a distributed file system and a NoSQL database to support efficient storage and rapid retrieval of big data while ensuring the security and reliability of the data; the data analysis module integrates multiple data processing frameworks to perform complex data processing tasks, including data mining, machine learning, and statistical analysis, to improve processing speed.
[0088] 4. The task scheduling module of the present invention optimizes system performance and ensures high availability of the system through dynamic resource allocation algorithm and load balancing technology; the system integrates intelligent algorithm modules such as machine learning, natural language processing, optimization algorithm and anomaly detection, which improves the intelligence level of data analysis and decision support. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] Figure 1 This is a flow chart of a data information processing method based on cloud computing proposed by the present invention. DETAILED DESCRIPTION
[0090] The technical solution of this patent is further described in detail below in conjunction with specific implementation methods.
[0091] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0092] Embodiment 1:
[0093] A data information processing system based on cloud computing, comprising:
[0094] Data collection module: responsible for collecting raw data from multiple sources (such as sensors, social media, enterprise databases, etc.); using efficient data stream processing algorithms, such as Kafka or Flume, to ensure real-time data collection and transmission;
[0095] Data preprocessing module: cleans, formats and transforms the collected raw data; uses ETL (extraction, transformation, loading) process and data quality scoring mechanism to improve data accuracy and availability;
[0096] Data storage module: uses distributed file systems (such as HDFS) and NoSQL databases (such as MongoDB) to support efficient storage and fast retrieval of big data; at the same time, data sharding and redundancy strategies are applied to ensure data security and reliability;
[0097] Data analysis module: Integrates multiple data processing frameworks (such as Spark and Hadoop) to perform complex data processing tasks, including data mining, machine learning, and statistical analysis; introduces the MapReduce programming model to optimize data processing procedures and improve processing speed;
[0098] Task scheduling module: through dynamic resource allocation algorithms (such as YARN), reasonably allocate computing resources and tasks to optimize system performance; use load balancing technology to avoid single point failures and ensure high availability of the system;
[0099] User interface module: provides a graphical interface that allows users to easily configure data processing parameters, monitor task progress, and display results; the application of RESTful API supports the integration of third-party services and applications.
[0100] The data information processing method of the system comprises the following steps:
[0101] S1: The system first acquires data through the data acquisition module;
[0102] S2: Cleaned and formatted by the pre-processing module;
[0103] S3: Data is stored in a distributed file system;
[0104] S4: The data analysis module and the intelligent algorithm module process the data according to the preset analysis model and algorithm;
[0105] S5: Optimize resource usage through task scheduling module;
[0106] S6: The analysis results are displayed to the user through the user interface module.
[0107] Wherein, the data acquisition module includes:
[0108] Real-time data stream receiving unit: uses Kafka or Flume to receive real-time data streams;
[0109] Multi-source data integration unit: Integrates data from sensors, social media, and enterprise databases;
[0110] Data verification and monitoring unit: Verify data integrity and monitor the quality of data flow.
[0111] Wherein, the data preprocessing module includes:
[0112] Data cleaning unit: remove duplicate and erroneous data;
[0113] Data conversion unit: convert data into a unified format and structure;
[0114] Data enrichment unit: Add additional information such as location, time tags, etc. to enhance the data.
[0115] Wherein, the data storage module includes:
[0116] Distributed storage unit: Use HDFS for distributed storage of data;
[0117] Cache unit: Use memory cache systems such as Redis to improve data access speed;
[0118] Data backup and recovery unit: ensure data security and support disaster recovery.
[0119] Wherein, the data analysis module includes:
[0120] Batch processing unit: Use Hadoop for big data batch processing;
[0121] Real-time analysis unit: Use Spark for real-time data calculation and analysis;
[0122] Model building unit: builds prediction models through machine learning libraries;
[0123] The data analysis module uses a linear regression model to construct a linear relationship between feature X and output y; the formula is:
[0124] y=β0+β1X1+β2X2+…+β n X n +∈
[0125] y is the response variable, that is, the value you want to predict;
[0126] X1,X2,…,X n is a characteristic variable used to predict different factors of y;
[0127] β0 is the intercept term, the expected value of y when all characteristic variables are 0;
[0128] β1,β2,…,β n is the coefficient of the corresponding feature, indicating the average impact of each unit increase of the feature on y;
[0129] ∈ is the error term, which represents the random variability that cannot be explained by the model.
[0130] Wherein, the task scheduling module includes:
[0131] Resource management unit: uses YARN for resource allocation and management;
[0132] Task queue unit: optimizes the order of task execution and manages data pipelines;
[0133] Fault recovery unit: implements automatic recovery and retry mechanism of tasks:
[0134]
[0135] x ij ∈{0,1}
[0136] m: is the total number of resources;
[0137] n: is the total number of tasks;
[0138] c ij : is the cost of assigning task j to resource i;
[0139] x ij : If task j is assigned to resource i, it is 1, otherwise it is 0;
[0140] r j : The amount of resources required for task j;
[0141] R i : The total amount of resource i.
[0142] Wherein, the user interface module includes:
[0143] Configuration interface unit: allows users to configure data processing parameters;
[0144] Monitoring dashboard unit: displays system status and processing progress in real time;
[0145] Result visualization unit: displays the analysis results in the form of charts and reports.
[0146] Embodiment 2:
[0147] A data information processing system based on cloud computing, this embodiment makes the following improvements on the basis of embodiment 1: it also includes:
[0148] Machine Learning Module: Apply supervised and unsupervised learning algorithms to predict data trends and pattern recognition; use algorithms such as random forests, support vector machines, and neural networks to train models based on historical data to intelligently predict and classify new data;
[0149] Natural Language Processing Module: For text data, it uses NLP technologies such as sentiment analysis, keyword extraction, and topic modeling to understand, parse, and generate human language, and supports in-depth analysis of data such as social media and customer feedback;
[0150] Optimization algorithm module: Use advanced optimization techniques such as genetic algorithm, simulated annealing and ant colony optimization to solve optimization problems in resource allocation, task scheduling and data processing, and improve system operation efficiency and resource utilization;
[0151] Anomaly Detection Module: Combines cluster analysis and outlier detection algorithms (such as DBSCAN and IsolationForest) to identify anomalies or outliers in a data set for application scenarios such as fraud detection and system health monitoring.
[0152] Wherein, the machine learning module includes:
[0153] Predictive Analysis Unit: Use random forest and support vector machine for trend prediction;
[0154] Cluster analysis unit: Apply K-means and hierarchical clustering to group data.
[0155] The machine learning module uses K-means to group data. The goal of the K-means algorithm is to minimize the sum of the distances from each point to its cluster center. The optimization problem of the algorithm is expressed as:
[0156]
[0157] in:
[0158] k is the number of clusters;
[0159] x j is a data point;
[0160] S i is the set of data points assigned to cluster i;
[0161] μ i is the center of cluster i.
[0162] Wherein, the natural language processing module includes:
[0163] Text understanding unit: realizes sentiment analysis and intent recognition;
[0164] Language Generation Unit: Generate reports and summaries.
[0165] Wherein, the optimization algorithm module includes:
[0166] Resource optimization unit: optimize resource usage through genetic algorithms;
[0167] Task Scheduling Optimization Unit: Improving task scheduling strategies using simulated annealing.
[0168] Wherein, the anomaly detection module includes:
[0169] Outlier detection unit: Use DBSCAN to identify abnormal data;
[0170] Abnormal pattern analysis unit: Analyze the patterns and causes of abnormal data;
[0171] Among them, the anomaly detection module divides the areas with sufficiently high density into clusters through DBSCAN, and finds abnormal points (noise) in the low-density areas. The neighborhood of the point is defined as all points in the area with point p as the center and Eps as the radius, which is specifically expressed as:
[0172] kN(p)={q∈D|dist(p,q)≤Eps}
[0173] in:
[0174] D is the data set;
[0175] dist(p,q) is the distance between point p and point q;
[0176] Eps is the given neighborhood radius.
[0177] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A data information processing system based on cloud computing, characterized in that: include: Data acquisition module: responsible for collecting raw data from multiple sources; Adopt data stream processing algorithm to ensure real-time data collection and transmission; Data preprocessing module: cleans, formats and converts the collected raw data; Use ETL processes and data quality scoring mechanisms to improve data accuracy and availability; Data storage module: uses distributed file system and NoSQL database to support efficient storage and fast retrieval of big data; at the same time, applies data sharding and redundancy strategies to ensure data security and reliability; Data analysis module: Integrates multiple data processing frameworks to perform complex data processing tasks, including data mining, machine learning, and statistical analysis; introduces the MapReduce programming model to optimize data processing procedures and improve processing speed; Task scheduling module: through dynamic resource allocation algorithm, reasonably allocate computing resources and tasks to optimize system performance; use load balancing technology to avoid single point failure and ensure high availability of the system; User interface module: provides a graphical interface that allows users to easily configure data processing parameters, monitor task progress, and display results; Application RESTful API supports integration of third-party services and applications.
2. A cloud computing-based data information processing system according to claim 1, characterized in that: The data information processing method of the system comprises the following steps: S1: The system first acquires data through the data acquisition module; S2: Cleaned and formatted by the pre-processing module; S3: Data is stored in a distributed file system; S4: The data analysis module and the intelligent algorithm module process the data according to the preset analysis model and algorithm; S5: Optimize resource usage through task scheduling module; S6: The analysis results are displayed to the user through the user interface module.
3. The data information processing system based on cloud computing according to claim 1, characterized in that: The data acquisition module comprises: Real-time data stream receiving unit: uses Kafka or Flume to receive real-time data streams; Multi-source data integration unit: Integrates data from sensors, social media, and enterprise databases; Data verification and monitoring unit: verify data integrity and monitor the quality of data flow; The data preprocessing module comprises: Data cleaning unit: remove duplicate and erroneous data; Data conversion unit: convert data into a unified format and structure; Data enrichment unit: adding additional information to enhance the data; The data storage module comprises: Distributed storage unit: Use HDFS for distributed storage of data; Cache unit: Use memory cache system to improve data access speed; Data backup and recovery unit: ensure data security and support disaster recovery.
4. The data information processing system based on cloud computing according to claim 1, characterized in that: The data analysis module comprises: Batch processing unit: Use Hadoop for big data batch processing; Real-time analysis unit: Use Spark for real-time data calculation and analysis; Model building unit: builds prediction models through machine learning libraries; The data analysis module uses a linear regression model to construct a linear relationship between feature X and output y; the formula is: y=β0+β1X1+β2X2+…+β n X n +∈ y is the response variable, that is, the value you want to predict; X1,X2,…,X n is a characteristic variable used to predict different factors of y; β0 is the intercept term, the expected value of y when all characteristic variables are 0; β1,β2,…,β n is the coefficient of the corresponding feature, indicating the average impact of each unit increase of the feature on y; ∈ is the error term, which represents the random variability that cannot be explained by the model.
5. The data information processing system based on cloud computing according to claim 1, characterized in that: The task scheduling module includes: Resource management unit: uses YARN for resource allocation and management; Task queue unit: optimizes the order of task execution and manages data pipelines; Fault recovery unit: implements automatic recovery and retry mechanism of tasks: x ij ∈{0,1} m: is the total number of resources; n: is the total number of tasks; c ij : is the cost of assigning task j to resource i; x ij : If task j is assigned to resource i, it is 1, otherwise it is 0; r j : The amount of resources required for task j; R i : The total amount of resource i.
6. The data information processing system based on cloud computing according to claim 1, characterized in that: The user interface module comprises: Configuration interface unit: allows users to configure data processing parameters; Monitoring dashboard unit: displays system status and processing progress in real time; Result visualization unit: displays the analysis results in the form of charts and reports.
7. The data information processing system based on cloud computing according to claim 1, characterized in that: Also includes: Machine Learning Module: Apply supervised and unsupervised learning algorithms to predict data trends and pattern recognition; use random forest, support vector machine or neural network algorithms to train models based on historical data to intelligently predict and classify new data; Natural Language Processing Module: For text data, it uses NLP technology to understand, parse and generate human language, and supports in-depth analysis of social media and customer feedback data; Optimization algorithm module: Use genetic algorithms, simulated annealing and ant colony optimization techniques to solve optimization problems in resource allocation, task scheduling and data processing, and improve system operation efficiency and resource utilization; Anomaly detection module: Combines cluster analysis and outlier detection algorithms to identify anomalies or outliers in a data set for fraud detection and system health monitoring application scenarios.
8. The data information processing system based on cloud computing according to claim 7, characterized in that: The machine learning module comprises: Predictive Analysis Unit: Use random forest and support vector machine for trend prediction; Cluster analysis unit: Apply K-means and hierarchical clustering to group data. The machine learning module uses K-means to group data. The goal of the K-means algorithm is to minimize the sum of the distances from each point to its cluster center. The optimization problem of the algorithm is expressed as: in: k is the number of clusters; x j is a data point; S i is the set of data points assigned to cluster i; μ i is the center of cluster i.
9. The data information processing system based on cloud computing according to claim 7, characterized in that: The natural language processing module comprises: Text understanding unit: realizes sentiment analysis and intent recognition; Language Generation Unit: Generate reports and summaries; The optimization algorithm module comprises: Resource optimization unit: optimize resource usage through genetic algorithms; Task Scheduling Optimization Unit: Improving task scheduling strategies using simulated annealing.
10. The data information processing system based on cloud computing according to claim 7, characterized in that: The anomaly detection module comprises: Outlier detection unit: Use DBSCAN to identify abnormal data; Abnormal pattern analysis unit: Analyze the patterns and causes of abnormal data; Among them, the anomaly detection module divides the area with sufficiently high density into clusters through DBSCAN, and finds abnormal points in the low-density area. The neighborhood of the point is defined as all points in the area with point p as the center and Eps as the radius, which is specifically expressed as: kN(p)={q∈D|dist(p,q)≤Eps} in: D is the data set; dist(p,q) is the distance between point p and point q; Eps is the given neighborhood radius.
Citation Information
Patent Citations
Large-scale data abnormity recognition method based on bidirectional sampling combination
CN104462184A
Visual big data machine learning system and method based on Ray and Spark
CN117171471A
