Information processing system based on big data

Abstract

The invention discloses an information processing system based on big data, and relates to the technical field of data processing. A real-time data processing engine is used for processing and analyzing real-time data streams and realizing rapid decision and response; comprising a distributed computing framework, a data acquisition module, a data preprocessing module, a memory computing module, a load balancing module, a data storage module and a data display module, the self-adaptive machine learning algorithm unit is used for automatically adjusting and optimizing an analysis model to improve the accuracy and efficiency of data processing; comprising the steps of data feature selection, data model establishment and evaluation, online learning and model optimization. The multi-layer data security protection unit is used for protecting the security of the data in the transmission and storage processes; the dynamic resource management unit is used for dynamically adjusting resource allocation according to the system load and the resource use condition and improving the resource use efficiency; and the anomaly detection and early warning unit is used for monitoring the running state of the system in real time and finding and responding to the abnormal condition in time.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an information processing system based on big data. Background Art

[0002] With the continuous development of science and technology, data has shown explosive growth, and the effective processing of big data has become increasingly important. At present, big data technology is still in a stage of dynamic change and rapid progress, requiring continuous technological innovation and system optimization to meet the growing data demand and processing efficiency.

[0003] However, the existing technology has low efficiency in data processing and is difficult to process large amounts of data in real time. At the same time, data analysis is not accurate enough, and data processing is vulnerable to external attacks and internal leaks. It lacks adaptive adjustment capabilities and is difficult to cope with changes in data patterns. Summary of the invention

[0004] In view of the above technical problems, the present invention proposes an information processing system based on big data, comprising: Real-time data processing engine: used to process and analyze real-time data streams to achieve rapid decision-making and response; including distributed computing framework, data acquisition module, data preprocessing module, memory computing module, load balancing module, data storage module, and data display module; Adaptive machine learning algorithm unit: used to automatically adjust and optimize analytical models to improve the accuracy and efficiency of data processing; including data feature selection, data model establishment and evaluation, online learning and model optimization; Multi-layer data security protection unit: used to protect the security of data during transmission and storage, and prevent data leakage and unauthorized access; including data encryption protection, access control, and intrusion detection; Dynamic resource management unit: used to dynamically adjust resource allocation according to system load and resource usage to improve resource utilization efficiency; including resource monitoring module, automatic expansion module, and resource scheduling module; Abnormal detection and early warning unit: used to monitor the system operation status in real time, and promptly detect and respond to abnormal situations; including real-time monitoring, abnormal monitoring, and automatic early warning.

[0005] Furthermore, the data is processed by the following process: S1. Data collection: using the data collection module to collect user data in real time, and storing the collected data in a database; S2, data preprocessing: using the data preprocessing module to clean, convert and standardize the data collected in S1 to improve data quality; S3, data feature processing: feature selection is performed on the preprocessed data in S2 to extract features that represent the essence of the data; S4, model training and evaluation: S41, model training: divide the data in S3 into training set and test set; use random forest algorithm, set the number of trees and corresponding parameters; train the model and adjust the parameters to optimize performance; S42, use K-fold cross validation method to evaluate the stability and accuracy of the model; calculate the performance index on each fold; calculate the average performance index and evaluate the overall performance of the model; S5. Model optimization: Based on the continuous update of new data, the model is optimized using the online learning and model optimization module to improve the accuracy and timeliness of the machine learning model; S6, data storage: storing the processed data in S3 in the data storage module according to feature classification, so as to facilitate subsequent query, call and analysis; S7, data display: using the data display module to display the analysis results in the form of charts or dashboards; S8, security monitoring: in the data processing flow in S1-S7, the multi-layer data security protection unit is used to ensure the security of the data; S9, exception handling: Use the anomaly detection and early warning system to monitor the data processing flow in S1-S7 in real time. Once an anomaly is detected, an early warning is immediately triggered and an abnormal notification is sent; S10. Feedback and optimization: Continuously adjust and optimize data processing processes and algorithms based on data display results and user feedback.

[0006] Furthermore, the data feature processing includes: S31. Data feature exploration: Before extracting features, explore and analyze the data to find out the data distribution, data type, missing values ​​and outliers; S32, feature extraction: use principal component analysis to calculate the data covariance moment, and calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and select the eigenvector as a new feature based on the eigenvalue; S33. Feature selection: Use recursive feature elimination to train a base model, score the base model weights, recursively remove small weight features, retrain the model and evaluate performance, and repeat this process until the required number of features is reached.

[0007] Furthermore, the distributed computing framework distributes data and computing tasks to multiple nodes for parallel processing, thereby improving data processing speed and scalability; the memory computing module utilizes data storage and computing in memory to reduce disk I / O operations and improve data processing speed; the load balancing module dynamically allocates tasks according to the load conditions of each node to prevent node overload and ensure stable system operation.

[0008] Furthermore, when performing data encryption: use strong encryption algorithms to encrypt data to ensure that the data cannot be interpreted even if it is intercepted; when performing access control: through identity authentication and permission management, ensure that only authorized users can access sensitive data; when performing intrusion detection: monitor network and system activities in real time, detect and respond to potential security threats.

[0009] Furthermore, the resource monitoring module is used to monitor the usage of system resources in real time; the automatic expansion module automatically increases or decreases resources according to the resource usage; and the resource scheduling module is used to intelligently schedule tasks to appropriate computing nodes to optimize resource usage.

[0010] Furthermore, the real-time monitoring module is used to collect and analyze system operation data in real time. When an abnormality is detected in S9, a notification is sent via email or text message to facilitate timely processing.

[0011] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention realizes rapid processing of big data through a real-time data processing engine, thereby solving the problem of low data processing efficiency; (2) The present invention can more accurately identify data patterns and trends through an adaptive machine learning algorithm unit, thereby improving the accuracy of data analysis; (3) The present invention effectively prevents data leakage and attacks through a multi-layer data security protection unit, thereby enhancing data security; (4) The present invention reduces resource waste and improves resource utilization efficiency through a dynamic resource management unit; (5) The present invention ensures stable operation of the system through an abnormality detection and early warning unit, thereby improving the reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The data processing flow chart of the present invention. DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with specific embodiments. The illustrative embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0014] Example: Figure 1 An information processing system based on big data is shown, comprising: Real-time data processing engine: used to process and analyze real-time data streams to achieve rapid decision-making and response; including distributed computing framework, data acquisition module, data preprocessing module, memory computing module, load balancing module, data storage module, and data display module; Adaptive machine learning algorithm unit: used to automatically adjust and optimize analytical models to improve the accuracy and efficiency of data processing; including data feature selection, data model establishment and evaluation, online learning and model optimization; Multi-layer data security protection unit: used to protect the security of data during transmission and storage, and prevent data leakage and unauthorized access; including data encryption protection, access control, and intrusion detection; Dynamic resource management unit: used to dynamically adjust resource allocation according to system load and resource usage to improve resource utilization efficiency; including resource monitoring module, automatic expansion module, and resource scheduling module; Abnormal detection and early warning unit: used to monitor the system operation status in real time, and promptly detect and respond to abnormal situations; including real-time monitoring, abnormal monitoring, and automatic early warning.

[0015] The following processes are involved in data processing: S1. Data collection: Use the data collection module to collect user data in real time and store the collected data in the database; S2, data preprocessing: Use the data preprocessing module to clean, convert and standardize the data collected in S1 to improve data quality; S3, data feature processing: feature selection is performed on the preprocessed data in S2 to extract features that represent the essence of the data; S4, model training and evaluation: S41, model training: divide the data in S3 into training set and test set; use random forest algorithm, set the number of trees and corresponding parameters; train the model and adjust the parameters to optimize performance; S42, use K-fold cross validation method to evaluate the stability and accuracy of the model; calculate the performance index on each fold; calculate the average performance index and evaluate the overall performance of the model; S5. Model optimization: Based on the continuous update of new data, the model is optimized using online learning and model optimization modules to improve the accuracy and timeliness of the machine learning model; S6, data storage: the processed data in S3 is stored in the data storage module according to feature classification to facilitate subsequent query, call and analysis; S7. Data display: Use the data display module to display the analysis results in the form of charts or dashboards; S8, security monitoring: In the data processing flow from S1 to S7, multi-layer data security protection units are used to ensure data security; S9, exception handling: Use the anomaly detection and early warning system to monitor the data processing flow in S1-S7 in real time. Once an anomaly is detected, an early warning is immediately triggered and an abnormal notification is sent; S10. Feedback and optimization: Continuously adjust and optimize data processing processes and algorithms based on data display results and user feedback.

[0016] Data feature processing includes: S31. Data feature exploration: Before extracting features, explore and analyze the data to find out the data distribution, data type, missing values ​​and outliers; S32, feature extraction: use principal component analysis to calculate the data covariance moment, and calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and select the eigenvector as a new feature based on the eigenvalue; S33. Feature selection: Use recursive feature elimination to train a base model, score the base model weights, recursively remove small weight features, retrain the model and evaluate performance, and repeat this process until the required number of features is reached.

[0017] The distributed computing framework distributes data and computing tasks to multiple nodes for parallel processing, improving data processing speed and scalability; the memory computing module uses data storage and computing in memory to reduce disk I / O operations and improve data processing speed; the load balancing module dynamically allocates tasks according to the load conditions of each node to prevent node overload and ensure stable system operation.

[0018] When performing data encryption: use strong encryption algorithms to encrypt data to ensure that the data cannot be interpreted even if it is intercepted; when performing access control: through identity authentication and permission management, ensure that only authorized users can access sensitive data; when performing intrusion detection: monitor network and system activities in real time, detect and respond to potential security threats.

[0019] The resource monitoring module is used to monitor the usage of system resources in real time. The automatic expansion module automatically increases or decreases resources according to resource usage. The resource scheduling module is used to intelligently schedule tasks to appropriate computing nodes to optimize resource usage.

[0020] The real-time monitoring module is used to collect and analyze system operation data in real time. When an abnormality is detected in S9, a notification is sent via email or SMS to facilitate timely processing.

Claims

1. An information processing system based on big data, characterized in that: include: Real-time data processing engine: used to process and analyze real-time data streams to achieve rapid decision-making and response; including distributed computing framework, data acquisition module, data preprocessing module, memory computing module, load balancing module, data storage module, and data display module; Adaptive machine learning algorithm unit: used to automatically adjust and optimize the analysis model to improve the accuracy and efficiency of data processing; Including data feature selection, data model building and evaluation, online learning and model optimization; Multi-layer data security protection unit: used to protect the security of data during transmission and storage, and prevent data leakage and unauthorized access; Including data encryption protection, access control, and intrusion detection; Dynamic resource management unit: used to dynamically adjust resource allocation according to system load and resource usage to improve resource utilization efficiency; Including resource monitoring module, automatic expansion module, and resource scheduling module; Abnormal detection and early warning unit: used to monitor the system operation status in real time, and promptly detect and respond to abnormal situations; Including real-time monitoring, abnormality monitoring and automatic early warning.

2. The information processing system based on big data according to claim 1, characterized in that: The following processes are involved in data processing: S1. Data collection: using the data collection module to collect user data in real time, and storing the collected data in a database; S2, data preprocessing: using the data preprocessing module to clean, convert and standardize the data collected in S1 to improve data quality; S3, data feature processing: feature selection is performed on the preprocessed data in S2 to extract features that represent the essence of the data; S4, model training and evaluation: S41, model training: divide the data in S3 into training set and test set; use the random forest algorithm, set the number of trees and corresponding parameters; train the model and adjust the parameters to optimize performance; S42, use K-fold cross validation method to evaluate the stability and accuracy of the model; Calculate the performance index on each fold; calculate the average performance index to evaluate the overall performance of the model; S5. Model optimization: Based on the continuous update of new data, the model is optimized using the online learning and model optimization module to improve the accuracy and timeliness of the machine learning model; S6, data storage: storing the processed data in S3 in the data storage module according to feature classification, so as to facilitate subsequent query, call and analysis; S7, data display: using the data display module to display the analysis results in the form of charts or dashboards; S8, security monitoring: in the data processing flow in S1-S7, the multi-layer data security protection unit is used to ensure the security of the data; S9, exception handling: Use the anomaly detection and early warning system to monitor the data processing flow in S1-S7 in real time. Once an anomaly is detected, an early warning is immediately triggered and an abnormal notification is sent; S10. Feedback and optimization: Continuously adjust and optimize data processing processes and algorithms based on data display results and user feedback.

3. The information processing system based on big data as claimed in claim 2, characterized in that: The data feature processing includes: S31. Data feature exploration: Before extracting features, explore and analyze the data to find out the data distribution, data type, missing values ​​and outliers; S32, feature extraction: use principal component analysis to calculate the data covariance moment, and calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and select the eigenvector as a new feature based on the eigenvalue; S33. Feature selection: Use recursive feature elimination to train a base model, score the base model weights, recursively remove small weight features, retrain the model and evaluate performance, and repeat this process until the required number of features is reached.

4. The information processing system based on big data as claimed in claim 2, characterized in that: The distributed computing framework distributes data and computing tasks to multiple nodes for parallel processing, thereby improving data processing speed and scalability; the memory computing module utilizes data storage and computing in memory to reduce disk I / O operations and improve data processing speed; the load balancing module dynamically distributes tasks according to the load conditions of each node, prevents node overload, and ensures stable operation of the system.

5. The information processing system based on big data as claimed in claim 2, characterized in that: When performing data encryption: use strong encryption algorithms to encrypt data to ensure that the data cannot be interpreted even if it is intercepted; when performing access control: through identity authentication and permission management, ensure that only authorized users can access sensitive data; when performing intrusion detection: monitor network and system activities in real time, detect and respond to potential security threats.

6. The information processing system based on big data as claimed in claim 2, characterized in that: The resource monitoring module is used to monitor the usage of system resources in real time; the automatic expansion module automatically increases or decreases resources according to the resource usage; and the resource scheduling module is used to intelligently schedule tasks to appropriate computing nodes to optimize resource usage.

7. The information processing system based on big data as claimed in claim 2, characterized in that: The real-time monitoring module is used to collect and analyze system operation data in real time. When an abnormality is detected in S9, a notification is sent via email or text message to facilitate timely processing.

Images