Internet data management system
Through distributed data acquisition, cloud computing storage, parallel processing and multi-level security measures, the efficiency and security problems of the Internet data management system under high concurrency and data volume growth are solved, and efficient, stable and secure data management is achieved.
Patent Information
- Application Number
- CN202410985160.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing Internet data management system has problems such as inefficiency, poor scalability and insufficient data security when processing large-scale data. Especially in high concurrency scenarios, it is difficult to adapt to the rapid growth of data volume, and insufficient data security, which is easy to Data breach or corruption caused by external attacks.
Adopt distributed data acquisition module, cloud computing-based distributed database storage module, parallel data processing module, resource scheduling module and encryption protection module, and through distributed networks, distributed databases, parallel computing framework, real-time resource monitoring and multi-level security Measures to ensure efficient processing and security of the system under high concurrency and data volume growth.
It significantly improves data processing speed and system stability, improves system scalability and data security, and ensures efficient processing and data security in high concurrency scenarios.
Smart Images

Figure CN118945163A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of Internet, and specifically relates to an Internet data management system. Background Art
[0002] With the rapid development of Internet technology, the explosive growth of data volume has made data management more and more complicated. Existing Internet data management systems have problems such as low efficiency, poor scalability, and insufficient data security when processing large-scale data.
[0003] The existing Internet data management systems have the following main shortcomings: 1) Performance degrades in high-concurrency scenarios; 2) The system has poor scalability and is difficult to adapt to the rapid growth of data volume; 3) Data security is insufficient and is prone to external attacks, resulting in data leakage or damage.
[0004] To this end, those skilled in the art have proposed an Internet data management system that can effectively handle high-concurrency requests, has good scalability to adapt to the growing amount of data, and ensures data security. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides an Internet data management system to solve the problems raised in the background technology.
[0006] An Internet data management system, comprising:
[0007] The data acquisition module adopts a distributed approach and uses multiple acquisition nodes to work in parallel;
[0008] The storage module is based on cloud computing and uses distributed database technology to store data efficiently and access it quickly;
[0009] The data processing module distributes data and builds a parallel computing framework by processing data in parallel on multiple nodes;
[0010] The resource scheduling module is used to monitor the data flow of the system in real time, set the threshold of each resource usage, and automatically adjust the resource allocation according to the monitoring data and threshold by the set scheduling algorithm;
[0011] The protection module encrypts stored and transmitted data, limits access to sensitive data to authorized users only, and monitors network traffic and system activities for intrusion detection.
[0012] Preferably, the data acquisition module is based on Ethernet, and a distributed network is established between the ground acquisition station and the central control station. Each acquisition station is connected through the network, and each acquisition node is set to be able to independently process local data. Data is transmitted between nodes through a security protocol, and a distributed recording method is adopted. When sending data to RTI, the data is first recorded through a recording interface and then forwarded.
[0013] Preferably, the storage module maps data to different storage nodes through a consistent hashing algorithm, and minimizes data migration when dynamically adding or reducing nodes by performing a ring mapping of the hash values of the data and the nodes; and each data item has multiple copies stored on different nodes. When the performance of a single node is lower than a preset threshold, the system automatically increases the number of nodes to share the load and redirects requests to other nodes.
[0014] Preferably, the data processing module first cleans and formats the raw water data, and then performs MapReduce job configuration, splits the input data into multiple data blocks, each data block is processed by a Map task, each Map task generates key-value pairs and sorts them by key; groups the output of the Map stage by key, and transfers the values with the same key to the same Reduce task, and then processes the value set of each key to generate the final result; submits the job to the cluster for execution, monitors the job status and performance, and collects the processing results from HDFS.
[0015] Preferably, the data traffic monitored in the resource scheduling module includes network bandwidth usage, CPU usage, and memory usage; the set scheduling algorithm is expressed as:
[0016]
[0017] Where R represents the resource, U(R) represents the utilization rate of resource R, and T high , T low They represent the upper and lower thresholds of resource usage respectively. Based on the comparison between resource usage and preset thresholds, a decision is made whether to adjust resources, and the system resources are adjusted by automated tools.
[0018] Preferably, the protection module uses AES to encrypt data, and uses SSL / TLS security protocol to encrypt data during data transmission; at the same time, a user identity authentication mechanism is implemented, and the principle of minimization of permissions is maintained, granting the lowest permissions; IDS and IPS are deployed, network traffic monitoring tools are configured, and automated tools are used to detect abnormal behavior.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] While maintaining data processing accuracy, the present invention significantly improves processing speed and system stability, especially in high-concurrency scenarios. In addition, the scalability and data security of the system are also greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 This is a block diagram of the Internet data management system of the present invention. DETAILED DESCRIPTION
[0022] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0023] As attached Figure 1 As shown:
[0024] Embodiment 1: The present invention provides an Internet data management system, comprising:
[0025] The data acquisition module adopts a distributed approach and uses multiple acquisition nodes to work in parallel;
[0026] The storage module is based on cloud computing and uses distributed database technology to store data efficiently and access it quickly;
[0027] The data processing module distributes data and builds a parallel computing framework by processing data in parallel on multiple nodes;
[0028] The resource scheduling module is used to monitor the data flow of the system in real time, set the threshold of each resource usage, and automatically adjust the resource allocation according to the monitoring data and threshold by the set scheduling algorithm;
[0029] The protection module encrypts stored and transmitted data, limits access to sensitive data to authorized users only, and monitors network traffic and system activities for intrusion detection.
[0030] The data acquisition module is based on Ethernet. A distributed network is established between the ground acquisition station and the central control station. Each acquisition station is connected through the network. Each acquisition node is set to independently process local data. Data is transmitted between nodes through a security protocol. A distributed recording method is adopted. When sending data to RTI, the data is first recorded through the recording interface and then forwarded. By establishing an efficient distributed network, designing acquisition nodes with independent processing capabilities, and realizing optimized data processing and transmission mechanisms, the efficiency and reliability of data acquisition are effectively improved.
[0031] The storage module maps data to different storage nodes through a consistent hashing algorithm to ensure that each node stores data evenly and avoids single-point performance bottlenecks. By performing a circular mapping of the hash values of data and nodes, data migration is minimized when dynamically adding or reducing nodes, ensuring that only a small portion of the data needs to be relocated when the number of nodes in the cluster changes, thereby maintaining the high availability and scalability of the system. In addition, each data item has multiple copies stored on different nodes. When the performance of a single node is lower than the preset threshold, the system automatically increases the number of nodes to share the load and redirects requests to other nodes.
[0032] The data processing module first cleans and formats the raw water data, and then configures the MapReduce job to split the input data into multiple data blocks. Each data block is processed by a Map task. Each Map task generates key-value pairs and sorts them by key. The output of the Map stage is grouped by key, and the values with the same key are transferred to the same Reduce task. Then, the value set of each key is processed to generate the final result. The job is submitted to the cluster for execution, the job status and performance are monitored, and the processing results are collected from HDFS. Through the above processing method, large-scale data sets can be effectively processed to improve the throughput and scalability of the system.
[0033] The data traffic monitored in the resource scheduling module includes network bandwidth usage, CPU usage, and memory usage; the set scheduling algorithm is expressed as:
[0034]
[0035] Where R represents the resource, U(R) represents the utilization rate of resource R, and T high 、T low They represent the upper and lower thresholds of resource usage respectively; based on the comparison between resource usage and preset thresholds, it is decided whether to adjust resources, and the system resources are adjusted by automated tools. Through real-time monitoring and intelligent adjustment of system resources, the stable operation of the system under high load is effectively guaranteed. Through reasonable scheduling algorithms and parameter settings, the optimal allocation of resources can be achieved, and the overall performance and availability of the system can be improved.
[0036] AES is used in the protection module to encrypt data, and SSL / TLS security protocols are used to encrypt data during data transmission. At the same time, a user authentication mechanism is implemented, and the principle of minimization of permissions is maintained, granting the lowest permissions. IDS and IPS are deployed, network traffic monitoring tools are configured, and automated tools are used to detect abnormal behavior. By implementing multi-level security protection measures, data security can be effectively protected, data leakage and unauthorized access can be prevented, and the overall security of the system can be improved together.
[0037] Additionally, in order to provide a concise description of exemplary embodiments, all features of an actual embodiment (ie, those features that are not relevant to the best mode presently contemplated for carrying out the invention or those that are not relevant to implementing the invention) may not be described.
[0038] It will be appreciated that in the development of any actual implementation, as in any engineering or design project, numerous implementation-specific decisions may be made. Such a development effort may be complex and time-consuming, but will be a routine task of design, fabrication, and production for those of ordinary skill having the benefit of this disclosure without undue experimentation.
[0039] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. An Internet data management system, characterized in that: include: The data acquisition module adopts a distributed approach and uses multiple acquisition nodes to work in parallel; The storage module is based on cloud computing and uses distributed database technology to store data efficiently and access it quickly; The data processing module distributes data and builds a parallel computing framework by processing data in parallel on multiple nodes; The resource scheduling module is used to monitor the data flow of the system in real time, set the threshold of each resource usage, and automatically adjust the resource allocation according to the monitoring data and threshold by the set scheduling algorithm; The protection module encrypts stored and transmitted data, limits access to sensitive data to authorized users only, and monitors network traffic and system activities for intrusion detection.
2. An Internet data management system as claimed in claim 1, characterized in that: The data acquisition module is based on Ethernet. A distributed network is established between the ground acquisition station and the central control station. Each acquisition station is connected through the network. Each acquisition node is set to be able to independently process local data. Data is transmitted between nodes through a security protocol and a distributed recording method is adopted. When sending data to the RTI, the data is first recorded through the recording interface and then forwarded.
3. An Internet data management system as claimed in claim 1, characterized in that: The storage module maps data to different storage nodes through a consistent hashing algorithm, and minimizes data migration when dynamically adding or reducing nodes by performing a circular mapping of the hash values of the data and the nodes; and each data item has multiple copies stored on different nodes. When the performance of a single node is lower than a preset threshold, the system automatically increases the number of nodes to share the load and redirects requests to other nodes.
4. An Internet data management system as claimed in claim 1, characterized in that: The data processing module first cleans and formats the raw water data, and then configures the MapReduce job to split the input data into multiple data blocks. Each data block is processed by a Map task. Each Map task generates key-value pairs and sorts them by key; the output of the Map stage is grouped by key, and the values with the same key are transmitted to the same Reduce task, and then the value set of each key is processed to generate the final result; the job is submitted to the cluster for execution, the job status and performance are monitored, and the processing results are collected from HDFS.
5. An Internet data management system as claimed in claim 1, characterized in that: The data traffic monitored in the resource scheduling module includes network bandwidth usage, CPU usage, and memory usage; the set scheduling algorithm is expressed as: Where R represents the resource, U(R) represents the utilization rate of resource R, and T high 、T low They represent the upper and lower thresholds of resource usage respectively. Based on the comparison between resource usage and preset thresholds, a decision is made whether to adjust resources, and the system resources are adjusted by automated tools.
6. An Internet data management system as claimed in claim 1, characterized in that: The protection module uses AES to encrypt data, and uses SSL / TLS security protocol to encrypt data during data transmission; at the same time, a user identity authentication mechanism is implemented, and the principle of minimization of permissions is maintained, granting the lowest permissions; IDS and IPS are deployed, network traffic monitoring tools are configured, and automated tools are used to detect abnormal behavior.
