Internet data management system

Through distributed data acquisition, cloud computing storage, parallel processing and multi-level security measures, the efficiency and security problems of the Internet data management system under high concurrency and data volume growth are solved, and efficient, stable and secure data management is achieved.

CN118945163AInactive Publication Date: 2024-11-12SHANGHAI LIGONG NETWORK TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410985160.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing Internet data management system has problems such as inefficiency, poor scalability and insufficient data security when processing large-scale data. Especially in high concurrency scenarios, it is difficult to adapt to the rapid growth of data volume, and insufficient data security, which is easy to Data breach or corruption caused by external attacks.

Method used

Adopt distributed data acquisition module, cloud computing-based distributed database storage module, parallel data processing module, resource scheduling module and encryption protection module, and through distributed networks, distributed databases, parallel computing framework, real-time resource monitoring and multi-level security Measures to ensure efficient processing and security of the system under high concurrency and data volume growth.

Benefits of technology

It significantly improves data processing speed and system stability, improves system scalability and data security, and ensures efficient processing and data security in high concurrency scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945163A_ABST
    Figure CN118945163A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of Internet, and provides an Internet data management system, which comprises a data acquisition module, a data processing module and a data management module, the storage module is used for efficiently storing and quickly accessing data by utilizing a distributed database technology based on cloud computing; the data processing module performs distributed processing on the data, and constructs a parallel computing framework by performing parallel processing on the data on a plurality of nodes; the resource scheduling module is used for monitoring the data flow of the system in real time, setting a threshold value used by each resource, and automatically adjusting resource allocation by a set scheduling algorithm according to the monitoring data and the threshold value; the protection module encrypts stored and transmitted data, sets that only an authorized user can access sensitive data, monitors network flow and system activity, and performs intrusion detection; according to the method, the processing speed and the system stability are remarkably improved while the data processing precision is kept, and the method is particularly excellent in performance in a high-concurrency scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of Internet, and specifically relates to an Internet data management system. Background Art

[0002] With the rapid development of Internet technology, the explosive growth of data volume has made data management more and more complicated. Existing Internet data management systems have problems such as low efficiency, poor scalability, and insufficient data security when processing large-scale data.

[0003] The existing Internet data management systems have the following main shortcomings: 1) Performance degrades in high-concurrency scenarios; 2) The system has poor scalability and is difficult to adapt to the rapid growth of data volume; 3) Data security is insufficient and is prone to external attacks, resulting in data leakage or damage.

[0004] To this end, those skilled in the art have proposed an Internet data management system that can effectively handle high-concurrency requests, has good scalability to adapt to the growing amount of data, and ensures data security. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an Internet data management system to solve the problems raised in the background technology.

[0006] An Internet data management system, comprising:

[0007] The data acquisition module adopts a distributed approach and uses multiple acquisition nodes to work in parallel;

[0008] The storage module is based on cloud computing and uses distributed database technology to store data efficiently and access it quickly;

[0009] The data processing module distributes data and builds a parallel computing framework by processing data in parallel on multiple nodes;

[0010] The resource scheduling module is used to monitor the data flow of the system in real time, set the threshold of each resource usage, and automatically adjust the resource allocation according to the monitoring data and threshold by the set scheduling algorithm;

[0011] The protection module encrypts stored and transmitted data, limits access to sensitive data to authorized users only, and monitors network traffic and system activities for intrusion detection.

[0012] Preferably, the data acquisition module is based on Ethernet, and a distributed network is established between the ground acquisition station and the central control station. Each acquisition station is connected through the network, and each acquisition node is set to be able to independently process local data. Data is transmitted between nodes through a security protocol, and a distributed recording method is adopted. When sending data to RTI, the data is first recorded through a recording interface and then forwarded.

[0013] Preferably, the storage module maps data to different storage nodes through a consistent hashing algorithm, and minimizes data migration when dynamically adding or reducing nodes by performing a ring mapping of the hash values ​​of the data and the nodes; and each data item has multiple copies stored on different nodes. When the performance of a single node is lower than a preset threshold, the system automatically increases the number of nodes to share the load and redirects requests to other nodes.

[0014] Preferably, the data processing module first cleans and formats the raw water data, and then performs MapReduce job configuration, splits the input data into multiple data blocks, each data block is processed by a Map task, each Map task generates key-value pairs and sorts them by key; groups the output of the Map stage by key, and transfers the values ​​with the same key to the same Reduce task, and then processes the value set of each key to generate the final result; submits the job to the cluster for execution, monitors the job status and performance, and collects the processing results from HDFS.

[0015] Preferably, the data traffic monitored in the resource scheduling module includes network bandwidth usage, CPU usage, and memory usage; the set scheduling algorithm is expressed as:

[0016]

[0017] Where R represents the resource, U(R) represents the utilization rate of resource R, and T high , T low They represent the upper and lower thresholds of resource usage respectively. Based on the comparison between resource usage and preset thresholds, a decision is made whether to adjust resources, and the system resources are adjusted by automated tools.

[0018] Preferably, the protection module uses AES to encrypt data, and uses SSL / TLS security protocol to encrypt data during data transmission; at the same time, a user identity authentication mechanism is implemented, and the principle of minimization of permissions is maintained, granting the lowest permissions; IDS and IPS are deployed, network traffic monitoring tools are configured, and automated tools are used to detect abnormal behavior.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] While maintaining data processing accuracy, the present invention significantly improves processing speed and system stability, especially in high-concurrency scenarios. In addition, the scalability and data security of the system are also greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a block diagram of the Internet data management system of the present invention. DETAILED DESCRIPTION

[0022] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0023] As attached Figure 1 As shown:

[0024] Embodiment 1: The present invention provides an Internet data management system, comprising:

[0025] The data acquisition module adopts a distributed approach and uses multiple acquisition nodes to work in parallel;

[0026] The storage module is based on cloud computing and uses distributed database technology to store data efficiently and access it quickly;

[0027] The data processing module distributes data and builds a parallel computing framework by processing data in parallel on multiple nodes;

[0028] The resource scheduling module is used to monitor the data flow of the system in real time, set the threshold of each resource usage, and automatically adjust the resource allocation according to the monitoring data and threshold by the set scheduling algorithm;

[0029] The protection module encrypts stored and transmitted data, limits access to sensitive data to authorized users only, and monitors network traffic and system activities for intrusion detection.

[0030] The data acquisition module is based on Ethernet. A distributed network is established between the ground acquisition station and the central control station. Each acquisition station is connected through the network. Each acquisition node is set to independently process local data. Data is transmitted between nodes through a security protocol. A distributed recording method is adopted. When sending data to RTI, the data is first recorded through the recording interface and then forwarded. By establishing an efficient distributed network, designing acquisition nodes with independent processing capabilities, and realizing optimized data processing and transmission mechanisms, the efficiency and reliability of data acquisition are effectively improved.

[0031] The storage module maps data to different storage nodes through a consistent hashing algorithm to ensure that each node stores data evenly and avoids single-point performance bottlenecks. By performing a circular mapping of the hash values ​​of data and nodes, data migration is minimized when dynamically adding or reducing nodes, ensuring that only a small portion of the data needs to be relocated when the number of nodes in the cluster changes, thereby maintaining the high availability and scalability of the system. In addition, each data item has multiple copies stored on different nodes. When the performance of a single node is lower than the preset threshold, the system automatically increases the number of nodes to share the load and redirects requests to other nodes.

[0032] The data processing module first cleans and formats the raw water data, and then configures the MapReduce job to split the input data into multiple data blocks. Each data block is processed by a Map task. Each Map task generates key-value pairs and sorts them by key. The output of the Map stage is grouped by key, and the values ​​with the same key are transferred to the same Reduce task. Then, the value set of each key is processed to generate the final result. The job is submitted to the cluster for execution, the job status and performance are monitored, and the processing results are collected from HDFS. Through the above processing method, large-scale data sets can be effectively processed to improve the throughput and scalability of the system.

[0033] The data traffic monitored in the resource scheduling module includes network bandwidth usage, CPU usage, and memory usage; the set scheduling algorithm is expressed as:

[0034]

[0035] Where R represents the resource, U(R) represents the utilization rate of resource R, and T high 、T low They represent the upper and lower thresholds of resource usage respectively; based on the comparison between resource usage and preset thresholds, it is decided whether to adjust resources, and the system resources are adjusted by automated tools. Through real-time monitoring and intelligent adjustment of system resources, the stable operation of the system under high load is effectively guaranteed. Through reasonable scheduling algorithms and parameter settings, the optimal allocation of resources can be achieved, and the overall performance and availability of the system can be improved.

[0036] AES is used in the protection module to encrypt data, and SSL / TLS security protocols are used to encrypt data during data transmission. At the same time, a user authentication mechanism is implemented, and the principle of minimization of permissions is maintained, granting the lowest permissions. IDS and IPS are deployed, network traffic monitoring tools are configured, and automated tools are used to detect abnormal behavior. By implementing multi-level security protection measures, data security can be effectively protected, data leakage and unauthorized access can be prevented, and the overall security of the system can be improved together.

[0037] Additionally, in order to provide a concise description of exemplary embodiments, all features of an actual embodiment (ie, those features that are not relevant to the best mode presently contemplated for carrying out the invention or those that are not relevant to implementing the invention) may not be described.

[0038] It will be appreciated that in the development of any actual implementation, as in any engineering or design project, numerous implementation-specific decisions may be made. Such a development effort may be complex and time-consuming, but will be a routine task of design, fabrication, and production for those of ordinary skill having the benefit of this disclosure without undue experimentation.

[0039] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An Internet data management system, characterized in that: include: The data acquisition module adopts a distributed approach and uses multiple acquisition nodes to work in parallel; The storage module is based on cloud computing and uses distributed database technology to store data efficiently and access it quickly; The data processing module distributes data and builds a parallel computing framework by processing data in parallel on multiple nodes; The resource scheduling module is used to monitor the data flow of the system in real time, set the threshold of each resource usage, and automatically adjust the resource allocation according to the monitoring data and threshold by the set scheduling algorithm; The protection module encrypts stored and transmitted data, limits access to sensitive data to authorized users only, and monitors network traffic and system activities for intrusion detection.

2. An Internet data management system as claimed in claim 1, characterized in that: The data acquisition module is based on Ethernet. A distributed network is established between the ground acquisition station and the central control station. Each acquisition station is connected through the network. Each acquisition node is set to be able to independently process local data. Data is transmitted between nodes through a security protocol and a distributed recording method is adopted. When sending data to the RTI, the data is first recorded through the recording interface and then forwarded.

3. An Internet data management system as claimed in claim 1, characterized in that: The storage module maps data to different storage nodes through a consistent hashing algorithm, and minimizes data migration when dynamically adding or reducing nodes by performing a circular mapping of the hash values ​​of the data and the nodes; and each data item has multiple copies stored on different nodes. When the performance of a single node is lower than a preset threshold, the system automatically increases the number of nodes to share the load and redirects requests to other nodes.

4. An Internet data management system as claimed in claim 1, characterized in that: The data processing module first cleans and formats the raw water data, and then configures the MapReduce job to split the input data into multiple data blocks. Each data block is processed by a Map task. Each Map task generates key-value pairs and sorts them by key; the output of the Map stage is grouped by key, and the values ​​with the same key are transmitted to the same Reduce task, and then the value set of each key is processed to generate the final result; the job is submitted to the cluster for execution, the job status and performance are monitored, and the processing results are collected from HDFS.

5. An Internet data management system as claimed in claim 1, characterized in that: The data traffic monitored in the resource scheduling module includes network bandwidth usage, CPU usage, and memory usage; the set scheduling algorithm is expressed as: Where R represents the resource, U(R) represents the utilization rate of resource R, and T high 、T low They represent the upper and lower thresholds of resource usage respectively. Based on the comparison between resource usage and preset thresholds, a decision is made whether to adjust resources, and the system resources are adjusted by automated tools.

6. An Internet data management system as claimed in claim 1, characterized in that: The protection module uses AES to encrypt data, and uses SSL / TLS security protocol to encrypt data during data transmission; at the same time, a user identity authentication mechanism is implemented, and the principle of minimization of permissions is maintained, granting the lowest permissions; IDS and IPS are deployed, network traffic monitoring tools are configured, and automated tools are used to detect abnormal behavior.