Network security server control cabinet based on big data analysis

Abstract

The invention discloses a network security server control cabinet based on big data analysis, and relates to the technical field of network security management, and the main points of the technical scheme are that the cabinet comprises a data collection module which is used for collecting data from a plurality of network devices in real time, the data comprises network traffic, access logs and device states, a distributed structure is adopted for data collection, and the data collection module is used for collecting data; the collected data is uniformly converted into a JSON format and is safely transmitted through a TLS / SSL encryption technology; the data storage module is designed to have high expandability and reliability, can support rapid reading and writing of large-scale data, and realizes efficient management and cost optimization of the data by optimizing a storage strategy; the system has the advantages that by integrating the big data technology, machine learning and an automatic response mechanism, the efficiency and accuracy of network security monitoring are improved, the false report rate and the missing report rate are reduced, and therefore the network environment is more effectively protected against invasion of various known and unknown threats.

Description

Technical Field

[0001] The present invention relates to the technical field of network security management, and more specifically, to a network security server control cabinet based on big data analysis. Background Art

[0002] With the rapid development of information technology, network security issues have become increasingly prominent, posing a serious threat to the information assets of enterprises and organizations. In order to effectively deal with these threats, network security monitoring systems have become an indispensable tool. Such systems usually rely on the collection and analysis of data such as network traffic and system logs to detect and respond to potential security threats.

[0003] In the existing network security monitoring technology, a variety of methods have been widely used, such as intrusion detection system (IDS), security event management (SIEM) solutions, etc. These technologies usually collect information such as data packets, system logs, user behavior, etc. in the network, and use rule matching, statistical analysis and other methods to identify abnormal behaviors and security threats. However, these methods face many challenges when processing large-scale network data.

[0004] First, traditional network security monitoring methods are often limited by data processing capabilities and storage capacity when dealing with massive amounts of data. As the scale of networks expands and the amount of data surges, these methods find it difficult to analyze all data in real time, resulting in potential threats being overlooked.

[0005] Second, existing monitoring technologies usually rely on predefined rules or patterns to identify threats, which makes them limited in their ability to detect new or unknown threats. Attackers often take advantage of this and evade detection by constantly changing their attack methods.

[0006] In addition, many existing network security monitoring systems lack accuracy when generating alerts, resulting in a large number of false positives and false negatives, which not only increases the workload of security personnel, but also may cover up real security incidents.

[0007] Therefore, in order to solve the above technical problems, the present invention proposes a network security server control cabinet based on big data analysis. Summary of the invention

[0008] In view of the deficiencies in the prior art, an object of the present invention is to provide a network security server control cabinet based on big data analysis.

[0009] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a network security server control cabinet based on big data analysis, a data acquisition module, which is used to collect data from a variety of network devices in real time, including network traffic, access logs and device status, adopt a distributed structure for data acquisition, and uniformly convert the collected data into JSON format, and transmit it securely through TLS / SSL encryption technology;

[0010] The data storage module is designed to be highly scalable and reliable, capable of supporting fast reading and writing of large-scale data, and achieving efficient data management and cost optimization through optimized storage strategies;

[0011] Data analysis module, which uses machine learning and data mining techniques to extract valuable information from stored data and accurately identify abnormal behaviors and security threats in the network through data preprocessing, feature extraction and mining algorithms;

[0012] The control and response module automatically adjusts the server security policy based on the results of the data analysis module, including updating firewall rules and blocking suspicious connections, so as to quickly respond to known and unknown security threats;

[0013] The monitoring and management module provides administrators with an interface that displays the real-time security status of the network and server, and allows administrators to manually adjust security policies or configurations to enhance the flexibility and controllability of the system.

[0014] Furthermore, the data collection module is further configured to collect data from routers, switches, firewalls, and server logs.

[0015] Furthermore, the data analysis module further includes data cleaning, standardization, normalization processing and preprocessing steps of reducing data dimension using PCA.

[0016] Furthermore, the data storage module adopts a distributed storage system to ensure high availability and fault tolerance of data.

[0017] Furthermore, the data analysis module further utilizes deep learning algorithms to perform anomaly detection and behavior analysis to improve the ability to identify complex security threats.

[0018] Furthermore, the control and response module also includes a threat intelligence integration function, which can automatically obtain and update external threat intelligence information to enhance the defense capability against new security threats.

[0019] Furthermore, the monitoring and management module provides a visual security dashboard that displays key security indicators and real-time threat intelligence to help administrators fully understand the network security status. The monitoring and management module also provides alarm and notification functions to report security incidents and abnormal situations to administrators in a timely manner for rapid response and processing.

[0020] Furthermore, a network security management method of a network security server controller comprises the following steps:

[0021] a. Collect data from network devices using the data acquisition module;

[0022] b. Storing the collected data in the data storage module;

[0023] c. Analyze data through the data analysis module to identify abnormal behaviors and security threats in the network;

[0024] d. Based on the data analysis results, the control and response module automatically adjusts the server security policy to respond to security threats in a timely manner;

[0025] e. Use the monitoring and management module to monitor the security status of the network and server in real time, and allow administrators to manually adjust security policies or configurations to ensure network security.

[0026] Furthermore, in step c, the data analysis module also uses historical data to train and optimize the model to improve the ability to predict and identify future security threats.

[0027] Furthermore, in step d, the control and response module automatically selects appropriate response measures according to the severity and type of the security threat, including isolating the infected system, blocking malicious traffic, etc.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. In the present invention, data collection and preprocessing:

[0030] The present invention designs an efficient data acquisition module that can collect data from a variety of network devices in real time, including network traffic, access logs, and device status. These data are the basis of network security analysis and are crucial for timely discovery and response to security threats. The data acquisition module adopts a distributed structure to ensure that data can be collected stably and efficiently in a high-concurrency environment. At the same time, the module also has a data preprocessing function, which can clean, format, convert, and encrypt the collected data to ensure the accuracy and security of the data.

[0031] 2. In the present invention, big data storage and management

[0032] In response to the storage and management needs of large-scale network data, the present invention proposes a data storage module with high scalability and reliability. The module adopts distributed storage technology, can support fast reading and writing of large-scale data, and realizes efficient data management by optimizing storage strategy. In addition, the module also has data backup and recovery functions to ensure that data can be quickly restored in the event of a failure, ensuring the continuity of network security analysis.

[0033] 3. In the present invention, intelligent data analysis:

[0034] The present invention uses machine learning and data mining technology to design an intelligent data analysis module. The module can extract valuable information from massive data and accurately identify abnormal behaviors and security threats in the network. Through data preprocessing, feature extraction and mining algorithms, the module can automatically discover potential threats in the network and provide strong support for subsequent security policy adjustments. This intelligent data analysis technology greatly improves the efficiency and accuracy of network security management.

[0035] 4. In the present invention, automatic control and response:

[0036] The present invention proposes an automated control and response module that can automatically adjust server security policies based on data analysis results. The module responds to known and unknown security threats in a timely manner by updating firewall rules, blocking suspicious connections, and so on. At the same time, the module can also automatically select appropriate response measures based on the severity and type of security threats to minimize the impact of security threats on the network. This automated control and response mechanism greatly improves the protection capability and response speed of network security. DETAILED DESCRIPTION

[0037] The present invention provides a network security server control cabinet based on big data analysis. The solution consists of the following modules: a data acquisition module, a data storage module, a data analysis module, a control and response module, and a monitoring and management module.

[0038] The data collection module is used to collect various types of data from the network. The sources of data include routers, switches, firewalls, server logs, etc. The types include network traffic, access logs, user behavior, device status, etc. The data collection module adopts a distributed structure to process multi-source data in a large-scale network environment. Each network node is equipped with a local collection agent to collect data in real time and transmit it to the central data storage and analysis system.

[0039] Network traffic data is obtained from routers and switches, and collected through mirror ports: when a router or switch receives or forwards traffic, the traffic is also copied to the configured mirror port connected to the data collector; through the Sys log protocol (system log protocol), access logs and device status are collected from firewalls, servers, intrusion registration systems (IDS) and other devices. To ensure security, logs are transmitted to a centralized storage server using the TCP protocol.

[0040] In addition to the above passive collection and monitoring methods, the system can also choose to use SNMP (Simple Network Management Protocol) to regularly poll traffic information through the router's SNMP interface, actively send detection requests to network devices, query device status information, and use tools such as Ping and Traceroute to analyze network topology and latency.

[0041] The collected data needs to be properly processed. Due to the characteristics of some connections, such as the retransmission mechanism of TCP connections to prevent loss, the collected data may contain redundant information; the data formats from different sources may not be consistent. For example, logs are usually in text format, while network traffic is usually in PCAP format. Therefore, the collection module needs to unify the data format into a common format. In this solution, JSON format is used for storage to facilitate subsequent analysis and processing. The processed data will be transmitted to the storage module for subsequent analysis. When these processed data are transmitted to the storage module, they are encrypted using a secure transmission protocol (TLS / SSL) to protect the data in transit from being intercepted or tampered with.

[0042] After the data is collected, it needs to be stored. The storage module needs to be highly scalable and reliable, and needs to be able to support fast reading and writing of large-scale data to meet the storage needs of large amounts of data. Data with high-frequency reading needs is stored on high-speed storage devices (such as solid-state drives SSDs), while data with low access frequencies is stored on relatively cheaper and higher-storage disk storage devices, such as ordinary mechanical hard drives, to optimize the balance between performance and cost.

[0043] The data analysis module is the core part of the whole solution. In this module, valuable information is extracted from the stored data and analyzed through machine learning, data mining and other technologies, so as to realize the identification of abnormal behavior, detection of security threats, etc. This module should have the ability to process data in real time.

[0044] In the above steps, the data collected and processed can be used as a large data set for data mining. Using data mining, outliers in the data set can be identified.

[0045] Before data mining, the collected network data needs to be cleaned, noise data, missing values ​​and duplicate data need to be removed, and the data needs to be standardized and normalized. Then, due to the huge amount of data, in order to ensure the speed of subsequent calculations, these data need to be regulated, and PCA (principal components analysis) is used to reduce the data dimension to reduce the calculation pressure. In addition, the original data should also be converted into a format suitable for mining algorithm processing. For example, the time series data is divided into windows of fixed length; the log is parsed, and the IP address, timestamp and other information are extracted from the text content, and it is converted into structured data in a tabular format for easy analysis.

[0046] Then, feature extraction is performed on the data. Feature extraction here refers to extracting content that can effectively represent the essence of the data from the original data for better analysis.

[0047] For network traffic, obtain its source IP and destination IP, protocol type, packet size and arrival time, source port and destination port used by the packet, etc., and summarize them according to the corresponding time window

[0048] For device status, obtain its CPU usage, memory usage, and network interface working status

[0049] After feature extraction is completed, it is normalized to ensure that the data format meets the input requirements of subsequent algorithms. Then, the mining algorithm can use these contents for detection. In this solution, the FP-Growth algorithm (an algorithm for mining frequent item sets) is used to detect frequent item sets.

[0050] First, convert the data into a transaction set. Divide the data into time windows, and the feature set in each time window is a transaction. For example, a transaction contains the previously extracted features: source IP, destination IP, protocol type, port number, CPU usage, local memory usage, etc. The feature values ​​of non-discrete values ​​need to be converted into discrete values ​​using interval partitioning and classification processing, and then the FP-Growth algorithm is executed.

[0051] When executing the algorithm, first, scan all transactions and calculate the support (frequency of occurrence in all transactions) of each feature item (for example: source IP = 192.168.1.1). Set a minimum support threshold, and feature items with support below the threshold will be filtered. For example, if the frequency of the feature item memory usage is less than 10%, this support item will be ignored. The remaining features are sorted from high to low (descending order) according to support, and the feature items are rearranged in this order in each transaction. Traverse the sorted transaction set from the beginning to build the FP tree. Each node of the FP tree represents a feature item, and the path identifies a group of related feature items in the transaction; if two transactions have the same prefix (that is, the first few items are the same), they share the same path node in the FP tree.

[0052] Next, frequent item sets need to be mined from the FP tree. Starting from the bottom of the tree, the conditional pattern base (the set of all paths containing a specific item) is mined layer by layer upwards; for each conditional pattern base, a smaller FP tree is recursively constructed and mining continues until all frequent item sets are found. According to the support threshold, all frequent item sets that meet the conditions are generated. These frequent item sets represent common behavior patterns in the network.

[0053] When identifying abnormal behavior, the mined frequent item sets are compared with the actual transaction sets, and transactions that do not appear in the frequent item sets are considered abnormal.

[0054] After identifying the anomaly, it is necessary to determine what kind of anomaly it is, such as DDOS attack, SQL injection, etc. For this purpose, an attack pattern rule library is established. The detected abnormal behavior will be matched with the rule library. If the match is successful, it means that the behavior belongs to the security threat type contained in the rule library. If the abnormal state cannot be successfully matched with the rule library, the abnormal state will be marked as a location, and the suspected threat will be submitted to the relevant staff (security experts) for manual review. At the same time, the relevant original data (original logs, etc.) will be submitted to the experts so that they can confirm whether this is a threat through more detailed analysis. If so, the anomaly will be marked and rules will be extracted for it.

[0055] The rule base needs to be updated regularly; for those abnormal records identified manually, they should be entered into the rule base.

[0056] After the data is analyzed by the data analysis module and the results are obtained (i.e., after abnormal behavior / security threats are identified), the network control machine can adjust the security policy of the server according to the analysis results, for example, update the firewall rules, block suspicious connections, isolate infected devices, etc. These adjustments to the server's security policy can be implemented using python scripts. For example, if you want to update the firewall rules, you can use the python script to automatically push the new rules to the firewall using the API or command line tool. Each security threat type corresponds to a unique number. When the threat type is obtained through the previous step, its number will be submitted to the python script of the control module, and the script will perform the corresponding operation according to its number. In particular, if the threat type does not exist in the current rule base and needs to be manually identified, the security policy requires the administrator to manually operate.

[0057] The monitoring and management module is not a specific link in the process, but an overall control module. This module can provide the administrator of the control machine with an interface to visualize the system status and display the security status of the network and server in real time. In particular, since the automatic adjustment of server security policies may not meet actual needs, the monitoring and management module also allows administrators to manually adjust security policies or configurations to better meet actual needs.

[0058] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any ordinary technician in the industry can smoothly implement the present invention as described above. However, any equivalent changes, modifications and evolutions made by technicians familiar with the profession without departing from the scope of the technical solution of the present invention using the technical content disclosed above are all equivalent embodiments of the present invention. At the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A network security server control cabinet based on big data analysis, characterized in that: include: The data collection module is used to collect data from a variety of network devices in real time, including network traffic, access logs, and device status. It uses a distributed structure to collect data and converts the collected data into a JSON format, which is securely transmitted through TLS / SSL encryption technology. A data storage module, used for storing the collected data; Data analysis module, which uses machine learning and data mining techniques to extract valuable information from stored data and accurately identify abnormal behaviors and security threats in the network through data preprocessing, feature extraction and mining algorithms; The control and response module automatically adjusts the server security policy based on the results of the data analysis module, including updating firewall rules and blocking suspicious connections to quickly respond to known and unknown security threats; The monitoring and management module provides administrators with an interface that displays the real-time security status of the network and server, and allows administrators to manually adjust security policies or configurations.

2. A network security server control cabinet based on big data analysis according to claim 1, characterized in that: The data collection module is further configured to collect data from routers, switches, firewalls, and server logs.

3. A network security server control cabinet based on big data analysis according to claim 2, characterized in that: The data analysis module further includes data cleaning, standardization, normalization and preprocessing steps of reducing data dimension using PCA.

4. A network security server control cabinet based on big data analysis according to claim 3, characterized in that: The data storage module adopts a distributed storage system.

5. A network security server control cabinet based on big data analysis according to claim 4, characterized in that: The data analysis module further uses deep learning algorithms to perform anomaly detection and behavior analysis to improve the ability to identify complex security threats.

6. A network security server control cabinet based on big data analysis according to claim 5, characterized in that: The control and response module also includes a threat intelligence integration function, which can automatically obtain and update external threat intelligence information to enhance the defense capability against new security threats.

7. A network security server control cabinet based on big data analysis according to claim 6, characterized in that: The monitoring and management module provides a visual security dashboard that displays key security indicators and real-time threat intelligence to help administrators fully understand the network security status. The monitoring and management module also provides alarm and notification functions to report security incidents and abnormal situations to administrators in a timely manner for rapid response and processing.

8. A network security management method for a network security server control cabinet, applied to a network security server control cabinet based on big data analysis as claimed in any one of claims 1 to 7, characterized in that: The following steps are involved: a. Collect data from network devices using the data acquisition module; b. Storing the collected data in the data storage module; c. Analyze data through the data analysis module to identify abnormal behaviors and security threats in the network; d. Based on the data analysis results, the control and response module automatically adjusts the server security policy to respond to security threats in a timely manner; e. Use the monitoring and management module to monitor the security status of the network and server in real time, and allow administrators to manually adjust security policies or configurations to ensure network security.

9. A network security management method for a network security server control cabinet according to claim 8, characterized in that: In step c, the data analysis module also uses historical data to train and optimize the model to improve the ability to predict and identify future security threats.

10. The network security management method of a network security server controller according to claim 9, characterized in that: In step d, the control and response module also automatically selects appropriate response measures based on the severity and type of security threats, including isolating infected systems and blocking malicious traffic.