Network resource intrusion detection method combined with big data analysis

Abstract

The invention relates to a network resource intrusion detection method in combination with big data analysis. The method comprises the following steps: S1, collecting Internet of Things data of an electric power information system; s2, preprocessing the acquired data; s3, performing analysis and modeling on normal behaviors and abnormal behaviors in the network by using a big data analysis technology based on the data preprocessed in the step S2; and S4, in a network operation process, monitoring newly generated data in real time, and comparing the newly generated data with the established behavior model to detect whether an intrusion behavior exists. Through comprehensive data acquisition, effective preprocessing, advanced behavior analysis and modeling, real-time monitoring and continuous optimization updating, the intrusion behavior in the network can be detected more accurately and efficiently, powerful guarantee is provided for network security, the security and protection capability of the network are enhanced, and the network security and protection capability are improved. Safety accidents caused by the fact that vulnerabilities are not found in time are reduced, and the recognition accuracy of network intrusion detection of the electric power information system is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data and intrusion detection, and relates to a network resource intrusion detection method, in particular to a network resource intrusion detection method combined with big data analysis. Background Art

[0002] With the rapid development of the Internet, the economy, society, culture and people's lives are increasingly dependent on the Internet. The rise and development of the Internet economy has greatly changed people's lives, work and thinking, and promoted economic development. Under the influence and strong promotion of the world's informationization trend, a digital boom has gradually emerged. At present, the "big data economy" is striding towards us. A large number of information systems have become key infrastructures, and many of the businesses have been integrated with international standards, such as resource sharing, e-commerce, and financial networks.

[0003] In the current context of digitalization and networking, the operation and management of power systems increasingly rely on the support of information technology. However, the networking of power systems also brings a series of cybersecurity challenges, because its critical nature and widespread application make it a target of malicious intrusion and attack.

[0004] Although there are many firewall products in the world, due to the limitations of firewall technology itself, it is impossible to completely protect the system from attacks by intelligent attackers such as hackers.

[0005] Therefore, it is necessary to provide a network resource intrusion detection method combined with big data analysis to solve the above technical problems.

[0006] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a network resource intrusion detection method combined with big data analysis, which can solve the problem that due to the limitations of firewall technology itself, it is impossible to completely protect the system from attacks by intelligent attackers such as hackers.

[0008] The present invention solves the practical problem by adopting the following technical solutions:

[0009] A network resource intrusion detection method combined with big data analysis includes the following steps:

[0010] S1. Collect IoT data from the power information system;

[0011] S2, preprocessing the collected data;

[0012] S3, based on the data preprocessed in step S2, using big data analysis technology to analyze and model normal and abnormal behaviors in the network;

[0013] S4. During network operation, new data is monitored in real time and compared with the established behavior model to detect whether there is any intrusion.

[0014] Moreover, the data in S1 includes: network traffic data, system logs and user behavior data;

[0015] Moreover, the specific steps of pre-processing in S2 include:

[0016] (1) Data cleaning: Data cleaning is performed on the data collected by S1;

[0017] (2) Data conversion: perform data conversion on the cleaned data;

[0018] (3) Feature extraction: Extract features from the converted data.

[0019] Moreover, the specific method steps of analysis and modeling in S3 include:

[0020] (1) Statistical analysis: determining the normal range and pattern of network resource usage by statistical analysis of the data pre-processed in step S2;

[0021] (2) Machine learning algorithm: Apply machine learning algorithm to perform behavior modeling and anomaly detection on the data preprocessed in step S2 in step (1).

[0022] Moreover, the specific steps of S4 include:

[0023] (1) Real-time data collection and processing: Continuously collect real-time data from the network and perform rapid preprocessing and feature extraction;

[0024] (2) Abnormality judgment: Real-time data is input into the trained model to calculate its deviation from the normal behavior pattern or to determine whether it is abnormal through the model's prediction. If the data features exceed the normal range or the model judges it as abnormal, it indicates that there is intrusion behavior.

[0025] (3) Early warning and response: When an abnormality is detected, early warning information is issued in a timely manner.

[0026] Moreover, after step (3) of step S4, the following steps are also included: continuously updating and expanding the data set used to train the model, including adding new normal data to reflect changes in the network environment, and collecting new intrusion samples to enhance the model's ability to detect new intrusion methods.

[0027] Moreover, after step S4, the following steps are also included:

[0028] S5. Use new data sets or actual network data to test and evaluate the model, and analyze the model's accuracy, recall rate, false alarm rate and other indicators;

[0029] S6. Based on the evaluation results, adjust the model parameters, optimize the algorithm selection or improve the model structure.

[0030] Advantages and beneficial effects of the present invention:

[0031] The present invention provides a network resource intrusion detection method combined with big data analysis, which improves the quality and availability of data through comprehensive data collection and effective preprocessing of the collected data, and uses big data analysis technology to analyze and model normal behavior and abnormal behavior in the network, so as to more accurately and efficiently detect intrusion behavior in the network. During the operation of the network, the newly generated data is monitored in real time and compared with the established behavior model to detect whether there is intrusion behavior, thereby providing strong protection for network security, enhancing the security and protection capabilities of the network, reducing safety accidents caused by failure to discover vulnerabilities in a timely manner, and improving the recognition accuracy of network intrusion detection in power information systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a preferred embodiment of the network resource intrusion detection method combined with big data analysis provided by the present invention. DETAILED DESCRIPTION

[0033] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0034] Please refer to Figure 1 , Figure 1 A flowchart of a preferred embodiment of the network resource intrusion detection method combined with big data analysis provided by the present invention.

[0035] A network resource intrusion detection method combined with big data analysis, such as Figure 1 As shown, the following steps are included:

[0036] S1. Collect IoT data from the power information system;

[0037] The data in S1 includes: network traffic data, system logs and user behavior data;

[0038] Among them, network traffic data: includes the source address, destination address, protocol type, port number, traffic size, data packet content, etc. Real-time network traffic information is obtained through network traffic monitoring equipment (such as network probes, traffic sensors, etc.).

[0039] System logs: including operating system logs, application logs, and security device logs (such as firewall logs, intrusion detection system logs, etc.). These logs record the operating status, event information, and possible abnormal situations of the system and applications.

[0040] User behavior data: includes the user's login time, login location, and operation behavior (such as file access, database operation, etc.). It can be collected from user authentication systems, application access records, etc.

[0041] S2. Preprocess the collected data to improve the quality and availability of the data;

[0042] The specific steps of preprocessing in S2 include:

[0043] (4) Data cleaning: Data cleaning is performed on the data collected by S1;

[0044] (5) Data conversion: perform data conversion on the cleaned data;

[0045] (6) Feature extraction: Extract features from the converted data.

[0046] The data cleaning is used to remove noise data, duplicate data and incomplete data; the data conversion is used to convert data in different formats into a unified format to facilitate subsequent analysis and processing; the feature extraction is used to extract key information that can reflect the characteristics of network resource intrusion from the original data.

[0047] The specific method of step (1) of S2 is: Data cleaning: removing noise data, duplicate data and incomplete data. For example, some obviously abnormal or erroneous data packet information in network traffic data is removed; and irregular or invalid records in system logs are sorted out.

[0048] The specific method of step (2) of S2 is: Data conversion: converting data of different formats into a unified format to facilitate subsequent analysis and processing, such as standardizing the time formats in various log files, converting IP addresses in network traffic data into digital formats, etc.

[0049] The specific method of step (3) of S2 is: Feature extraction: extract key information that can reflect the characteristics of network resource intrusion from the original data. For example, extract the length distribution of data packets and statistical characteristics of traffic (such as mean, variance, peak value, etc.) from network traffic data; extract the occurrence frequency and time series characteristics of specific events from system logs; extract the mode and habit characteristics of user operations from user behavior data.

[0050] Since the amount of data collected and pre-processed is usually very large, appropriate big data storage technology and management systems can be selected to store and manage this data, such as:

[0051] Distributed file system: Hadoop's HDFS (Hadoop Distributed File System) can store large-scale data in a distributed manner on multiple nodes, improving storage capacity and reliability.

[0052] Distributed databases: such as HBase and Cassandra, which can support efficient storage and fast query of large-scale data and are suitable for storing structured and semi-structured data.

[0053] S3, based on the data preprocessed in step S2, using big data analysis technology to analyze and model normal and abnormal behaviors in the network;

[0054] The specific steps of analysis and modeling in S3 include:

[0055] (1) Statistical analysis: Determine the normal range and pattern of network resource usage through statistical analysis of the data preprocessed in step S2; for example, calculate the mean and standard deviation of network traffic in different time periods to determine the fluctuation range of normal traffic; analyze the common operation patterns and frequencies of user behaviors in specific scenarios, etc.

[0056] (2) Machine learning algorithm: Apply machine learning algorithm to perform behavior modeling and anomaly detection on the data preprocessed in step S2 in step (1).

[0057] The commonly used algorithms for step (2) of S3 include the following mature algorithms in the prior art:

[0058] a. Supervised learning algorithms: such as decision trees, support vector machines, and naive Bayes, use labeled normal data and intrusion data for training to establish a classification model to determine whether new data is an intrusion.

[0059] b. Unsupervised learning algorithms: such as clustering algorithms (K-Means clustering, DBSCAN, etc.), which can divide data into different clusters and discover abnormal clusters that are significantly different from normal behavior patterns; autoencoders can learn the normal feature representation of data and detect abnormal data by reconstructing errors.

[0060] c. Deep learning algorithms: such as convolutional neural networks (CNN), recurrent neural networks (RNN) and their variants (such as long short-term memory networks (LSTM), gated recurrent units (GRU), etc.), can automatically learn complex features and patterns from large amounts of data, and conduct in-depth analysis of network traffic data, system log data, etc. to detect intrusion behaviors.

[0061] S4. During network operation, new data is monitored in real time and compared with the established behavior model to detect whether there is intrusion behavior;

[0062] In the S4, during the network operation, the newly generated data is monitored in real time and compared with the established behavior model to detect whether there is intrusion behavior. The specific steps include:

[0063] (1) Real-time data collection and processing: Continuously collect real-time data from the network and perform rapid preprocessing and feature extraction to ensure the timeliness and accuracy of the data.

[0064] (2) Abnormality judgment: Input real-time data into the trained model and calculate its deviation from the normal behavior pattern or judge whether it is abnormal through the model's prediction. If the data features exceed the normal range or the model judges it as abnormal, it means there is intrusion.

[0065] (3) Early warning and response: When an anomaly is detected, an early warning message is issued in a timely manner to notify relevant security personnel or systems to take appropriate measures, such as blocking suspicious network connections, isolating affected systems or resources, and further investigating and analyzing the anomaly. At the same time, the relevant anomaly information and processing process are recorded for subsequent auditing and analysis.

[0066] It also includes: continuously updating and expanding the data set used to train the model, including adding new normal data to reflect changes in the network environment, and collecting new intrusion samples to enhance the model's ability to detect new intrusion methods. New data can be obtained by sharing data with other security systems and obtaining the latest threat intelligence from security vendors.

[0067] S5. Use new data sets or actual network data to test and evaluate the model, analyze the model's accuracy, recall rate, false alarm rate and other indicators, and understand the model's performance;

[0068] S6. Based on the evaluation results, adjust the model parameters, optimize the algorithm selection or improve the model structure to improve the detection performance of the model.

[0069] The working principle of the network resource intrusion detection method combined with big data analysis provided by the present invention is as follows:

[0070] First, collect IoT data from the power information system; then pre-process the collected data to improve the quality and availability of the data; then use big data analysis technology to analyze and model normal and abnormal behaviors in the network; during network operation, monitor the newly generated data in real time and compare it with the established behavior model to detect whether there is intrusion; use new data sets or actual network data to test and evaluate the model, analyze the model's accuracy, recall rate, false alarm rate and other indicators to understand the model's performance; finally, based on the evaluation results, adjust the model's parameters, optimize the algorithm selection or improve the model structure to improve the model's detection performance.

[0071] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A network resource intrusion detection method combined with big data analysis, characterized in that: The following steps are involved: S1. Collect IoT data from the power information system; S2, preprocessing the collected data; S3, based on the data preprocessed in step S2, using big data analysis technology to analyze and model normal and abnormal behaviors in the network; S4. During network operation, new data is monitored in real time and compared with the established behavior model to detect whether there is any intrusion.

2. According to the network resource intrusion detection method combined with big data analysis according to claim 1, it is characterized by: The data in S1 includes: network traffic data, system logs and user behavior data.

3. According to claim 1, a network resource intrusion detection method combined with big data analysis is characterized in that: The specific steps of preprocessing in S2 include: (1) Data cleaning: Data cleaning is performed on the data collected by S1; (2) Data conversion: Perform data conversion on the cleaned data; (3) Feature extraction: Extract features from the converted data.

4. The network resource intrusion detection method combined with big data analysis according to claim 1 is characterized in that: The specific steps of analysis and modeling in S3 include: (1) Statistical analysis: determining the normal range and pattern of network resource usage by statistical analysis of the data pre-processed in step S2; (2) Machine learning algorithm: Apply machine learning algorithm to perform behavior modeling and anomaly detection on the data preprocessed in step S2 in step (1).

5. The network resource intrusion detection method combined with big data analysis according to claim 1 is characterized in that: The specific steps of S4 include: (1) Real-time data collection and processing: Continuously collect real-time data from the network and perform rapid preprocessing and feature extraction; (2) Abnormality judgment: Real-time data is input into the trained model to calculate its deviation from the normal behavior pattern or to determine whether it is abnormal through the model's prediction. If the data features exceed the normal range or the model determines it to be abnormal, it indicates that there is intrusion behavior; (3) Early warning and response: When an abnormality is detected, early warning information is issued in a timely manner.

6. A network resource intrusion detection method combined with big data analysis according to claim 5, characterized in that: After step (3) of step S4, the following steps are also included: Continuously update and expand the data set used to train the model, including adding new normal data to reflect changes in the network environment and collecting new intrusion samples to enhance the model's ability to detect new intrusion methods.

7. The network resource intrusion detection method combined with big data analysis according to claim 1 is characterized in that: After step S4, the following steps are also included: S5. Use new data sets or actual network data to test and evaluate the model, and analyze the model's accuracy, recall, and false positive rate indicators; S6. Based on the evaluation results, adjust the model parameters, optimize the algorithm selection or improve the model structure.

Images