Network traffic analysis and intrusion detection method based on multi-level security management and control system

Abstract

The application discloses a network flow analysis and intrusion detection method based on a multilevel security management and control system, and relates to the technical fields of Internet of Things and network security. The application comprises the following steps: collecting data packets of network flow data and caching key attributes of the data packets; performing data cleaning on the cached data packets to obtain to-be-detected data; performing abnormality detection on the to-be-detected data; when attack flow is detected, storing the attack flow into a database and performing alarm processing; the application optimizes the performance of the system and improves the credibility of transmission quality through network flow analysis; through the construction of a real-time detection framework, the related time sequence overhead is reduced, the detection capability is greatly enhanced, the poor performance of a prediction model in performance is avoided, and the problem of the effectiveness of the prediction model in processing complex network environments is solved.

Description

Technical Field

[0001] This invention relates to the fields of Internet of Things and network security technology, and in particular to a method for network traffic analysis and intrusion detection based on a multi-level security control system. Background Technology

[0002] With the development of big data, AI, and 5G, intelligent transportation integrated management platforms rely on advanced technologies to organically integrate the Internet of Things, cloud computing, artificial intelligence, geographic information systems, and real-time communication technologies, constructing a powerful and intelligent platform that provides comprehensive and efficient monitoring, management, and optimization for urban traffic. While these platforms bring numerous conveniences and benefits to urban transportation, their rapid development may also bring some drawbacks and challenges. As the nation places increasing emphasis on cybersecurity, network traffic analysis and intrusion detection—monitoring and analyzing network traffic to detect and identify potential intrusions and take timely measures to prevent and respond to cyberattacks—have become crucial functions for intelligent transportation integrated management platforms in terms of cybersecurity assurance, data analysis and decision support, and real-time event response. However, traditional network traffic analysis and intrusion detection methods for intelligent transportation integrated management platforms still face challenges in areas such as selecting the optimal number of features for real-time traffic detection, feature extraction, and the real-time nature of computational analysis. Traditional network traffic analysis and intrusion detection systems suffer from significant lag, failing to provide timely and effective detection, analysis, and alarm processing for attack traffic. As attackers employ increasingly sophisticated attack techniques, and traditional intelligent transportation integrated management platform network traffic analysis and intrusion detection methods rarely focus on real-time attack identification, it is difficult to extract effective features to detect new types of attacks.

[0003] Therefore, there is an urgent need for a network traffic analysis and intrusion detection method based on a multi-level security management system to overcome the limitations of existing technologies. Summary of the Invention

[0004] The purpose of this invention is to provide a network traffic analysis and intrusion detection method based on a multi-level security management system to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] Network traffic analysis and intrusion detection methods based on a multi-level security control system include:

[0007] Collect network traffic data packets and extract key attributes of the packets for caching;

[0008] Clean the cached data packets to obtain the data to be tested;

[0009] Perform anomaly detection on the data to be tested;

[0010] When attack traffic is detected, it is stored in the database and an alarm is triggered.

[0011] As a preferred embodiment of the network traffic analysis and intrusion detection method based on a multi-level security management system of the present invention, the step of collecting network traffic data packets and extracting key attributes of the data packets for caching includes:

[0012] When a large amount of real-time network traffic data is input into the security layer of the IoT smart terminal, the network traffic data is collected through a micro-batch processing mechanism. The network traffic data of this time interval is accumulated on the main thread according to a preset time interval Δt to obtain the initial data packets.

[0013] By using preset necessary fields and real-time calculated traffic entropy values, key attributes in the data packets are extracted as data packet header attributes, and the data packets are cached to obtain cached data packets. The necessary fields include network layer, transport layer, and time.

[0014] As a preferred embodiment of the network traffic analysis and intrusion detection method based on a multi-level security management system of the present invention, the step of cleaning the cached data packets to obtain the data to be detected includes:

[0015] Establish a queue connection and listen for cached data packets;

[0016] The cached data packets are read in batches according to the time window. A window calculation is triggered every time interval. The cached data packets are statistically analyzed based on the data packet header attributes. The multiple dimensions of attributes include data packet dimension attributes and session dimension attributes.

[0017] Within a given time interval, the number of data packets with the same destination IP as the current packet is counted and used as an attribute of the data packet dimension.

[0018] By reassembling TCP sessions in cached data packets, and counting the data packets of the same TCP session during the reassembly process, the number of sessions with the same destination IP as the current session within a time interval is counted as an attribute of the statistical session dimension.

[0019] The various attributes are added to the packet header attributes, and cached packets with the same packet header attributes as the current packet are removed to obtain the data to be detected.

[0020] As a preferred embodiment of the network traffic analysis and intrusion detection method based on a multi-level security management system of the present invention, the anomaly detection of the data to be detected includes:

[0021] The quality threshold is set by combining the quality threshold summarization algorithm, BUS algorithm, K-means algorithm, and data summarization algorithm. The data to be tested is then optimized and clustered to obtain the summary data.

[0022] The K-means algorithm is used to perform initial clustering of the data to be detected, thereby obtaining the initial centroid set of the data to be detected.

[0023] By introducing a quality threshold summary algorithm based on time intervals and combining it with an initial centroid set to cluster the data to be detected, a high-quality subset of data is obtained.

[0024] A primary compressed summary is obtained by performing conventional mode compression on a high-quality subset of data using the BUS algorithm to preserve data features with high gain and low loss.

[0025] The primary compressed digest is supplemented in a non-scale manner using a data aggregation algorithm to form a complete dual-channel compressed digest, thus obtaining the aggregated data.

[0026] Historical network traffic datasets are extracted, and threshold correlation algorithms, genetic algorithms, and particle swarm optimization algorithms are combined to extract a subset of abnormal features from the historical network traffic datasets. In the first stage, MLP, KNN, DT, and RF machine learning frameworks are used to analyze the extracted anomalous feature subsets to detect the type of network traffic data. When the type is malicious traffic, the second stage is triggered. In the second stage, voting classifiers and CatBoost ensemble learning mechanisms are used to identify the nature of the attack. The classifiers corresponding to the first and second stages with the best detection performance are integrated to construct a real-time detection framework.

[0027] The aggregated data is input into the real-time detection framework, and the type of aggregated data is output. When the type of aggregated data is malicious traffic, the attack nature of the aggregated data is also output. The type and attack nature of the aggregated data are attached to the aggregated data to form attack traffic.

[0028] As a preferred embodiment of the network traffic analysis and intrusion detection method based on a multi-level security management system of the present invention, the step of obtaining the initial centroid set of the data to be detected by initial clustering using the K-means algorithm includes:

[0029] A test set is constructed based on the data to be tested. Data points with a specified number of clusters are randomly selected from the test set as centroids to construct a centroid set.

[0030] Calculate the distance from each data point in the set to be tested to all centroids and assign it to the cluster with the nearest centroid;

[0031] Update the centroid of each cluster until the change in the centroid is less than the change threshold, then stop updating and output the initial centroid set;

[0032] Updating the centroid of each cluster involves calculating the mean of the data points within each cluster as the new centroid of the cluster.

[0033] As a preferred embodiment of the network traffic analysis and intrusion detection method based on a multi-level security management system of the present invention, the step of obtaining a high-quality data subset by introducing a quality threshold summary algorithm for time intervals and combining it with an initial centroid set to cluster the data to be detected includes:

[0034] The data to be tested is segmented according to time intervals and a subset to be tested is constructed. d = 1, 2, ..., n, where n represents the total number of subsets to be divided for testing;

[0035] Calculate the subset to be tested The distance between data point j and each centroid in the initial centroid set is calculated, and the centroids in the initial centroid set are sorted in descending order according to the distance.

[0036] When the distance to the first and second centroids is less than the quality threshold, data point j is assigned to the cluster of the first and second centroids.

[0037] Remove the subset to be tested Data point j in the middle;

[0038] When the subset to be tested When all data points are empty, count the data points within each cluster, calculate the mean of the data points within each cluster as the new centroid of the cluster, output the cluster centroid, and obtain a high-quality subset of data.

[0039] As a preferred embodiment of the network traffic analysis and intrusion detection method based on a multi-level security management system of the present invention, the method involves using the BUS algorithm to perform conventional mode compression on a high-quality subset of data to preserve data characteristics with high gain and low loss, thereby obtaining a primary compressed digest; and then using a data aggregation algorithm to perform non-scale supplementation on the primary compressed digest to form a complete digest with dual-channel compression, obtaining aggregated data including:

[0040] Extract closed frequent itemsets from high-quality subsets of data to construct candidate sets. Candidate sets = closed frequent itemsets High-quality data subsets;

[0041] Candidates are selected from the candidate set, and the compression gain of the candidate is calculated. When the compression gain is greater than the gain threshold, the candidate is recorded in the gain candidate set. The information loss of the candidate in the gain candidate set is calculated. The candidate in the gain candidate set is sorted in descending order according to the information loss. The first candidate is selected as the best candidate and recorded in the current summary set and the candidate set is updated. Updating the candidate set includes deleting the best candidate until the number of summaries in the current summary set is equal to the preset number of summaries. The current summary set is output as the primary compressed summary.

[0042] Extract closed frequent itemsets from high-quality subsets of data to construct candidate sets. Candidate sets = closed frequent itemsets High-quality data subsets;

[0043] Candidates are selected from the candidate set, and the compression gain of the candidate is calculated. When the compression gain is greater than the gain threshold, the candidate is recorded in the gain candidate set. The information loss of the candidate in the gain candidate set is calculated. The candidate in the gain candidate set is sorted in ascending order according to the information loss. The first candidate is selected as the best candidate and recorded in the current summary set and the candidate set is updated. Updating the candidate set includes deleting the best candidate until the number of summaries in the current summary set is equal to the preset number of summaries. The current summary set is output as the reverse compressed summary.

[0044] The primary compressed digest and the reverse compressed digest are merged to form a complete digest with dual-channel compression, thus obtaining the summary data.

[0045] As a preferred embodiment of the network traffic analysis and intrusion detection method based on a multi-level security management system of the present invention, the construction of the real-time detection framework includes:

[0046] Extract the original feature set from the historical network traffic dataset, perform feature dimensionality reduction on the original feature set, and obtain the abnormal feature subset:

[0047] Calculate any two features in the original feature set f 1 and f The correlation coefficient of 2, calculated using the following formula:

[0048]

[0049] in, x i , y i They represent f 1 and f 2. Corresponding values ​​of two features , They represent f Calculate the average and 1 f Calculate the average value of 2;

[0050] When the absolute value of the correlation coefficient is less than the coefficient threshold, any feature is selected as a record in the feature set after dimensionality reduction.

[0051] The reduced feature set is input into a genetic algorithm or particle swarm optimization algorithm to output the first anomaly-related feature for final screening.

[0052] Genetic algorithms or particle swarm optimization algorithms run directly on the original feature set and find a subset of key features that distinguish attacks from normal ones through global optimization.

[0053] Apply a threshold correlation algorithm to the feature subset to remove strongly correlated features and output a second anomaly-related feature that is correlated with the anomaly.

[0054] The first anomaly-related feature and the second anomaly-related feature are merged to form an anomaly feature subset;

[0055] In the first stage, MLP, KNN, DT, and RF machine learning frameworks are built and trained respectively, taking anomaly feature subsets as input and outputting the type of network traffic data; in the second stage, voting classifier and CatBoost ensemble learning mechanisms are built and trained respectively to identify the nature of attacks.

[0056] Feature samples with specific attack labels are extracted from the database as test sets and input into the first and second stages to calculate the type accuracy of MLP, KNN, DT and RF, as well as the property accuracy of voting classifier and CatBoost.

[0057] Based on the accuracy of the type, MLP, KNN, DT, and RF are sorted in descending order, and the first machine learning framework is selected as the machine learning framework for the first stage. Based on the accuracy of the property, the ensemble learning mechanism with the highest property accuracy is selected as the ensemble learning mechanism for the second stage, thus obtaining the real-time detection framework.

[0058] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: The network traffic analysis and intrusion detection method based on a multi-level security management system provided by this invention includes: collecting network traffic data packets and extracting key attributes of the data packets for caching; cleaning the cached data packets to obtain the data to be detected; performing anomaly detection on the data to be detected; and storing the attack traffic in the database and performing alarm processing when attack traffic is detected. This invention optimizes system performance and improves the reliability of transmission quality through anomaly detection, avoids the situation where the prediction model performs poorly in terms of performance, and solves the problem of its effectiveness in handling complex network environments. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0060] Figure 1 This is a schematic diagram of the method flow in an embodiment of the present invention;

[0061] Figure 2 This is a schematic diagram of the micro-batch processing mechanism in an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the data cleaning process in an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of the anomaly detection process in an embodiment of the present invention;

[0064] Figure 5 This is an intelligent transportation integrated management platform based on a multi-level security control system, as described in this embodiment of the invention.

[0065] Figure 6 This is a schematic diagram of the distributed detection framework process in an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0067] like Figure 1 As shown, this is an embodiment of the present invention, which provides a network traffic analysis and intrusion detection method based on a multi-level security management system, including:

[0068] Collect network traffic data packets and extract key attributes of the packets for caching;

[0069] Clean the cached data packets to obtain the data to be tested;

[0070] Perform anomaly detection on the data to be tested;

[0071] When attack traffic is detected, it is stored in the database and an alarm is triggered.

[0072] Specifically, the process of collecting network traffic data packets and extracting key attributes of the packets for caching includes:

[0073] When a large amount of real-time network traffic data is input into the security layer of the IoT smart terminal in a multi-level security control system, a micro-batch processing mechanism is used to collect data packets from the network traffic data. On the main thread, network traffic data for this time interval is accumulated according to a preset time interval Δt to obtain initial data packets. The micro-batch processing mechanism collects network data packets on the main thread and accumulates the data packets for this time interval according to a certain time interval Δt. Then, protocol parsing and attribute extraction are performed in other threads to improve the throughput of the data acquisition sensors. Figure 2 As shown;

[0074] Use micro-batch processing mechanisms to improve the throughput of the data acquisition unit.

[0075] By using preset necessary fields and real-time calculated traffic entropy values, key attributes in the data packets are extracted as data packet header attributes, and the data packets are cached to obtain cached data packets. The necessary fields include network layer, transport layer, and time.

[0076] Specifically, the process of cleaning cached data packets includes, for example... Figure 3 As shown, the data to be detected includes:

[0077] Establish a queue connection and listen for cached data packets;

[0078] The cached data packets are read in batches according to the time window. A window calculation is triggered every time interval. The cached data packets are statistically analyzed based on the data packet header attributes. The multiple dimensions of attributes include data packet dimension attributes and session dimension attributes.

[0079] Within a given time interval, the number of data packets with the same destination IP as the current packet is counted and used as an attribute of the data packet dimension.

[0080] By reassembling TCP sessions in cached data packets, and counting the data packets of the same TCP session during the reassembly process, the number of sessions with the same destination IP as the current session within a time interval is counted as an attribute of the statistical session dimension.

[0081] The various attributes are added to the packet header attributes, cached packets with the same packet header attributes as the current packet are removed, some redundant information is removed, and the data to be detected is obtained.

[0082] Specifically, the anomaly detection of the data to be detected, such as... Figure 4 As shown, it includes:

[0083] A quality threshold is set by combining quality threshold summarization algorithms, BUS algorithms, K-means algorithms, and data aggregation algorithms. The data to be tested is optimized and clustered. Data optimization and clustering reduce network traffic, save time in critical applications, and obtain summary data.

[0084] The K-means algorithm is used to perform initial clustering of the data to be detected, thereby obtaining the initial centroid set of the data to be detected.

[0085] By introducing a quality threshold summary algorithm based on time intervals and combining it with an initial centroid set to cluster the data to be detected, a high-quality subset of data is obtained.

[0086] A primary compressed summary is obtained by performing conventional mode compression on a high-quality subset of data using the BUS algorithm to preserve data features with high gain and low loss.

[0087] The primary compressed digest is supplemented in a non-scale manner using a data aggregation algorithm to form a complete dual-channel compressed digest, thus obtaining the aggregated data.

[0088] Historical network traffic datasets are extracted, and threshold correlation algorithms, genetic algorithms, and particle swarm optimization algorithms are combined to extract a subset of abnormal features from the historical network traffic datasets. In the first stage, MLP, KNN, DT, and RF machine learning frameworks are used to analyze the extracted anomalous feature subsets to detect the type of network traffic data. When the type is identified as malicious traffic, the second stage is triggered. In the second stage, voting classifiers and CatBoost ensemble learning mechanisms are used to identify the nature of the attack. The classifiers from the first and second stages, which have the best detection performance, are integrated to construct a real-time detection framework. This combination not only reduces associated time-series overhead but also significantly enhances detection capabilities. When abnormal or attack traffic is detected, the attack traffic is stored in a distributed repository as training data for updating the classification model, and an alert is triggered. The classification model supports online updates.

[0089] The aggregated data is input into the real-time detection framework, and the type of aggregated data is output. When the type of aggregated data is malicious traffic, the attack nature of the aggregated data is also output. The type and attack nature of the aggregated data are attached to the aggregated data to form attack traffic.

[0090] Specifically, the initial centroid set of the data to be detected, obtained by initial clustering the data using the K-means algorithm, includes:

[0091] The K-Means algorithm defines the number of clusters by the parameter k and finds a partition that minimizes the sum of squared errors, assumed to be the distance between a data point and the centroid of the cluster it is assigned to. The algorithm initializes with k random points. All data points are then assigned to their nearest initial centroid, forming k clusters. The centroids of the clusters are recalculated until they converge. Clustering can be used as a summarizing method by making the cluster centroids a summary of the input dataset.

[0092] A test set is constructed based on the data to be tested. Data points with a specified number of clusters are randomly selected from the test set as centroids to construct a centroid set.

[0093] Calculate the distance from each data point in the set to be tested to all centroids and assign it to the cluster with the nearest centroid;

[0094] Update the centroid of each cluster until the change in the centroid is less than the change threshold, then stop updating and output the initial centroid set;

[0095] Updating the centroid of each cluster involves calculating the mean of the data points within each cluster as the new centroid of the cluster.

[0096] Specifically, the step of introducing a quality threshold summarization algorithm based on time intervals and combining it with an initial centroid set to cluster the data to be detected, thereby obtaining a high-quality data subset, includes:

[0097] The data to be tested is segmented according to time intervals and a subset to be tested is constructed. d = 1, 2, ..., n, where n represents the total number of subsets to be divided for testing;

[0098] Calculate the subset to be tested The distance between data point j and each centroid in the initial centroid set is calculated, and the centroids in the initial centroid set are sorted in descending order according to the distance.

[0099] When the distance to the first and second centroids is less than the quality threshold, data point j is assigned to the cluster of the first and second centroids.

[0100] Remove the subset to be tested Data point j in the middle;

[0101] When the subset to be tested When all data points are empty, count the data points within each cluster, calculate the mean of the data points within each cluster as the new centroid of the cluster, output the cluster centroid, and obtain a high-quality subset of data.

[0102] Specifically, the BUS algorithm is used to compress a high-quality subset of data using a conventional mode to preserve data features with high gain and low loss, obtaining a primary compressed digest. A data aggregation algorithm is then used to supplement the primary compressed digest in a non-scale manner, forming a complete dual-channel compressed digest. The resulting aggregated data includes:

[0103] Extract closed frequent itemsets from high-quality subsets of data to construct candidate sets. Candidate sets = closed frequent itemsets High-quality data subsets;

[0104] In practical applications, there is no need for manual determination; closed frequent itemsets can be generated directly using a dedicated algorithm.

[0105] CLOSET algorithm: Based on the FP-Growth idea, it utilizes closure expansion and pruning;

[0106] CHARM algorithm: Based on vertical data format (Tidsets), it quickly determines closure by intersection.

[0107] Candidates are selected from the candidate set, and the compression gain of the candidate is calculated. When the compression gain is greater than the gain threshold, the candidate is recorded in the gain candidate set. The information loss of the candidate in the gain candidate set is calculated. The candidate in the gain candidate set is sorted in descending order according to the information loss. The first candidate is selected as the best candidate and recorded in the current summary set and the candidate set is updated. Updating the candidate set includes deleting the best candidate until the number of summaries in the current summary set is equal to the preset number of summaries. The current summary set is output as the primary compressed summary.

[0108] For example: Input dataset D 2 and the final summary quantity I 2; Generate candidate set , = D 2 Closed set of frequent itemsets + D 2 ;

[0109] mingain = minimum (gain AND gain , where all gains ( C i , S c Candidates equal to or less than 1 will be ignored. c This is a summary of the current situation.

[0110] AND gain 1 AND gain mingain , The remaining candidates use gain ( C i , S c Sort them.

[0111] The candidate with the lowest return In the middle, return the candidate with the lowest loss. As the best candidate.

[0112] S 1 {Summaries in covered by In order to calculate C best Add to The current summary set is obtained from the middle. S 1; Output the current summary set as a primary compressed summary.

[0113] The BUS algorithm incrementally selects the best candidate from a candidate set by summarizing a quantity parameter. This minimizes information loss at each step to achieve a certain compression gain. In each iteration, a routine selection method is used to choose a candidate from the candidate set, which represents a set containing all closed frequent itemsets. The algorithm selects from the generated candidate content to achieve the maximum compression gain and minimum information loss. This continues until the current summarization quantity equals the summarization quantity, thus achieving the desired compression.

[0114] Extract closed frequent itemsets from high-quality subsets of data to construct candidate sets. Candidate sets = closed frequent itemsets High-quality data subsets;

[0115] Candidates are selected from the candidate set, and the compression gain of the candidate is calculated. When the compression gain is greater than the gain threshold, the candidate is recorded in the gain candidate set. The information loss of the candidate in the gain candidate set is calculated. The candidate in the gain candidate set is sorted in ascending order according to the information loss. The first candidate is selected as the best candidate and recorded in the current summary set and the candidate set is updated. Updating the candidate set includes deleting the best candidate until the number of summaries in the current summary set is equal to the preset number of summaries. The current summary set is output as the reverse compressed summary.

[0116] The data aggregation algorithm modifies the BUS algorithm for merging. The nature of the BUS algorithm makes its objective function easily modifiable. After generating closed itemsets, it searches for candidates that produce the maximum compression gain and minimum information loss. This is achieved by modifying the second step, instead of searching for the minimum information loss, to find the maximum value of the inverse information loss.

[0117] The primary compressed digest and the reverse compressed digest are merged to form a complete digest with dual-channel compression, thus obtaining the summary data.

[0118] Specifically, the construction of the real-time detection framework includes:

[0119] Extract the original feature set from the historical network traffic dataset, perform feature dimensionality reduction on the original feature set, and obtain the abnormal feature subset:

[0120] Calculate any two features in the original feature set f 1 and f The correlation coefficient between two features is calculated using the formula: the result varies between -1 and +1. If the value of the correlation coefficient between two features is close to +1 or -1, it means that there is a strong correlation between them. Conversely, if they are adjacent to zero, there is no relationship between the two features.

[0121]

[0122] in, x i , y i They represent f 1 and f 2. The i-th corresponding value of two features, , They represent f Calculate the average and 1 f 2 is the average value, and N represents the number of corresponding values ​​for the feature;

[0123] When the absolute value of the correlation coefficient is less than the coefficient threshold, any feature is selected as a record in the feature set after dimensionality reduction.

[0124] The reduced feature set is input into a genetic algorithm or particle swarm optimization algorithm to output the first anomaly-related feature for final screening.

[0125] Genetic algorithms or particle swarm optimization algorithms run directly on the original feature set and find a subset of key features that distinguish attacks from normal ones through global optimization.

[0126] Apply a threshold correlation algorithm to the feature subset to remove strongly correlated features and output a second anomaly-related feature that is correlated with the anomaly.

[0127] The first anomaly-related feature and the second anomaly-related feature are merged to form an anomaly feature subset;

[0128] First, the TC algorithm is applied to minimize the number of features transmitted by GA and PSO, thus reducing overall computation time. This is because the TC algorithm randomly selects a single feature from two strongly correlated features, which could lead to information loss. Therefore, in the latter two cases, GA and PSO are initially used to find the most informative features. Next, the TC algorithm filters out highly correlated features.

[0129] In the first stage, MLP, KNN, DT, and RF machine learning frameworks are built and trained respectively, taking anomaly feature subsets as input and outputting the type of network traffic data; in the second stage, voting classifier and CatBoost ensemble learning mechanisms are built and trained respectively to identify the nature of attacks.

[0130] In order to solve this type of nonlinear classification problem during the classifier selection process, a multi-model collaborative approach is adopted, which combines DT, KNN, MLP and RF to transform the feature space, fuse local decisions and optimize the global model. In stage 2, voting classifiers and CatBoost are used to increase model diversity and automatic feature processing, which significantly improves the classification performance in complex scenarios.

[0131] Feature samples with specific attack labels are extracted from the database as test sets and input into the first and second stages to calculate the type accuracy of MLP, KNN, DT and RF, as well as the property accuracy of voting classifier and CatBoost.

[0132] Based on the accuracy of the type, MLP, KNN, DT, and RF are sorted in descending order, and the first machine learning framework is selected as the machine learning framework for the first stage. Based on the accuracy of the property, the ensemble learning mechanism with the highest property accuracy is selected as the ensemble learning mechanism for the second stage, thus obtaining the real-time detection framework.

[0133] like Figure 5 The diagram illustrates an embodiment of the present invention, providing a structural diagram of a smart transportation integrated management platform based on a multi-level security control system. This platform includes an IoT smart terminal security layer, an IoT multi-level secure communication layer, a data control layer, a business service support layer, and a business application layer. The IoT smart terminal security layer integrates various real-time traffic access scenarios, including sensing devices, mobile terminals, IoT terminals, communication devices, smart vehicles, and industrial control systems. The IoT multi-level secure communication layer includes a distributed detection framework, communication protocols, and security auditing and monitoring. The data control layer includes data transmission, data storage, data disposal, and data recovery. The business service support layer includes application services, application access, application operation, technical support, terminal management, information management, and application management. The business application layer includes digital infrastructure management, scientific management of maintenance decisions, information management of road network operation, high-quality management of public transportation, collaborative management of emergency command, and intelligent vehicle-road collaboration.

[0134] The IoT smart terminal security layer includes existing front-end sensing devices and is equipped with various supporting infrastructures to ensure the normal operation of various sensing devices. It can also realize interactive functions such as device detection, signal control, and information dissemination.

[0135] The multi-layered secure communication layer of the Internet of Things (IoT) includes a distributed detection framework, communication protocols, and security auditing and monitoring. For example... Figure 6As shown, the distributed detection framework based on ensemble learning consists of five modules: a distributed acquisition module, a data cleaning module, an anomaly detection and classification module, a distributed storage module, and an alarm management module. When a large amount of real-time network traffic data is input into the security layer of the IoT smart terminal, the distributed acquisition module in the multi-level secure communication layer of the IoT uses a micro-batch processing mechanism to improve the throughput of the data collector and extracts key attributes from the data packets for repeated processing. The data cleaning module cleans the data based on the key attributes of the acquired data packets, including the calculation of statistical attributes of the data packets, TCP session reassembly, and the calculation of session statistical attributes. Finally, the cleaned data is used as the data source for the anomaly detection and classification module. The anomaly detection and classification module includes four sub-modules: data aggregation, time management, decision management, and anomaly detection. Among them, the data aggregation module sets quality thresholds based on quality threshold summarization algorithms, BUS algorithms, K-means algorithms, and data aggregation algorithms, optimizes and clusters the data, and then sends the aggregated data to the time information management module and the decision management module. The time information management module is responsible for allocating the time interval for the aggregated data. The decision management module monitors the entire system and has all control over the data aggregation and anomaly classification modules. The anomaly detection module combines threshold correlation algorithms with genetic algorithms and particle swarm optimization. In the first stage, MLP, KNN, DT, and RF machine learning frameworks are used to analyze extracted features to detect malicious traffic. The second stage identifies the nature of the attack with the help of two ensemble learning mechanisms: voting classifiers and CatBoost. The most suitable first-stage and second-stage classifiers are further integrated to build a real-time framework. The anomaly detection module is the core of the anomaly detection classification module. When abnormal or attack traffic is detected, it stores the attack traffic in a distributed repository as training data for updating the classification model, and simultaneously handles alarms. The classification model supports online updates. The alarm management module provides security analysts with query functions for anomaly alarms and specific attributes of abnormal traffic. The distributed storage module stores detected abnormal data packets, as well as training and testing data for the classification model. Communication protocols ensure smooth information transmission between devices, improving the overall system's operating efficiency and reliability. Security auditing and monitoring are responsible for storing and analyzing system operating status, security events, and alarm information. By persistently storing and visually analyzing logs, security administrators can understand the system's operational status and the occurrence of security incidents, thereby enabling them to adjust and optimize system configurations in a timely manner.

[0136] The data management layer connects to various traffic data sources to acquire real-time front-end sensing data, including basic road element data, video surveillance information, traffic control data, traffic guidance information, and traffic flow. By efficiently integrating, processing, and analyzing large-scale, multi-source data, it ensures the comprehensiveness and real-time nature of the information, providing a reliable foundation for subsequent analysis and applications. Encryption and access control measures ensure that traffic data is protected from unauthorized access and malicious attacks.

[0137] The business service support layer provides decision-makers with comprehensive traffic condition assessments through data analysis and visualization, helping them to formulate more scientific urban traffic planning and management strategies. It also provides functional implementation and enhancement services for upper-layer applications, enabling resource visualization and command and dispatch.

[0138] The business application layer is the top layer of the system. Its main task is to utilize the underlying data and services provided by the support layer to carry out various practical applications. For the full lifecycle management of smart highways, it constructs digital management of infrastructure, scientific management of maintenance decisions, information-based management of road network operation, high-quality management of public travel, collaborative management of emergency command, and intelligent management of vehicle-road cooperation.

[0139] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any other combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0141] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A network traffic analysis and intrusion detection method based on a multi-level security control system, characterized in that, include: Collect network traffic data packets and extract key attributes of the packets for caching; Clean the cached data packets to obtain the data to be tested; Perform anomaly detection on the data to be tested; The quality threshold is set by combining the quality threshold summarization algorithm, BUS algorithm, K-means algorithm, and data summarization algorithm. The data to be tested is then optimized and clustered to obtain the summary data. The K-means algorithm is used to perform initial clustering of the data to be detected, thereby obtaining the initial centroid set of the data to be detected. By introducing a quality threshold summary algorithm based on time intervals and combining it with an initial centroid set to cluster the data to be detected, a high-quality subset of data is obtained. The BUS algorithm is used to perform conventional mode compression on a high-quality subset of data to preserve data features with high gain and low loss, resulting in a preliminary compressed summary. This includes: extracting closed frequent itemsets from the high-quality subset of data and constructing a candidate set, where the candidate set = closed frequent itemsets. High-quality data subsets; Candidates are selected from the candidate set, and the compression gain of the candidate is calculated. When the compression gain is greater than the gain threshold, the candidate is recorded in the gain candidate set, and the information loss of the candidate in the gain candidate set is calculated. The candidates in the gain candidate set are sorted in descending order according to the information loss. The first candidate is selected as the best candidate and recorded in the current summary set and the candidate set is updated. The updating of the candidate set includes deleting the best candidate until the number of summaries in the current summary set is equal to the preset number of summaries. The current summary set is then output as the primary compressed summary. The primary compressed digest is supplemented in an unscaled manner by a data aggregation algorithm to form a complete dual-channel compressed digest. The aggregated data includes: sorting the candidates in the gain candidate set in ascending order according to information loss, selecting the first candidate as the best candidate to record in the current summary set and updating the candidate set. The updating of the candidate set includes deleting the best candidate until the number of summaries in the current summary set is equal to the preset number of summaries, and outputting the current summary set as the reverse compressed digest. The primary compressed digest and the reverse compressed digest are merged to form a complete dual-channel compressed digest, thus obtaining the summary data. Historical network traffic datasets are extracted, and threshold correlation algorithms, genetic algorithms, and particle swarm optimization algorithms are combined to extract a subset of abnormal features from the historical network traffic datasets. In the first stage, MLP, KNN, DT, and RF machine learning frameworks are used to analyze the extracted anomalous feature subsets to detect the type of network traffic data. When the type is malicious traffic, the second stage is triggered. In the second stage, voting classifiers and CatBoost ensemble learning mechanisms are used to identify the nature of the attack. The classifiers corresponding to the first and second stages with the best detection performance are integrated to construct a real-time detection framework. The aggregated data is input into the real-time detection framework, and the type of aggregated data is output. When the type of aggregated data is malicious traffic, the attack nature of the aggregated data is also output. The type and attack nature of the aggregated data are attached to the aggregated data to form attack traffic. When attack traffic is detected, it is stored in the database and an alarm is triggered.

2. The network traffic analysis and intrusion detection method based on a multi-level security management system according to claim 1, characterized in that, The process of collecting network traffic data packets and extracting key attributes of the packets for caching includes: When a large amount of real-time network traffic data is input into the security layer of the IoT smart terminal in the multi-level security management system, the network traffic data is collected through a micro-batch processing mechanism. The network traffic data of the time interval is accumulated on the main thread according to a preset time interval to obtain the initial data packets. By using preset necessary fields and real-time calculated traffic entropy values, key attributes in the data packets are extracted as data packet header attributes, and the data packets are cached to obtain cached data packets.

3. The network traffic analysis and intrusion detection method based on a multi-level security management system according to claim 1, characterized in that, The process of cleaning the cached data packets to obtain the data to be detected includes: Establish a queue connection and listen for cached data packets; The cached data packets are read in batches according to the time window. A window calculation is triggered once every time interval. The attributes of the cached data packets are statistically analyzed based on the attributes of the data packet header. The attributes of the cached data packets include data packet-level attributes and session-level attributes. Within a given time interval, the number of data packets with the same destination IP as the current packet is counted and used as an attribute of the data packet dimension. By reassembling TCP sessions in cached data packets, and counting the data packets of the same TCP session during the reassembly process, the number of sessions with the same destination IP as the current session within a time interval is counted as an attribute of the statistical session dimension. The various attributes are added to the packet header attributes, and cached packets with the same packet header attributes as the current packet are removed to obtain the data to be detected.

4. The network traffic analysis and intrusion detection method based on a multi-level security management system according to claim 1, characterized in that, The initial centroid set of the data to be detected, obtained by initial clustering using the K-means algorithm, includes: A test set is constructed based on the data to be tested. Data points with a specified number of clusters are randomly selected from the test set as centroids to construct a centroid set. Calculate the distance from each data point in the set to be tested to all centroids and assign it to the cluster with the nearest centroid; Update the centroid of each cluster until the change in the centroid is less than the change threshold, then stop updating and output the initial centroid set; Updating the centroid of each cluster involves calculating the mean of the data points within each cluster as the new centroid of the cluster.

5. The network traffic analysis and intrusion detection method based on a multi-level security management system according to claim 1, characterized in that, The method of introducing a quality threshold summary algorithm based on time intervals and clustering the data to be detected using an initial centroid set to obtain a high-quality data subset includes: The data to be tested is segmented according to time intervals and a subset to be tested is constructed. d = 1, 2, ..., n, where n represents the total number of subsets to be divided for testing; Calculate the subset to be tested The distance between data point j and each centroid in the initial centroid set is calculated, and the centroids in the initial centroid set are sorted in descending order according to the distance. When the distance to the first and second centroids is less than the quality threshold, data point j is assigned to the cluster of the first and second centroids. Remove the subset to be tested Data point j in the middle; When the subset to be tested When all data points are empty, count the data points within each cluster, calculate the mean of the data points within each cluster as the new centroid of the cluster, output the cluster centroid, and obtain a high-quality subset of data.

6. The network traffic analysis and intrusion detection method based on a multi-level security control system according to claim 1, characterized in that, The construction of the real-time detection framework includes: Extract the original feature set from the historical network traffic dataset, perform feature dimensionality reduction on the original feature set, and obtain the abnormal feature subset: Apply a threshold correlation algorithm to the original feature set to calculate the threshold correlation between any two features in the original feature set. f 1 and f The correlation coefficient of 2, calculated using the following formula: ； in, x i , y i They represent f 1 and f 2. The i-th corresponding value of two features, , They represent f Calculate the average and 1 f 2 is the average value, and N represents the number of corresponding values ​​for the feature; When the absolute value of the correlation coefficient is less than the coefficient threshold, any feature is selected as a record in the feature set after dimensionality reduction. The reduced feature set is input into a genetic algorithm or particle swarm optimization algorithm to output the first anomaly-related feature for final screening. Genetic algorithms or particle swarm optimization algorithms run directly on the original feature set and find a subset of key features that distinguish attacks from normal ones through global optimization. Apply a threshold correlation algorithm to the feature subset to remove strongly correlated features and output a second anomaly-related feature that is correlated with the anomaly. The first anomaly-related feature and the second anomaly-related feature are merged to form an anomaly feature subset; In the first stage, MLP, KNN, DT, and RF machine learning frameworks are built and trained respectively, taking anomaly feature subsets as input and outputting the type of network traffic data; in the second stage, voting classifier and CatBoost ensemble learning mechanisms are built and trained respectively to identify the nature of attacks. Feature samples with specific attack labels are extracted from the database as test sets and input into the first and second stages to calculate the type accuracy of MLP, KNN, DT and RF, as well as the property accuracy of voting classifier and CatBoost. Based on the accuracy of the type, MLP, KNN, DT, and RF are sorted in descending order, and the first machine learning framework is selected as the machine learning framework for the first stage. Based on the accuracy of the property, the ensemble learning mechanism with the highest property accuracy is selected as the ensemble learning mechanism for the second stage, thus obtaining the real-time detection framework.

Images