Bad traffic monitoring method and system based on LSTM and clustering algorithm

Abstract

The present invention discloses a bad traffic monitoring method and system based on LSTM and clustering algorithm, which belongs to the field of network security technology, and utilizes BPF technology to efficiently aggregate network traffic; accurately identifies fragmented traffic through LSTM network, and uses its special memory unit structure to capture complex dependencies; adopts FCM clustering algorithm to reorganize the session, restore the complete network session, and quickly identify malicious traffic through feature matching; finally, timely disposal measures are taken for confirmed malicious traffic. The present invention realizes the rapid and accurate detection and identification of bad website traffic fragmentation behavior, and solves the technical problems that traditional methods are difficult to deal with; through efficient caching and reorganization mechanism, the efficiency and accuracy of malicious traffic processing are improved; intelligent disposal measures are adopted to ensure timely and effective containment of malicious traffic; the overall protection capability of network security is improved, and the security of enterprise network resources and user data is protected.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to a bad traffic monitoring method and system based on LSTM and clustering algorithm. Background Art

[0002] With the rapid development of Internet technology, website security issues have become increasingly prominent and have become an important issue that needs to be resolved urgently. Conventional malicious behavior traffic detection mainly relies on content restoration or behavioral feature detection of traffic response packets (such as HTTP POST), and restoration or action detection of request content of traffic request packets (such as HTTP GET). However, in the design of the TCP protocol, the maximum limit of a single TCP payload packet is 1480 bytes, while the content of the HTTP GET packet is usually small, often not exceeding 500 bytes, so traffic fragmentation rarely occurs. This makes the previous detection methods appear stretched when facing fragmentation behavior, unable to effectively identify and deal with malicious traffic in fragments, and unable to perform effective detection and disposal.

[0003] The Chinese invention patent with application number CN201811536096.8 discloses a "method for detecting abnormal network traffic", including: step 1) collecting known network traffic and preprocessing it to obtain preprocessed network traffic; step 2) establishing an LSTM neural network model; using the preprocessed network traffic obtained in step 1) as input data, training the LSTM neural network model, and obtaining a trained LSTM neural network model; step 3) capturing unknown network traffic and preprocessing the unknown network traffic to obtain preprocessed unknown network traffic; step 4) using the preprocessed unknown network traffic obtained in step 3) as input data, inputting it into the trained LSTM neural network model obtained in step 2), identifying the preprocessed unknown network traffic, and outputting the identification result; step 5) responding according to the identification result obtained in step 4); if the unknown network traffic is identified as intrusion traffic, the network with the intrusion traffic is cut off. Long Short-Term Memory (LSTM) is a variant of Recurrent Neural Network (RNN), which aims to solve the gradient vanishing and gradient exploding problems of traditional RNN when processing long sequences. LSTM introduces a special memory unit and gating mechanism to more effectively capture and process long-term dependencies in sequence data.

[0004] LSTM (Long Short-Term Memory) has several major problems or flaws, including high computational complexity, difficulty in interpretation, and the need for a large amount of data for training.

[0005] 1) High computational complexity: Compared with traditional RNN (recurrent neural network), LSTM has higher computational complexity. This is because LSTM introduces gating mechanism and long-term memory mechanism, which requires more parameters and computation, making the training process more complicated and time-consuming.

[0006] 2) Difficult to explain: The complexity of LSTM makes its internal operating mechanism less intuitive and it is difficult to explain the decision-making process of the network.

[0007] 3) Requires a lot of data for training: LSTM has more parameters to train, so it needs more data to avoid overfitting. If the training data is insufficient, LSTM may face the problem of insufficient generalization ability.

[0008] The existence of the above problems requires comprehensive consideration of the advantages and disadvantages of LSTM when applying it, as well as whether it is suitable for specific application scenarios and data conditions.

[0009] The current traffic slicing mechanism is used by criminals in illegal fields. These black and gray traffic slicing behaviors are mixed with normal traffic slicing, which brings great challenges to detection and disposal, and further increases the difficulty of detection and disposal. This attack method cleverly mixes normal traffic with malicious traffic, making it difficult to effectively identify and distinguish normal traffic, which may lead to serious consequences such as significant degradation of website performance and leakage of sensitive data.

[0010] Traditional security protection measures often fail to provide adequate protection against this new type of attack, resulting in serious threats to website security. Traditional security protection measures are powerless against this new type of attack and are unable to provide effective protection, so a new detection and disposal system and method needs to be developed. Summary of the invention

[0011] The technical problem solved by the present invention is to provide a bad traffic monitoring method and system based on LSTM and clustering algorithm, which can realize efficient identification and accurate disposal of fragmented traffic by combining the linkage mechanism of BPF and LSTM.

[0012] Technical solution: In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0013] A bad traffic monitoring method based on LSTM and clustering algorithm includes the following steps:

[0014] S1: Traffic aggregation and screening based on BPF: collect and analyze website traffic data in real time on the network server or proxy server, classify, count and monitor network traffic, and promptly detect and handle abnormal traffic;

[0015] S2: Sharding behavior recognition and traffic caching based on LSTM;

[0016] S3: Use the FCM algorithm to reorganize the session and match the features to restore the complete network session;

[0017] S4: Identify and handle malicious traffic through feature matching and automated handling technology;

[0018] Use predictive blocking technology based on probability models to accurately block and intercept malicious website traffic; use rule engine technology to automate the processing and implementation of disposal measures.

[0019] Furthermore, in step S1, the method for implementing traffic aggregation and screening based on BPF is:

[0020] Let R be a set of filtering rules, where each rule r∈R is a logical expression used to determine whether a packet p satisfies the rule;

[0021] For each arriving data packet p, traverse the filter rule set R, and if the data packet p satisfies rule r, it is captured;

[0022] Let G be a set of packet groups, where each group g∈G corresponds to a specific attribute; for each captured packet p, add it to the corresponding group g according to its attribute;

[0023] For each group g, define the filtering condition S(g); if the group g satisfies the filtering condition S(g), then output the relevant information of the group;

[0024] R={r1, r2, …, rn}: filtering rule set;

[0025] P={p1, p2, …, pm}: data packet set;

[0026] G = {g1, g2, ..., gk}: data packet grouping set;

[0027] S(g): Packet screening condition, for each data packet pi∈P:

[0028] Traverse the filter rule set R, if If rj(pi)=true, the data packet pi is captured;

[0029] According to the attributes of the data packet pi, add it to the corresponding group gl∈G;

[0030] For each group gl∈G: If S(gl)=true, output relevant information of gl.

[0031] Furthermore, in step S2, with the assistance of LSTM, the traffic fragmentation behavior is accurately identified. LSTM captures the complex dependencies and long-term dependencies in the sequence through its memory unit structure. When a specific marker MF≠0 appears in the traffic, the data packets under the same session will begin to be processed by LSTM to determine which new information will be stored in the memory unit and which information will be forgotten or output. Finally, the identified fragmented traffic will be cached. The specific implementation process is as follows:

[0032] S21: Data preprocessing:

[0033] Preprocess the captured fragmented data packets, including removing unnecessary header information, checking and verifying, etc., to reduce the amount of data to be processed later, and extract the key features of the fragmented data packets, including sequence number, offset, and data length. The key features will be used as the input of the LSTM network;

[0034] S22: Sequence construction:

[0035] Arrange the fragmented data packets under the same session in the order of sequence numbers to form a time series; extract the feature vector of each data packet and use it as a time step input of the LSTM network;

[0036] S23: LSTM forward propagation:

[0037] The feature vector is input into the LSTM network, which processes the input data through its memory cells and three gating mechanisms. At each time step, the LSTM network calculates a new hidden state and cell state based on the output of the previous time step and the input of the current time step. The hidden state contains a summary of all the input information so far, while the cell state stores long-term dependency information.

[0038] S24: Output processing:

[0039] The final output of the LSTM network is a classification result: normal shard or malicious shard, or a probability value indicating whether the shard sequence is abnormal; based on the output result, it is decided whether to cache the current shard sequence for subsequent processing, or whether to immediately trigger the exception handling mechanism.

[0040] Furthermore, in step 3, the implementation process of using the FCM algorithm for session reorganization and feature matching is as follows:

[0041] S31: Behavior data collection: collect users' online behavior data, including access time, access duration, access frequency and access content type;

[0042] S32: Feature construction: constructing user behavior feature vector based on behavior data;

[0043] S33: FCM clustering: Use the FCM algorithm to cluster user behavior feature vectors and group users with similar behavior patterns into one category;

[0044] S34: Abnormal behavior identification: Analyze the clustering results and identify abnormal clusters that are significantly different from normal user behavior patterns.

[0045] Furthermore, in step S33, the FCM algorithm is used to cluster the user behavior feature vectors, and users with similar behavior patterns are grouped into one category. The specific implementation process is as follows:

[0046] S331: Data preprocessing and feature extraction

[0047] In the session reassembly phase, the fragmented traffic in the cache needs to be reassembled to restore the complete network session. Then, key features are extracted from the reassembled session, including domain name, IP address, port number, URL path, and request method.

[0048] S332: Initialization

[0049] Determine the number of clusters c, and randomly initialize c cluster centers v1, v2, ..., v c ;

[0050] Set the initial value of the membership matrix U to a matrix whose elements are one-tenth of c, representing all clusters to which each data point may initially belong; set the iteration stop conditions, including the maximum number of iterations or the objective function change being less than a certain threshold;

[0051] S333: Iterative Optimization

[0052] For each iteration, the following steps are performed:

[0053] Update the membership matrix U: For each data point x j And each cluster center v i , update the membership u ij , the formula is as follows:

[0054]

[0055] Where m represents the fuzzification parameter; x j represents the jth data point, which is a behavioral feature vector containing the eigenvalues ​​of each data point; v i represents the i-th cluster center, which is also a vector, indicating the center position of the cluster; v k Represents the kth cluster center (k≠i), which is used as a comparison object when calculating membership; represents the jth data point x j To cluster center v iThe Euclidean distance of represents the jth data point x j To the kth cluster center v k The Euclidean distance of

[0056] Update cluster center V:

[0057] For each cluster center v i , update its position according to the membership matrix, the cluster center feature vector v i The calculation formula is:

[0058]

[0059] Among them, v i represents the i-th cluster center, x j represents the behavioral feature vector of the jth data point, u ij represents the membership of the jth data point to the i-th cluster, n is the total number of users, Indicates u ij m to the power of;

[0060] S334: Convergence judgment

[0061] Check the change of the objective function or whether the maximum number of iterations has been reached. If the stopping condition is met, the iteration ends.

[0062] Furthermore, in step S3, decoupled feature library association is also used in session reorganization and feature matching, and the specific contents include:

[0063] (1) Feature library modularization

[0064] First, different types of feature libraries are set as independent modules; each module contains a specific feature set and the matching logic associated with it;

[0065] Fraud signature library module: contains domain names, IP addresses and URL pattern features related to fraud activities;

[0066] Network attack signature library module: contains the signatures of SQL injection and cross-site scripting network attacks;

[0067] Gambling feature library module: contains domain names and URL path features related to gambling websites;

[0068] (2) Standardization of feature library interface

[0069] Define unified interface standards for interaction between each feature library module and other parts of the system; these interfaces include feature loading, feature matching, and feature update functions;

[0070] Loading interface: used to load feature data from feature library files.

[0071] Matching interface: Matches the provided network traffic features in the feature library and returns the matching results.

[0072] Update interface: used to load new features into the system when the feature library is updated;

[0073] (3) Feature matching process integration

[0074] In the session reorganization and feature matching stage, the decoupled feature library association process is integrated into the system. The specific steps are as follows (31)~(34):

[0075] (31) After the conversation is reorganized, extract the key features of the conversation;

[0076] (32) According to the feature type, the matching interface of the corresponding feature library module is called; if the extracted features contain URLs, the matching interfaces of the fraud feature library module and the network attack feature library module are called for matching respectively.

[0077] (33) Collect the matching results of each feature library module and make a comprehensive judgment; if a session has matching items in multiple feature libraries, it may mean that the session involves multiple types of malicious behaviors;

[0078] (34) Based on the matching results, execute corresponding disposal measures, including marking the session, blocking the connection and / or sending a warning;

[0079] (4) Feature library configuration and management

[0080] Configuration management tools allow system administrators to configure the use of signature libraries according to actual needs;

[0081] Enable / disable signature library: The administrator can enable or disable specific signature library modules according to the current security threat situation;

[0082] Priority setting: Set priorities for different signature library modules so that when multiple signature libraries have matching items, they will be processed in order of priority;

[0083] Dynamic update: Supports dynamic update of feature libraries. When new malicious traffic features are discovered, these features are added to the corresponding feature libraries in a timely manner and loaded into the system through the update interface.

[0084] (5) System monitoring and feedback

[0085] Monitor the usage of the feature library, collect key indicators such as matching success rate and false alarm rate, and optimize and adjust the feature library based on these indicators;

[0086] Performance monitoring: monitor the performance of feature matching to ensure that the system can maintain stable operation under high load conditions;

[0087] Result feedback: The matching results and the effects of the disposal measures are fed back to the feature library maintenance team so as to optimize the quality and accuracy of the feature library based on the feedback.

[0088] Furthermore, in step S4, the specific process of the predictive blocking technology based on the probability model is as follows:

[0089] S41: Building a probabilistic prediction model

[0090] 1) Collect historical traffic data, including normal traffic and known malicious traffic;

[0091] 2) Extract features from these traffic data to extract packet size, access frequency, session duration, and specific byte patterns;

[0092] 3) Use machine learning algorithms to train probabilistic prediction models to distinguish between normal and malicious traffic;

[0093] S42: Real-time traffic prediction

[0094] Perform the same feature extraction on the current real-time traffic data; input the extracted features into the trained probability prediction model to obtain the probability value of the current traffic being malicious traffic;

[0095] S43: Threshold determination and ban

[0096] Set a threshold to determine whether the traffic is malicious. The threshold is adjusted according to the actual application scenario and accuracy requirements. If the probability value exceeds the set threshold, the traffic is considered malicious and is immediately blocked. The blocking process includes blocking the connection, adding to the blacklist, and sending a warning message.

[0097] S44: Model update and optimization

[0098] Regularly collect new traffic data, including normal traffic and newly emerging malicious traffic; use new data to update and retrain the probability prediction model to improve the accuracy and adaptability of the model; adjust model parameters and thresholds based on actual application results to optimize the blocking effect.

[0099] Furthermore, in step S4, the specific process of realizing the automated processing and implementation of disposal measures by using rule engine technology is as follows:

[0100] 1) Define the rules

[0101] Define a series of clear handling rules based on business needs and security policies. These rules describe the handling measures to be taken in different situations, including blocking access and sending warnings.

[0102] 2) Rule engine configuration:

[0103] Select and configure the rule engine system, which has the ability to handle complex logic, respond quickly and execute efficiently; import the defined disposal rules into the rule engine and make necessary configurations to ensure that the rules can be triggered and executed correctly;

[0104] 3) Data input and processing

[0105] Collect relevant event or behavior data from monitoring systems, log records, and sensor data sources in real time or periodically; pre-process the collected data to ensure data accuracy and consistency;

[0106] 4) Rule matching:

[0107] The rule engine matches the input data according to the imported rules; it involves conditional judgment and pattern recognition of the data; if the data meets the triggering conditions of a certain rule, the rule engine will trigger the corresponding disposal measures;

[0108] 5) Implementation of disposal measures:

[0109] According to the triggered rules, the rule engine automatically executes corresponding disposal measures, including sending warning messages and blocking network access processes;

[0110] 6) Monitoring and feedback:

[0111] Monitor the execution process of the rule engine in real time to ensure that the disposal measures are correctly executed; collect feedback information after the disposal measures are executed to make necessary adjustments and optimizations to the rules;

[0112] 7) Rule optimization and update: Regularly review and optimize the rules based on monitoring results and feedback information; modify, add or delete rules.

[0113] The present invention also discloses a bad traffic monitoring system based on LSTM and clustering algorithm, including a traffic aggregation module, a sharding behavior recognition and traffic caching module, a session reorganization and feature matching module and a disposal execution module;

[0114] The traffic aggregation module collects website traffic data on the network server or proxy server, aggregates and analyzes the traffic data in real time using BPF, and provides a capability foundation for traffic segmentation and malicious feature behavior detection;

[0115] The fragmentation behavior identification and traffic caching module: uses LSTM to filter out traffic with fragments according to the fragmentation identifier in TCP transmission and put it into the cache;

[0116] The session reorganization and feature matching module: uses the FCM algorithm to reorganize the sharded sessions in the cache, and extracts the domain name, IP, port, and URL features, collides with the front-end feature library, and sends execution information to the execution unit when it is confirmed to meet the conditions;

[0117] The processing execution module starts processing execution after receiving the object information and the execution action to be executed.

[0118] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0119] After adopting the technical solution of the present invention, it can significantly solve the problem of detecting and handling bad website traffic fragmentation behavior faced by the current network security field, and bring a series of significant technical effects, thereby completing the closed loop of the invention logic.

[0120] First, the present invention realizes efficient identification and separation of fragmented traffic by integrating the linkage mechanism of BPF and LSTM. BPF technology ensures the real-time convergence and monitoring of massive traffic, providing a solid foundation for subsequent in-depth analysis. The LSTM model, with its unique memory unit structure, can accurately capture complex dependencies and long-term dependencies in the sequence, significantly improving the accuracy of fragmented behavior identification. This innovation directly solves the technical bottleneck that traditional methods are difficult to effectively identify and distinguish fragmented traffic, making it possible to detect malicious fragmented traffic.

[0121] Secondly, after identifying malicious fragmented traffic, the present invention adopts an efficient caching mechanism for storage, and combines pattern recognition and machine learning technology to achieve rapid identification and caching of malicious traffic. This step not only improves processing efficiency, but also provides a reliable data source for subsequent session reorganization and feature matching. Through session reorganization, the present invention can restore the complete network session, extract key malicious features, and quickly match them with a known malicious traffic feature library, thereby achieving accurate identification of malicious behavior.

[0122] In the disposal execution link, the present invention designs a predictive blocking technology based on a probability model and an automated disposal technology based on a rule engine to ensure timely and effective disposal of malicious traffic. These measures can not only effectively curb the spread and diffusion of malicious traffic, but also promptly repair the affected systems and protect the security of the target website. At the same time, through real-time monitoring and evaluation of disposal results, the present invention ensures the effectiveness and timeliness of disposal measures, and further enhances the overall protection capability of network security.

[0123] To sum up, the present invention firstly realizes the rapid and accurate detection and identification of bad website traffic fragmentation behavior, and solves the technical problems that are difficult to deal with by traditional methods; secondly, through efficient caching and reorganization mechanisms, the efficiency and accuracy of malicious traffic processing are improved; thirdly, intelligent disposal measures are adopted to ensure the timely and effective containment of malicious traffic; fourthly, the overall protection capability of network security is improved, and the security of enterprise network resources and user data is protected. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] Figure 1 is a schematic diagram of a network topology structure of Embodiment 1 of the present invention;

[0125] Figure 2 This is a flow chart for detecting and handling malicious traffic fragmentation behavior in Example 3.

[0126] Figure 3 This is the multi-environment traffic designated behavior detection and handling flow chart of Example 3.

[0127] Figure 4 This is a data processing flow chart of Example 3.

[0128] Figure 5 This is the LSTM basic flow chart of Example 3.

[0129] Figure 6 This is a schematic diagram of FCM session reorganization in Example 3.

[0130] Figure 7 This is a flow chart of the predictive ban technology based on probability model in Example 3.

[0131] Figure 8 This is a detailed step diagram of the predictive blocking technology based on the probability model of Example 3.

[0132] Fig. 9 It is a flowchart of the automated processing and implementation using rule engine technology in Example 3. DETAILED DESCRIPTION

[0133] The present invention is further illustrated below in conjunction with specific examples. The examples are implemented based on the technical solutions of the present invention. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0134] Example 1

[0135] like Figure 1As shown, in this embodiment, the bad traffic monitoring device based on LSTM and clustering algorithm includes a convergence and diversion device, a dedicated collector or server, a disposal platform and a switch, etc. Among them, the convergence and diversion device is responsible for traffic convergence and BPF processing; the dedicated collector or server is responsible for real-time collection and analysis of website traffic data (LSTM and FCM execution), the disposal platform is used to execute disposal actions, and the switch is used to complete the necessary network networking.

[0136] Example 2

[0137] This embodiment discloses a bad traffic monitoring system based on LSTM and clustering algorithm, using Linux and other operating systems to provide a stable server operating environment, using a database to store traffic data, sharding behavior recognition results, malicious traffic feature library, etc.; using machine learning frameworks such as LSTM, FCM, and TensorFlow to train and run models. Four modules, including traffic aggregation module, sharding behavior recognition and traffic caching module, session reorganization and feature matching module, and disposal execution module, are used to realize the functions of the present invention.

[0138] Traffic aggregation module: By collecting website traffic data on the network server or proxy server, BPF is used to aggregate and analyze the traffic data in real time, providing a capability foundation for traffic segmentation and malicious feature behavior detection.

[0139] Fragmentation behavior identification and traffic caching module: With the help of LSTM, fragmented traffic is filtered out according to the fragmentation identifier in TCP transmission and put into the cache.

[0140] Session reorganization and feature matching module: Use the FCM algorithm to reorganize the sharded sessions in the cache, and extract features such as domain name, IP, port, URL, etc., collide with the previous feature library, and send execution information to the execution unit if it is confirmed that the conditions are met.

[0141] Disposal execution module: starts the disposal execution after receiving the object information that needs to be executed and the execution action.

[0142] Example 3

[0143] like Figure 2 As shown, this embodiment discloses a bad traffic monitoring method based on LSTM and clustering algorithm. First, the packet filter BPF is used to efficiently aggregate and monitor massive traffic in real time; secondly, the traffic fragmentation behavior is accurately identified through the LSTM network to capture the complex dependencies and long-term dependencies in the sequence, thereby improving the accuracy of identification; then, the identified fragmented traffic is cached and the session is reorganized to restore the complete network session; finally, through feature matching and automatic disposal, malicious traffic is quickly and accurately identified and disposed of to ensure website security.

[0144] The present invention separates the fragmented traffic from the ordinary traffic; further distinguishes the unconventional fragments from the normal fragments; then restores and reassembles the fragmented packets, and then conducts in-depth analysis and identification of the reassembled data; and performs corresponding disposal according to the analysis results. The technology of the present invention provides a new and powerful guarantee for website security, and effectively responds to the threats brought by bad website traffic fragmentation behavior; specifically includes steps S1 to S4:

[0145] S1: Traffic aggregation and screening based on BPF: collect and analyze website traffic data in real time on the network server or proxy server, classify, count and monitor network traffic, and promptly detect and handle abnormal traffic;

[0146] S2: Sharding behavior recognition and traffic caching based on LSTM:

[0147] S3: Use the FCM algorithm to perform session reorganization and feature matching to restore the complete network session;

[0148] S4: Quickly and accurately identify and handle malicious traffic through feature matching and automated handling technology.

[0149] This embodiment records the detailed implementation process of the above steps in detail:

[0150] S1: Traffic aggregation and filtering based on BPF

[0151] Collect and analyze website traffic data in real time on the network server or proxy server to achieve efficient aggregation of massive traffic.

[0152] BPF (Berkeley Packet Filter) is an efficient packet filter that allows user space programs to filter packets received by the network interface. BPF consists of a network tap and a data filter, which can provide flexible and efficient packet capture and filtering functions. The specific operation process in the system is:

[0153] Let R be a set of filtering rules, where each rule r∈R is a logical expression used to determine whether a packet p satisfies the rule; for each arriving packet p, traverse the filtering rule set R, and if p satisfies rule r, it is captured.

[0154] Let G be a set of data packet groups (aggregations), where each group g∈G corresponds to a specific attribute (such as the source IP address); for each captured data packet p, add it to the corresponding group g according to its attributes; for each group g, define the filtering condition S(g) (such as the number of data packets in the group, the total number of bytes, etc.), if the group g meets the filtering condition S(g), then output the relevant information of the group.

[0155] R={r1, r2, …, rn}: filtering rule set;

[0156] P={p1, p2, …, pm}: data packet set;

[0157] G = {g1, g2, ..., gk}: data packet grouping set;

[0158] S(g): Packet screening condition, for each data packet pi∈P:

[0159] Traverse the filter rule set R, if If rj(pi)=true, the data packet pi is captured;

[0160] According to the attributes of the data packet pi, add it to the corresponding group gl∈G;

[0161] For each group gl∈G: If S(gl)=true, output relevant information of group gl.

[0162] The above process is expressed mathematically as follows:

[0163] Input condition discrimination: For each c∈C, there is a discriminant function Dc(p)=true or false.

[0164] Capturing fragmentation behavior: If c∈C such that Dc(p)=true and F(p) indicates fragmentation, then capture p.

[0165] Identify the MF≠0 flag: For a captured packet p, if MF(p)≠0, the condition is satisfied.

[0166] The specific process of using BPF to perform traffic aggregation and filtering in the present invention is described as S101 to S105:

[0167] S101: Packet Capture and Preprocessing

[0168] Use BPF's network tap function to capture packets on the network interface. The efficiency of BPF ensures low latency and high throughput packet capture. Preprocess the captured packets, including removing unnecessary header information, checksum verification, etc., to reduce the amount of data to be processed later.

[0169] S102: Intelligent filtering and classification

[0170] This embodiment applies a preset set of filtering rules, which are defined based on multiple features such as source / destination IP addresses, port numbers, and protocol types, and uses BPF data filters to filter data packets. Intelligent classification is performed on the filtered data packets. Efficient data structures such as hash tables, Bloom filters, or tree structures (such as red-black trees, prefix trees, Trie, etc.) are used to classify data packets according to specific fields, ensuring that each classification can be quickly located.

[0171] S103: Traffic Statistics and Aggregation

[0172] Traffic statistics are collected for each classified data packet, including key indicators such as the number of packets and the number of bytes. Aggregation algorithms such as sliding windows and exponential moving average (EMA) are used to smooth the traffic data and reduce the impact of short-term fluctuations on the analysis results.

[0173] The traffic aggregation algorithm used in this embodiment is the exponential moving average (EMA): EMA is a weighted moving average method that gives higher weights to recent data points; the calculation formula is:

[0174] ,

[0175] Among them, EMA t is the EMA value at the current time point, F t is the actual flow value at the current time point,

[0176] is the EMA value at the previous time point, α is the smoothing factor (0<α<1). By adjusting the value of α, the sensitivity of EMA to current traffic and historical traffic can be controlled.

[0177] S104: Abnormal detection and alarm

[0178] Applications such as threshold detection, statistical anomaly detection, and machine learning models are used to detect anomalies in traffic data. When abnormal traffic is detected, an alarm mechanism is triggered to notify the administrator via logs, emails, and text messages.

[0179] S105: Data storage and query optimization

[0180] Sampling efficient database architecture, such as time series database (TSDB), is used to store aggregated traffic data and analysis results. Implement fast query interface to support multi-dimensional and multi-time granularity traffic data query and analysis.

[0181] In addition, in addition to the above-mentioned BPF for traffic aggregation and screening, this embodiment also combines deep learning to efficiently process and accurately identify complex network traffic, providing a solid foundation for subsequent traffic segmentation and malicious feature behavior detection. The specific process is as follows S111~S114:

[0182] S111: Data collection and preprocessing

[0183] Data collection: First, a large amount of image data needs to be collected as training sets and test sets. These data should cover all categories that need to be classified.

[0184] Data preprocessing: including steps such as image resizing, normalization, data augmentation (such as rotation, scaling, flipping, etc.) to improve the generalization ability and training efficiency of the model.

[0185] S112: Building a Model

[0186] Select model architecture: Select a suitable CNN architecture according to task requirements, such as AlexNet, VGG, ResNet, etc. These architectures usually include basic components such as convolutional layers, pooling layers, and fully connected layers.

[0187] Design the network structure: determine the number of network layers, convolution kernel size, step size, padding method and other parameters. For example, you can design a network that contains multiple convolutional layers and pooling layers, and finally output the classification results through a fully connected layer.

[0188] S113: Training Model

[0189] Initialize Weights and Biases: Randomly initialize the weights and biases in the network.

[0190] Forward propagation: The input image passes through the convolutional layer, pooling layer and fully connected layer to obtain the output of the network.

[0191] Calculate the loss function: Calculate the difference between the output category and the actual category using a cross entropy loss function, for example.

[0192] Backpropagation: Calculate the gradient of the loss function and update the weights and biases using the gradient descent algorithm.

[0193] Iterative training: Repeat the process of forward propagation, loss function calculation, and backpropagation until the model converges or reaches the preset number of iterations.

[0194] S114: Testing and Evaluation

[0195] Test model: Use the test set to evaluate the performance of the model and calculate indicators such as accuracy, precision, and recall.

[0196] Adjust the model: Adjust the model parameters or structure according to the test results to improve the model performance. The algorithms involved take the convolution layer and pooling layer in the convolutional neural network as an example:

[0197] y_{ij} = \sum_{k=0}^{K-1} \sum_{l=0}^{L-1} x_{kl} \cdot w_{ij,kl} +b_i \];

[0198] Among them, \(x_{kl} \) is part of the input, \(w_{ij,kl} \) is part of the convolution kernel (filter), \(b_i \) is the bias term, and \(y_{ij} \) is the output after the convolution operation.

[0199] Commonly used pooling operations include maximum pooling and average pooling. The mathematical expression of maximum pooling is:

[0200] [y_i = \max(x_{i1}, x_{i2}, \dots, x_{in}) \]

[0201] The mathematical expression of average pooling is:

[0202] [ y_i = \frac{1}{n} \sum_{j=1}^{n} x_{ij} \]

[0203] Among them, \(x_{ij} \) is the pixel value within the pooling window, and \(y_i \) is the output after the pooling operation.

[0204] In the traffic aggregation link, the present invention fully considers the multi-type screening capabilities that adapt to various network conditions. For example, the technology can be connected to the core nodes of the city network to solve the detection and defense problems at the city level; it can also be connected to the network outlets on the enterprise side to solve the security detection and defense problems of the enterprise. In addition, it also supports deployment and docking at other network nodes, and can be flexibly used in combination with network traffic aggregation and collection equipment and open source traffic analysis software such as Snort.

[0205] S2: LSTM assists in sharding behavior recognition and traffic caching.

[0206] In the segmentation behavior identification and traffic caching link, the present invention realizes the flexible ability to track multiple behaviors, such as traffic segmentation behavior, specified IP, port or size, etc. Long short-term memory network (LSTM) is a special recurrent neural network that overcomes the shortcomings of traditional RNN by introducing a "memory unit" structure. The main associated components of LSTM and their functions are as follows:

[0207] Input Gate: Determines which new information will be stored in the memory cell. It includes the sigmoid activation function (determines the update part) and the tanh activation function (generates candidate values).

[0208] Memory cell (Cell State): The core of LSTM, which can retain information for a long time in a time series. Through the interaction of the forget gate and the input gate, it learns how to selectively remember or forget information.

[0209] Output Gate: Determines the next hidden state (i.e. the output of the next time step). Use the sigmoid activation function to determine which parts of the memory cell will be output, and multiply it with the tanh activation value of the memory cell to get the final output.

[0210] Forget Gate: Determines which information is forgotten from the memory unit and uses the sigmoid activation function to output a value between 0 and 1, indicating the proportion of retained information.

[0211] For content that is assembled and meets the specified requirements (such as a certain IP address, a single session with n shards, communication with a certain domain name, etc.), LSTM will output it to the specified path.

[0212] This embodiment uses LSTM to accurately identify traffic fragmentation behavior. LSTM can capture complex dependencies and long-term dependencies in the sequence through its special memory unit structure, significantly improving the accuracy of fragmentation behavior identification.

[0213] When a specific mark (such as MF≠0) appears in the traffic, the data packets under the same session will begin to be processed by LSTM, which will determine which new information will be stored in the memory unit and which information will be forgotten or output. The implementation process is as follows:

[0214] S21: Data preprocessing:

[0215] The captured fragmented data packets are preprocessed, including removing unnecessary header information, checksum verification, etc., to reduce the amount of data to be processed later. The key features of the fragmented data packets, such as sequence number, offset, data length, etc., are extracted, and these features will be used as input to the LSTM network.

[0216] S22: Sequence construction:

[0217] The fragmented data packets under the same session are arranged in sequence number order to form a time series. For each data packet, its feature vector is extracted and used as a time step input of the LSTM network.

[0218] S23: LSTM forward propagation:

[0219] The feature vector is input into the LSTM network, which processes the input data through its memory cells and three gating mechanisms (forget gate, input gate, output gate). At each time step, the LSTM network calculates a new hidden state and cell state based on the output of the previous time step and the input of the current time step. The hidden state contains a summary of all the input information so far, while the cell state saves long-term dependency information.

[0220] S24: Output processing:

[0221] The final output of the LSTM network can be a classification result (such as normal sharding or malicious sharding), or a probability value indicating whether the sharding sequence is abnormal. Based on the output result, it is decided whether to cache the current sharding sequence for subsequent processing, or whether to trigger the exception handling mechanism immediately.

[0222] The specific algorithm and process of LSTM network include S211~S213:

[0223] S211: Identify input conditions (optional: all traffic, specified IP, specified domain name, specified size traffic packets, etc.); S212: Identify and capture traffic fragmentation behavior corresponding to the conditions; S213: Further identify the MF≠0 mark. The corresponding detailed calculation process is as follows:

[0224] S211: Determine input conditions

[0225] C is an input condition set, where C can be one or more of {all traffic, specified IP, specified domain name, specified size traffic packet}.

[0226] For each condition c∈C, define a discriminant function D c (p), where p is the packet, D c (p) Returns a Boolean value (true or false) indicating whether packet p satisfies condition c.

[0227] S212: Identify and capture traffic fragmentation behavior corresponding to the conditions

[0228] For data packet p that meets condition c, check its fragmentation flag. Let F(p) be a function that returns the fragmentation information of data packet p (e.g., whether it is fragmented, fragment offset, etc.).

[0229] If F(p) indicates that the packet is fragmented, then the packet is captured.

[0230] S213: further identifying the MF≠0 mark

[0231] For the captured fragmented data packet p, check its MF (More Fragments) flag; let MF(p) be a function that returns the MF flag value (0 or 1) of data packet p.

[0232] If MF(p)≠0, it means that the data packet is part of a fragment sequence and is not the last fragment.

[0233] The above steps are described by mathematical expressions or conditions:

[0234] Input condition discrimination: For each c∈C, there is a discrimination function D c (p) = true or false.

[0235] Capturing sharding behavior: If Make D c If (p) = true and F(p) indicates fragmentation, then capture p.

[0236] Identify the MF≠0 flag: For a captured p, if MF(p)≠0, the condition is satisfied.

[0237] In addition, this embodiment improves LSTM, and the improvements include the following aspects:

[0238] 1. Feature selection and fusion:

[0239] The existing technology usually only focuses on basic sharding features (such as serial number, offset, etc.). In addition to traditional sharding features (such as serial number, offset, data length), this embodiment introduces more advanced features, such as statistical features based on traffic data (such as packet size distribution, time interval distribution) and content-based features (such as specific byte patterns). Through feature fusion (such as feature concatenation, feature weighting, etc.), these features are effectively combined to improve the recognition ability of the LSTM network. This feature selection and fusion method significantly improves the recognition accuracy because it provides a more comprehensive description of sharding behavior, which is different from the existing technology.

[0240] For feature selection, this embodiment defines basic features and advanced features:

[0241] The basic feature is F basic =[f1,f2,f3,…,f n ], where f1 is the sequence number, f2 is the offset, f3 is the data length, etc. The advanced feature is F advanced =[f n+1 ,f n+2 ,…,f m ], where f n+1 is the packet size distribution, f n+2 is the time interval distribution, f n+3for specific byte patterns, etc.

[0242] Feature fusion can be achieved by feature concatenation or weighting method:

[0243] Feature splicing: Feature splicing can be expressed as:

[0244] F merged =F basic ⊕F advanced；

[0245] Among them, F merged represents a combined feature, and ⊕ represents a feature concatenation operation.

[0246] Feature weighting: If you want to give different weights to different features, you can use the weighted sum method:

[0247] F weighted =w1f1+w2f2+…+w n f n +v1f n+1 +v2f n+2 +…+v k f m

[0248] Among them, F weighted Represents the final weighted value, w i , i∈[1,n] and v j , j∈[1,n] are the weights of basic features and advanced features respectively.

[0249] 2. Introducing the Attention Mechanism

[0250] The attention mechanism has been widely used in the fields of NLP and image processing, but it is not commonly used in LSTM models in network traffic analysis. Introducing the attention mechanism in the LSTM network enables the network to automatically focus on the important parts of the fragment sequence and ignore the unimportant parts. The attention mechanism can help the LSTM network better capture the key dependencies in the fragment sequence and improve the accuracy and efficiency of recognition. This innovation makes the model more robust and effective in dealing with complex fragment behaviors.

[0251] Introducing the attention mechanism into the LSTM network can effectively improve the performance of the network when processing complex sequence data. The following is the specific process of applying the attention mechanism to the LSTM network and combining the two.

[0252] S221: Input preparation

[0253] The input sequence is divided into multiple time steps, usually represented as X=[x1,x2,…,x T ]Where T is the number of time steps.

[0254] S222: LSTM processing

[0255] The LSTM network processes the input sequence to generate a hidden state h t :

[0256]

[0257] Where: i t It is the input gate, which controls the importance of the current input information; f t is the forget gate, which controls the retention of the previous hidden state information; t It is the output gate, which determines the output of the current hidden state;

[0258] t is the index of the previous time step, indicating the time step in the sequence currently being processed; x t Represents the input feature vector of the current time step, which is the input feature data of the current time step.

[0259] It is the hidden state of the previous time step, which contains the sequence information up to the previous time step; that is, the LSTM output of the previous time step, which contains the historical information of the sequence.

[0260] Weight matrix ( W f ,W i ,W c ,W o ): Used to connect the input and output of different layers and is learned through the training process. i is the weight matrix of the input gate; used to calculate the input gate activation value; W f is the weight matrix of the forget gate, which is used to calculate the activation value of the forget gate. o is the weight matrix of the output gate, which is used to calculate the activation value of the output gate; W c is the weight matrix of the candidate cell state, which is used to calculate the candidate cell state.

[0261] Bias term (b f ,b i ,b o ): Provides an offset for the activation function, which helps adjust the output range of the network. b i is the bias term of the input gate; b f is the bias term of the forget gate; b o is the bias term of the output gate.

[0262] Cell state (c t , ): The core memory unit in LSTM, used to store long-term dependency information. t is the cell state, is a candidate cell state; It is the cell state at the previous time step, which contains the sequence information up to the previous time step.

[0263] Activation function (σ, tanh): used to introduce nonlinear factors and enhance the expressive power of the network.

[0264] S223: Calculate attention weight

[0265] For each time step t, calculate the attention weight α t :

[0266]

[0267] Among them, e t Represents the current hidden state h t The relevant score indicates the importance of this time step;

[0268] v a represents the weight vector used to calculate the attention score;

[0269] W a : Represents the weight matrix of the hidden state, used to adjust the dimension of the hidden state.

[0270] U a : Represents the weight matrix of the query vector, which is used to adjust the dimension of the query vector.

[0271] α t : Normalized attention weight, indicating the importance of time step t.

[0272] e j is a quantitative indicator of the strength of the match or correlation between the hidden state and the query vector at time step j. When calculating the attention weights, these scores are used to determine the contribution of each time step to the final output.

[0273] S224: Weighted summation

[0274] The weighted sum of all hidden states is performed using the attention weights to obtain the weighted context vector c, which contains the aggregated information of important time steps:

[0275] .

[0276] S225: Final output

[0277] Combine the context vector with the current hidden state to make a prediction:

[0278]

[0279] in, represents the predicted output at time step t; W y represents the weight matrix of the output layer; b y Represents the bias term of the output layer.

[0280] 3. Multi-scale LSTM network:

[0281] Multi-scale processing is common in image processing, but it is a novel concept in network traffic analysis. This embodiment has a multi-scale LSTM network structure, in which LSTM networks at different levels process slice sequences of different time scales. The low-level LSTM network processes slice sequences of short time scales to capture local features; the high-level LSTM network processes slice sequences of long time scales to capture global features. Through multi-scale fusion, comprehensive analysis and accurate recognition of slice sequences are achieved. This structure makes the model's recognition of slice behavior more comprehensive and in-depth, which is different from the prior art.

[0282] The multi-scale LSTM network and the attention mechanism in 2) are used simultaneously. The specific algorithm and process of the multi-scale LSTM network are as follows: S231~S232.

[0283] The multi-scale LSTM network processes both short-time scale and long-time scale fragmented sequences simultaneously to achieve a comprehensive analysis of network traffic. The following is the specific process and algorithm of the multi-scale LSTM network:

[0284] S231: Input sequence preparation

[0285] Represent the input data as a time series X=[x1,x2,…,x T ], and divide it into short time scale subsequences X according to the time scale low and the long time scale subsequence X high .

[0286] S232: Short time scale LSTM processing

[0287] The low-level LSTM network processes the short-time-scale fragmented sequence to extract local features. The calculation process is as follows:

[0288] For each time step t (t=1,2,…,T, where T is the length of the segmented sequence), the calculation of the LSTM unit can be expressed as:

[0289] Forget Gate:

[0290]

[0291] Among them, W f is the weight matrix of the forget gate, is the hidden state h at the previous time step t−1 and the input x at the current time step t splicing, b f is the bias term, and σ is the sigmoid activation function.

[0292] Input Gate and Candidate Cell State:

[0293]

[0294]

[0295] Among them, W i and W C are the weight matrices of the input gate and candidate cell states, respectively, and b i and b c is the corresponding bias term, tanh is the hyperbolic tangent activation function, Represents a candidate cell state.

[0296] Cell status update:

[0297]

[0298] Where ⊙ represents element-by-element multiplication.

[0299] Output Gate and Hidden State Update:

[0300]

[0301]

[0302] Among them, W o is the weight matrix of the output gate, b o is the bias term. For the entire shard sequence, the LSTM network iterates the above calculation process from the first time step to the last time step. Ultimately, the hidden state of the last time step of each shard can be regarded as the local feature representation of the shard.

[0303] It should be noted that the above formula describes the calculation process of a single LSTM unit. In practical applications, LSTM layers usually contain multiple such units, each with its own set of parameters and calculated in parallel. In addition, if LSTM layers are stacked, the output of the previous layer will be used as the input of the next layer. If a bidirectional LSTM is used, there will be two independent LSTM layers processing the forward and reverse sequences respectively, and their hidden states will be spliced ​​or averaged to get the final output.

[0304] 4. Online learning and updating:

[0305] Implement the online learning function of the LSTM network, so that the network can continuously update its parameters and weights according to the actual traffic data. Through incremental learning or online transfer learning technology, the LSTM network can adapt to the ever-changing network environment and attack methods.

[0306] 5. Ensemble learning and model fusion:

[0307] The LSTM network is integrated with other machine learning models (such as random forests, support vector machines, etc.) to improve the overall recognition performance through model fusion technology. The prediction results of multiple models are combined through voting, weighted averaging, etc. to obtain more robust and accurate recognition results. The specific process of integrated learning and model fusion is:

[0308] S241: Model selection:

[0309] Choose from a variety of machine learning models such as Random Forest (RF), Support Vector Machine (SVM), and deep learning models such as LSTM.

[0310] S241: Model training:

[0311] For each selected model, the same training data is used for training to ensure that each model can learn different features.

[0312] S243: Prediction generation:

[0313] Use the trained model to predict the test data and obtain the prediction results of multiple models.

[0314] Assuming there are n models, the prediction results can be expressed as:

[0315]

[0316] in, Represents the predicted output of the i-th model, where i is 1~n.

[0317] S244: Result Fusion:

[0318] Use voting or weighted average to merge the results. The implementation of voting (majority voting) is as follows:

[0319]

[0320] Among them, k is all possible class labels and 1 is the indicator function.

[0321] Weighted average method:

[0322]

[0323] Among them, w i is the weight of the ith model, reflecting its importance in the overall decision, and can be set based on the performance of the model on the validation set.

[0324] S245: Model evaluation: Use methods such as cross-validation to evaluate the performance of the fused model, such as accuracy, precision, and recall.

[0325] In this way, ensemble learning can take advantage of the advantages of multiple models and significantly improve the recognition performance of fragmented traffic.

[0326] The fragmented traffic finally identified will be cached for subsequent processing. This embodiment adopts a cache mechanism integrating multiple technologies: the cache mechanism combines traffic classification technology based on pattern recognition and traffic identification technology based on machine learning to achieve rapid identification and caching of malicious website traffic.

[0327] Traditional caching mechanisms often focus on a single data storage function, while the present invention uses pattern recognition and machine learning technology, which not only realizes the efficient storage of sharded traffic, but also improves the utilization value of cached data. Traditional caching mechanisms are often static and cannot be adjusted according to the characteristics of the traffic and processing requirements. The caching mechanism in this algorithm combines pattern recognition and machine learning technology. The caching mechanism can dynamically adjust the caching strategy according to the characteristics of the traffic and processing requirements, such as cache size, cache time, priority setting, etc. This dynamic adaptability not only improves the utilization of the cache, but also speeds up the processing speed, which is a major improvement over the existing caching mechanism.

[0328] The traffic classification and identification process is placed in front and integrated into the cache mechanism, making the entire processing process more intelligent. Rapid classification through pattern recognition and in-depth identification using machine learning ensure that the data in the cache is preliminarily screened and valuable shard traffic.

[0329] S3: Perform session reorganization and feature matching based on the FCM algorithm.

[0330] The FCM (Fuzzy C-means) clustering algorithm is used to reorganize the session of the fragmented traffic in the cache (such as Figure 6), restoring the complete network session. Figure 6 This is a diagram of FCM session reassembly. First, for a new fragment that arrives, packet confirmation is performed: determine which package the fragment belongs to based on the source address, destination address, and identifier; then partial confirmation is performed: determine which part of the original package the fragment belongs to based on the offset; then use the FCM algorithm to cluster user behavior feature vectors and classify users with similar behavior patterns into one category; finally, the data enters the fragment reassembly buffer.

[0331] Extract key features such as domain name, IP, port, URL, etc. in the session and match them with known malicious website traffic features. The matching process uses a fast matching algorithm based on hash tables to achieve rapid identification and feature extraction of malicious website traffic in the cache.

[0332] The existence of malicious traffic can be further confirmed through pattern recognition technology, providing an accurate basis for subsequent disposal. In the bad traffic monitoring system based on LSTM and clustering algorithms, pattern recognition technology is mainly used to further confirm the existence of malicious traffic as a supplement or verification method for the recognition results of machine learning models. In the session reorganization and feature matching stage, pattern recognition technology can quickly verify and identify potential malicious traffic features.

[0333] like Figure 4 and 6 As shown, the specific steps of using the FCM algorithm to perform session reorganization and feature matching include S31 to S34:

[0334] S31: Behavior data collection: Collect users' online behavior data, such as access time, access duration, access frequency, access content type, etc.

[0335] S32: Feature construction: construct a user behavior feature vector based on the behavior data.

[0336] S33: FCM clustering: Use the FCM algorithm to cluster user behavior feature vectors and classify users with similar behavior patterns into one category.

[0337] In the bad traffic monitoring system based on LSTM and clustering algorithm, using FCM (Fuzzy C-means) algorithm for session reorganization and feature matching is a key link. The following will elaborate on the specific implementation algorithm process and formula of using FCM algorithm for session reorganization and feature matching, and explore its innovations and effects.

[0338] The FCM (Fuzzy C-Means) algorithm is a fuzzy clustering method based on an objective function. It divides several data points into multiple fuzzy groups and attempts to minimize a given objective function, usually minimizing the sum of weighted square distances between data points and cluster centers. The FCM algorithm allows a data point to belong to multiple clusters, and the degree of belonging to each cluster is determined by the degree of membership. The specific implementation process is as follows S331~S334:

[0339] S331: Data preprocessing and feature extraction

[0340] In the session reorganization phase, the fragmented traffic in the cache needs to be reorganized to restore the complete network session. Then, key features such as domain name, IP address, port number, URL path, request method, etc. are extracted from the reorganized session.

[0341] S332: Initialization

[0342] Determine the number of clusters c (i.e. the expected number of clusters to be divided into). Randomly initialize c cluster centers v1, v2, ..., v c .

[0343] Set the initial value of the membership matrix U, which can usually be set to a matrix with all elements equal to one-tenth of c, representing all clusters that each data point may initially belong to. Set the iteration stop condition, such as the maximum number of iterations or the objective function change is less than a certain threshold.

[0344] S333: Iterative Optimization

[0345] For each iteration, the following steps are performed:

[0346] Update the membership matrix U: For each data point x j And each cluster center v i , update the membership u ij , the formula is as follows:

[0347]

[0348] Here, m is the fuzzification parameter (usually greater than 1).

[0349] c: The number of clusters, that is, the number of cluster centers.

[0350] x j : The jth data point, which is a behavioral feature vector containing the eigenvalues ​​of each data point. j The membership value of the i-th cluster is a number between 0 and 1.

[0351] v i: The i-th cluster center, which is also a vector, indicating the center position of the cluster.

[0352] v k : The kth cluster center (k≠i), used as a comparison object when calculating the membership.

[0353] m: Fuzzification coefficient (also called fuzzy index or smoothing parameter), which is a real number greater than 1. It controls the degree of fuzziness of the membership value. The larger m is, the more fuzzy the membership value is (that is, the membership values ​​are closer).

[0354] ‖⋅‖: represents the norm of the vector. Euclidean distance (i.e. 2-norm) is usually used to calculate the distance from the data point to the cluster center. is the jth data point x j To cluster center v i The Euclidean distance of is the jth data point x j To the kth cluster center v k The Euclidean distance of

[0355] Update cluster center v:

[0356] For each cluster center v i , update its position according to the membership matrix, the cluster center feature vector v i The calculation formula is:

[0357]

[0358] Among them, v i is the i-th cluster center, x j is the behavioral feature vector of the jth data point, u ij is the membership of the jth data point to the i-th cluster, m is the fuzzification parameter, n is the total number of users, Indicates u ij to the power of m.

[0359] S334: Convergence judgment

[0360] Check the change of the objective function or whether the maximum number of iterations has been reached. If the stopping condition is met, the iteration ends.

[0361] S34: Abnormal behavior identification: Analyze the clustering results and identify abnormal clusters that are significantly different from normal user behavior patterns. Users in these clusters may have visited bad websites. The specific method process is as follows S341: ~ S345:

[0362] S341: Clustering results analysis

[0363] After clustering by the FCM algorithm, multiple cluster centers and the degree of membership of each data point (i.e., user behavior feature vector) to each cluster are obtained. Each cluster represents a group of users with similar behavior patterns.

[0364] S342: Calculate clustering features

[0365] For each cluster, the characteristic vector of its cluster center can be calculated. This vector is the weighted average of all user behavior characteristic vectors in the cluster, and the weight is the membership degree.

[0366] S343: Comparing clustering features with normal behavior patterns

[0367] Define one or more reference cluster centers v of normal behavior patterns normal ; These reference centers can be obtained by analyzing a large amount of known normal user behavior data.

[0368] For each cluster center v i , calculate its difference with the normal behavior pattern reference cluster v normal The distance or similarity between them. Commonly used distance metrics include Euclidean distance, Manhattan distance, etc., and the similarity metric can be cosine similarity.

[0369] S344: Identifying abnormal clusters

[0370] Set a threshold δ to determine whether the cluster is abnormal. If a cluster center v i With the normal behavior pattern reference cluster center v normal If the distance between them is greater than δ, or the similarity is less than a certain threshold, the cluster is considered abnormal.

[0371] S345: Handling abnormal clustering

[0372] For the identified abnormal clusters, further analyze the user behavior data, especially those users with high membership, who may be involved in the visit of bad websites or malicious traffic activities. Further measures can be taken, such as marking these users, restricting their access rights, or conducting in-depth investigations.

[0373] Specific innovations include the following aspects:

[0374] 1) Multi-feature fusion clustering:

[0375] Traditional FCM algorithms usually perform clustering based on only a single or a few features. In the present invention, multiple key features such as domain name, IP, port, URL, etc. are combined for clustering, which improves the accuracy and practicality of clustering.

[0376] In multi-feature fusion clustering, the clustering effect can be improved by combining multiple features. For example:

[0377] Suppose there is a set of network traffic data with the following features:

[0378] Domain Name: such as example.com, malicious-site.com;

[0379] IP Address: such as 192.168.1.1, 10.0.0.1;

[0380] Port number: such as 80 (HTTP), 443 (HTTPS);

[0381] URL Path: such as / login, / api / data.

[0382] Example:

[0383] In traditional FCM algorithms, clustering is performed based on a single feature (such as IP address), which may cause the same type of traffic to be classified into different clusters. For example, 192.168.1.1 and 10.0.0.1 can both be malicious traffic, but if only the IP address is considered, they may be classified into different classes.

[0384] In multi-feature fusion clustering, the present invention combines these features together:

[0385] For example.com, the characteristics are:

[0386] Domain name: example.com;

[0387] IP address: 192.168.1.1;

[0388] Port: 80;

[0389] URL path: / login.

[0390] For malicious-site.com, the characteristics are:

[0391] Domain name: malicious-site.com;

[0392] IP address: 10.0.0.1;

[0393] Port: 443;

[0394] URL path: / api / data.

[0395] By combining these features, the clustering algorithm can identify that example.com and malicious-site.com belong to the same type of traffic, even though they have different IP addresses and ports. Ultimately, the clustering results will better reflect the true situation of the traffic, thereby improving the accuracy and practicality of malicious traffic identification.

[0396] 2) Dynamic cluster center update:

[0397] The cluster centers are continuously updated through iterative optimization, making the clustering results more stable and consistent with actual traffic characteristics.

[0398] 3) Combination with LSTM results:

[0399] The FCM clustering results can be combined with the LSTM model's identification results to further improve the accuracy of malicious traffic identification. For example, traffic that LSTM identifies as suspicious but with low confidence can be further verified using the FCM clustering results. The specific process of combining the FCM clustering results with the LSTM model's identification results can be divided into the following steps:

[0400] 1. Data preprocessing

[0401] Before traffic analysis, network traffic data must first be preprocessed, including denoising, normalization, etc. This step ensures that the input data is suitable for subsequent LSTM and FCM algorithms.

[0402] 2. Use LSTM for traffic identification

[0403] Model training: First, use historical traffic data to train the LSTM model. LSTM learns the temporal characteristics of traffic sequences and predicts the label of each traffic segment (such as normal, suspicious, malicious, etc.).

[0404] Model prediction: For new traffic data, the LSTM model outputs the classification results and corresponding confidence levels (such as probability values) for each traffic segment. Suppose a traffic segment is identified as "suspicious", but the confidence level is low (such as below a certain threshold).

[0405] 3. Clustering using FCM

[0406] Feature extraction: Extract multiple features (such as domain name, IP address, port, URL, etc.) from the same traffic data and input these features into the FCM clustering algorithm.

[0407] Clustering results: FCM clusters traffic segments into different groups. Each group represents traffic segments with similar characteristics.

[0408] 4. Combining LSTM and FCM results

[0409] Threshold setting: Set a confidence threshold (such as 0.7). If the LSTM prediction confidence is lower than the threshold, the classification result of the traffic segment is considered unreliable.

[0410] Clustering verification:

[0411] For a traffic segment identified as "suspicious" by LSTM, the FCM cluster to which the traffic segment belongs is checked.

[0412] If the traffic segment is in the same cluster as other known malicious traffic segments, the confidence level of its maliciousness can be further improved, and a more accurate judgment can be made. Conversely, if the traffic segment is in the same cluster as most normal traffic segments, it can be considered unlikely to be malicious traffic.

[0413] 5. Final Decision

[0414] Decision making: Based on the recognition results of LSTM and the clustering results of FCM, the final decision is made. If the confidence of LSTM is low, but the clustering results of FCM support the suspicious judgment of LSTM, the traffic can be marked as "malicious".

[0415] Output results: Output the final traffic classification results, record the identification results and decision basis for subsequent analysis and model optimization.

[0416] Through this combination, the FCM clustering results can effectively make up for the shortcomings of LSTM in some cases and improve the accuracy and reliability of malicious traffic identification.

[0417] 4) Fast feature matching:

[0418] After extracting clustering features, a fast matching algorithm based on hash tables is used to match them with the malicious traffic feature library, which improves matching efficiency and accuracy.

[0419] In addition, in the session reorganization and feature matching link, the present invention innovates the decoupled feature library association. This design can flexibly include fraud features, network attack features, gambling features and other black and gray features. These features can act alone or in parallel, which is convenient for on-demand configuration according to actual conditions. In the bad traffic monitoring process based on LSTM and clustering algorithm, the purpose of designing decoupled feature library association is to provide higher flexibility and scalability, so that the system can easily integrate different types of malicious traffic feature libraries and configure them according to actual security requirements. The following is a specific method for implementing decoupled feature library association, including (1) to (5):

[0420] (1) Modular design of feature library

[0421] First, different types of feature libraries (such as fraud feature library, network attack feature library, gambling feature library, etc.) are designed as independent modules. Each module contains a specific feature set and the matching logic associated with it.

[0422] Fraud feature library module: Contains features such as domain names, IP addresses, URL patterns, etc. related to fraud activities.

[0423] Network attack signature library module: contains signatures of network attacks such as SQL injection and cross-site scripting (XSS).

[0424] Gambling feature library module: contains features such as domain names and URL paths related to gambling websites.

[0425] (2) Standardization of feature library interface

[0426] Define a set of unified interface standards for the interaction between each feature library module and other parts of the system. These interfaces should include features such as feature loading, feature matching, and feature updating.

[0427] Loading interface: used to load feature data from feature library files.

[0428] Matching interface: matches the provided network traffic features (such as domain name, IP, URL, etc.) in the feature library and returns the matching results.

[0429] Update interface: used to load new features into the system when the feature library is updated.

[0430] (3) Feature matching process integration

[0431] In the session reorganization and feature matching stage, the decoupled feature library association process is integrated into the system. The specific steps are as follows (31)~(34):

[0432] (31): After the session is reorganized, extract the key features in the session (such as domain name, IP, URL, request method, etc.).

[0433] (32): According to the feature type, call the matching interface of the corresponding feature library module. For example, if the extracted feature contains a URL, call the matching interface of the fraud feature library module and the network attack feature library module to perform matching respectively.

[0434] (33): Collect the matching results of each feature library module and make a comprehensive judgment. If a session has matching items in multiple feature libraries, it may mean that the session involves multiple types of malicious behaviors.

[0435] (34): Based on the matching results, execute corresponding disposal measures, such as marking the session, blocking the connection, sending a warning, etc.

[0436] (4) Feature library configuration and management

[0437] Provides a set of configuration management tools that allow system administrators to configure the use of signature libraries according to actual needs.

[0438] Enable / disable signature library: Administrators can enable or disable specific signature library modules based on the current security threat situation.

[0439] Priority setting: Set priorities for different signature library modules so that when there are matching items in multiple signature libraries, they are processed in order of priority.

[0440] Dynamic update: Supports dynamic update of the feature library. When new malicious traffic features are discovered, these features can be added to the corresponding feature library in a timely manner and loaded into the system through the update interface.

[0441] (5) System monitoring and feedback

[0442] Monitor the usage of the feature library, collect key indicators such as matching success rate and false alarm rate, and optimize and adjust the feature library based on these indicators.

[0443] Performance monitoring: Monitor the performance of feature matching to ensure that the system can maintain stable operation under high load conditions.

[0444] Result feedback: The matching results and the effects of the disposal measures are fed back to the feature library maintenance team so that they can optimize the quality and accuracy of the feature library based on the feedback.

[0445] This embodiment further confirms the existence of malicious traffic through pattern recognition technology. The following is the specific implementation process and algorithm of the pattern recognition technology, including (1) to (5):

[0446] (1) Feature extraction

[0447] After the session is reorganized, key features are first extracted from the reorganized session. These features may include:

[0448] Domain name: The domain name of the website being visited.

[0449] IP address: The IP address of the server being accessed.

[0450] Port number: The access port number.

[0451] URL path: the specific page path visited.

[0452] Request method (such as GET, POST, etc.).

[0453] Request parameters: The query string in the URL or the data in the POST request body.

[0454] Traffic patterns: statistical characteristics such as packet size distribution, request frequency, session duration, etc.

[0455] (2) Feature preprocessing

[0456] Preprocess the extracted features, including:

[0457] Data cleaning: Remove invalid or abnormal data.

[0458] Normalization: Convert features of different magnitudes to the same magnitude for subsequent processing.

[0459] Encoding: Encode categorical features, such as One-Hot Encoding or Label Encoding.

[0460] (3) Constructing a pattern recognition model

[0461] The pattern recognition model can be a rule-based matching system or a simple machine learning model (such as decision tree, support vector machine, etc.) to quickly verify malicious traffic characteristics. The following is the implementation process of the rule-based matching system:

[0462] Rule definition: Define a series of rules based on known malicious traffic characteristics and security policies. These rules can include:

[0463] Domain blacklist: A list of domain names that are known to be malicious.

[0464] IP Blacklist: A list of known malicious IP addresses.

[0465] URL pattern matching: Defines common patterns of malicious URLs, such as containing specific keywords or paths.

[0466] Traffic anomaly rules: define the thresholds for traffic anomalies (such as too high request frequency, abnormal data packet size, etc.).

[0467] Rule matching: Match the extracted features with the defined rules:

[0468] Domain name and IP match: Check if the domain name and IP address are in the blacklist.

[0469] URL pattern matching: Use regular expressions or string matching algorithms to check whether the URL matches malicious patterns.

[0470] Traffic anomaly detection: Determine whether there is abnormal behavior based on traffic statistical characteristics.

[0471] (4) Result determination

[0472] Determine whether the traffic is malicious based on the rule matching results:

[0473] If the traffic characteristics match any malicious rule, it is considered malicious traffic.

[0474] If the traffic characteristics do not match any malicious rules, but the machine learning model determines it to be malicious (for example, the LSTM model outputs a high probability), the final judgment is made by combining the results of both.

[0475] (5) Feedback and optimization

[0476] Feedback mechanism: Feedback the judgment results into the system to update the blacklist, optimize the rule set or training data.

[0477] Continuous optimization: Regularly update rule sets and models based on new malicious traffic samples and security policies.

[0478] The algorithm example is as follows (rule-based matching):

[0479] function match_malicious_traffic(session_features):

[0480] # Assume that session_features is a dictionary containing domain name, IP, URL and other features

[0481] malicious = False

[0482] # Domain blacklist matching

[0483] if session_features['domain'] in domain_blacklist:

[0484] malicious = True

[0485] # IP blacklist matching

[0486] if session_features['ip'] in ip_blacklist:

[0487] malicious = True

[0488] # URL pattern matching (using regular expressions)

[0489] for pattern in url_patterns:

[0490] if re.match(pattern, session_features['url']):

[0491] malicious = True

[0492] break

[0493] # Traffic anomaly detection (this is pseudo code, actual implementation needs to be based on specific rules)

[0494] # if check_traffic_anomaly(session_features):

[0495] # malicious = True

[0496] return malicious

[0497] S4: Disposal execution: Quickly and accurately identify and dispose of malicious traffic through feature matching and automated disposal technology.

[0498] After discovering malicious website traffic, take appropriate disposal measures in a timely manner, such as blocking IP addresses, intercepting access, etc., to effectively curb the spread and diffusion of malicious traffic. Monitor and evaluate the disposal results in real time to ensure the effectiveness and timeliness of the disposal measures. First, use rule engine technology to conduct precise sampling and disposal to achieve automated processing and implementation of disposal measures. Then use predictive blocking technology based on probability models and fuzzy disposal to accurately block and intercept malicious website traffic, or use any of these technologies.

[0499] The following is the specific process of using rule engine technology to realize the automated processing and implementation of disposal measures, including (1) to (6):

[0500] (1) Define the rules:

[0501] Define a series of clear disposal rules based on business needs and security policies. These rules describe the disposal measures to be taken in different situations, such as blocking access, sending warnings, recording logs, etc. Rules can include logic such as conditional judgment, pattern matching, time window restrictions, etc.

[0502] (2) Rule engine configuration:

[0503] Select and configure a suitable rule engine system. The rule engine should be able to handle complex logic, respond quickly, and execute efficiently. Import the defined disposal rules into the rule engine and make necessary configurations, such as setting the priority of the rules and defining the dependencies between the rules.

[0504] (3) Data input and processing:

[0505] Collect relevant event or behavior data from monitoring systems, log records, sensors and other data sources in real time or regularly. Preprocess the collected data, such as format conversion, cleaning, and deduplication, to ensure data accuracy and consistency.

[0506] (4) Rule matching and execution:

[0507] The rule engine matches the input data according to the imported rules. This usually involves processes such as conditional judgment and pattern recognition of the data. If the data meets the triggering conditions of a rule, the rule engine will trigger the corresponding disposal measures. The execution of disposal measures should be quick and accurate to minimize potential security risks or damages.

[0508] (5) Monitoring and feedback:

[0509] Monitor the execution process of the rule engine in real time to ensure that the disposal measures are correctly executed. Collect feedback information after the disposal measures are executed, such as execution results, scope of impact, etc. Make necessary adjustments and optimizations to the rules based on the feedback information, such as modifying rule conditions, adjusting priorities, etc.

[0510] (6) Rule optimization and update:

[0511] Regularly review and optimize the rules to adapt to the ever-changing network environment and security threats. When new security vulnerabilities or attack methods are discovered, update the rule base in a timely manner to ensure that the system can respond effectively.

[0512] In the processing execution link, the present invention has carried out verification tests on interception, diversion, printing records, and DNS rejection resolution, IP address prohibition, TCP session packet deconstruction and other capabilities in the interception action, to ensure that appropriate processing measures can be selected as needed according to different situations.

[0513] The following is the specific implementation process of the predictive ban technology based on the probability model, including (1) to (5):

[0514] S41: Building a probabilistic prediction model

[0515] (1) Data collection and preprocessing:

[0516] Collect a large amount of historical traffic data, including normal traffic and known malicious traffic. Preprocess the data, including removing invalid data, formatting, and feature extraction. Features may include packet size, access frequency, session duration, specific byte patterns, etc.

[0517] (2) Model training:

[0518] Use machine learning algorithms (such as logistic regression, decision tree, random forest, neural network, etc.) to train the probability prediction model. Divide the preprocessed data into a training set and a test set, use the training set to train the model, and verify the performance of the model on the test set. Adjust the model parameters through cross-validation, grid search, etc. to optimize the prediction ability of the probability prediction model.

[0519] S42: Real-time traffic prediction:

[0520] The same feature extraction is performed on the current real-time traffic data; the extracted feature data is input into the trained probability prediction model to obtain the probability value that the current traffic is malicious traffic.

[0521] S43: Threshold determination and ban

[0522] Set one or more thresholds (such as 0.8) to determine whether the traffic is malicious. Based on the comparison between the probability value output by the probability prediction model and the threshold, decide whether to take blocking measures. If the probability value exceeds the set threshold, the traffic is considered malicious and immediately performs blocking operations, such as blocking the connection, adding to the blacklist, sending a warning message, etc.

[0523] S44: Model update and optimization

[0524] Regularly collect new traffic data, including normal traffic and newly emerging malicious traffic. Use new data to update and retrain the probability prediction model to improve its accuracy and adaptability; adjust model parameters and thresholds based on actual application results to optimize the blocking effect.

[0525] The innovation and advantages of the predictive blocking technology based on the probability model in the present invention are mainly reflected in the following aspects:

[0526] 1) Data-driven decision making:

[0527] Traditional blocking technologies often rely on fixed rules or thresholds to determine whether traffic is malicious. This approach is not flexible enough when facing complex and ever-changing network attacks. However, predictive blocking technology based on probability models uses machine learning algorithms to learn the characteristic differences between normal and malicious traffic from historical traffic data, thereby being able to more accurately predict the maliciousness of current traffic. This data-driven approach makes blocking decisions more intelligent and accurate.

[0528] 2) Dynamic adaptability:

[0529] As network environments and attack methods continue to change, traditional static rules will soon become outdated. However, predictive blocking technology based on probability models can adapt to these changes by regularly updating the model. By introducing new traffic data to retrain the model, the model's predictive ability and accuracy can be continuously improved, thereby maintaining the effectiveness and timeliness of the system.

[0530] 3) Fine-grained processing:

[0531] The probability prediction model can not only determine whether the traffic is malicious, but also give the probability value of the traffic belonging to the malicious category. This allows the system to take different degrees of disposal measures according to the probability value, such as immediately banning high-probability malicious traffic, while taking warnings or further observation strategies for low-probability traffic. This fine-grained processing method improves the flexibility and efficiency of the system.

[0532] 4) Reduce false positives and false negatives:

[0533] Traditional rule-based blocking systems often have high false positive and false negative rates. However, predictive blocking technology based on probability models can more comprehensively assess the maliciousness of traffic by combining multiple features for prediction, thereby reducing the occurrence of false positives and false negatives to a certain extent.

[0534] This embodiment uses rule engine technology to realize automatic processing and implementation of disposal measures. When the malicious traffic is confirmed to exist, execution information is sent to the execution unit, and the disposal execution module is started to perform corresponding processing.

[0535] In the bad traffic monitoring method based on LSTM and clustering algorithm, the automation and implementation innovation of disposal measures by using rule engine technology is mainly reflected in the following aspects:

[0536] 1) Flexibility and scalability:

[0537] Traditional automated processing systems often rely on hard-coded logic and are unable to flexibly respond to changing network environments and security threats. However, rule engine technology allows new security requirements and policies to be adapted by configuring rules rather than modifying code, greatly improving the flexibility and scalability of the system.

[0538] 2) Simplification of complex logic:

[0539] Network security measures often involve complex logical judgments and multiple condition combinations. The rule engine provides a visual rule editor or scripting language to make the construction and maintenance of complex logic more intuitive and simple.

[0540] 3) Quick response:

[0541] The rule engine can process input data in real time and quickly trigger corresponding disposal measures according to preset rules. This rapid response capability is crucial for network security and can block malicious traffic in time to reduce potential damage.

[0542] 4) Multi-source data integration:

[0543] The system may need to process security events and traffic data from multiple data sources. The rule engine can easily integrate these data sources and trigger different processing rules based on the data from different sources.

[0544] The core content of the present invention includes four aspects: traffic convergence, fragmentation behavior identification and traffic caching, session reorganization and feature matching, and disposal execution. First, the BPF technology is used to efficiently aggregate network traffic, providing a solid foundation for subsequent analysis. Secondly, the fragmentation traffic is accurately identified through the LSTM network, and its special memory unit structure can capture complex dependencies and significantly improve the recognition accuracy. Furthermore, the FCM clustering algorithm is used to reorganize the session, restore the complete network session, and quickly identify malicious traffic through feature matching. Finally, timely disposal measures are taken for confirmed malicious traffic to ensure network security.

[0545] 1) BPF and LSTM linkage mechanism: Combine efficient traffic aggregation with deep learning models to achieve accurate identification of sharded traffic.

[0546] 2) Application of LSTM in sharding behavior recognition: LSTM memory units are used to capture complex dependencies in traffic sharding sequences and improve recognition accuracy.

[0547] 3) Application of FCM clustering algorithm in session reconstruction: Restore network sessions through fuzzy C-means clustering to provide complete data for malicious feature extraction.

[0548] These key technologies and innovative methods together constitute a comprehensive and efficient detection and disposal system for bad website traffic fragmentation behavior of the present invention, which is the core point of the present invention.

[0549] The present invention can be widely used in network security protection, fraud defense, network gambling detection, network problem location, network experimental analysis and research, etc., providing a new and effective protection method for the network security field.

[0550] The bad traffic monitoring method and system based on LSTM and clustering algorithm of the present invention realizes the rapid and accurate detection and management of bad website traffic fragmentation behavior by combining LSTM and clustering algorithm, and significantly improves network security and stability. At the same time, the present invention effectively protects enterprise network resources and improves network security protection capabilities. In summary, the beneficial effects of the present invention are significant, and a new and effective protection method is provided for the field of network security.

[0551] The present invention aims to solve the increasingly prominent website security issues in the current Internet environment, especially the challenges brought by the new attack method of bad website traffic fragmentation. With the rapid development of Internet technology, network attack methods are becoming more and more complex and hidden. Among them, bad website traffic fragmentation cleverly mixes malicious traffic with normal traffic to evade the detection of traditional security protection measures, resulting in serious consequences such as website performance degradation and sensitive data leakage.

[0552] The solution of the present invention combines TCP protocol characteristics, traffic collection and analysis, abnormal feature recognition and machine learning algorithm design to provide relevant technologies and solutions for the above problems.

[0553] In addition to realizing the above functions, the present invention also has the following extended application scenarios:

[0554] 1) Cloud security service: Integrate the present invention into cloud security services to provide cloud tenants with real-time traffic slicing behavior detection and protection, thereby enhancing the security of cloud applications.

[0555] 2) Edge computing environment: The present invention is deployed on edge computing nodes to perform real-time analysis on the traffic generated by edge devices, reduce data transmission delays, and improve the response speed of security protection.

[0556] 3) IoT security: With the popularization of IoT devices, IoT security has become a new challenge. The present invention can be applied to IoT scenarios to perform segmented behavior detection on the communication traffic between IoT devices to prevent malicious attacks and data leakage.

[0557] 4) Internal network security of an enterprise: The present invention is deployed in an enterprise network to monitor and analyze traffic in and out of the enterprise network, timely discover and prevent potential security threats, and protect the core assets of the enterprise.

[0558] To sum up, although there are a variety of alternative implementation methods and extended application scenarios, the present invention has formed a comprehensive and efficient detection and disposal solution for bad website traffic fragmentation behavior by comprehensively applying advanced technologies such as BPF, LSTM, FCM, etc. Its advancement and innovation lie in the synergy of its overall architecture and the optimized coordination of each link, thereby achieving effective response to complex network attack behaviors.

[0559] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A bad traffic monitoring method based on LSTM and clustering algorithm, characterized in that: The following steps are involved: S1: Traffic aggregation and screening based on BPF: collect and analyze website traffic data in real time on the network server or proxy server, classify, count and monitor network traffic, and promptly detect and handle abnormal traffic; The implementation method is: Let R be a set of filtering rules, where each rule r∈R is a logical expression used to determine whether a packet p satisfies the rule; For each arriving data packet p, traverse the filter rule set R, and if the data packet p satisfies rule r, it is captured; Let G be a set of packet groups, where each group g∈G corresponds to a specific attribute; for each captured packet p, add it to the corresponding group g according to its attribute; For each group g, define the filtering condition S(g); If the group g meets the screening condition S(g), then output the relevant information of the group; R ={ r 1, r 2, ..., rn }: filter rule set; P ={ p 1, p 2, ..., pm }: data packet set; G ={ g 1, g 2, ..., gk }: data packet grouping set; S ( g ): Packet filtering conditions, for each data packet pi ∈ P : Traversing the filter rule set R ,if Make rj ( pi )=true, the packet is captured pi ; According to this data packet pi The attribute is added to the corresponding group gl ∈ G middle; For each group gl ∈ G :if S ( gl )=true, then output gl relevant information; S2: Implement sharding behavior recognition and traffic caching based on LSTM. With the help of LSTM, the traffic sharding behavior is accurately recognized. LSTM captures the complex dependencies and long-term dependencies in the sequence through its memory unit structure. When a specific mark MF≠0 appears in the traffic, the data packets under the same session will start LSTM processing to determine which new information will be stored in the memory unit and which information will be forgotten or output. Finally, the identified sharding traffic will be cached. The specific implementation process is as follows: S21: Data preprocessing: Preprocess the captured fragmented data packets, including removing unnecessary header information, checking and verifying, to reduce the amount of data to be processed later, and extract the key features of the fragmented data packets, including sequence number, offset, and data length. The key features will be used as the input of the LSTM network; S22: Sequence construction: Arrange the fragmented data packets under the same session in the order of sequence numbers to form a time series; extract the feature vector of each data packet and use it as a time step input of the LSTM network; S23: LSTM forward propagation: The feature vector is input into the LSTM network, which processes the input data through its memory cells and three gating mechanisms. At each time step, the LSTM network calculates a new hidden state and cell state based on the output of the previous time step and the input of the current time step. The hidden state contains a summary of all the input information so far, while the cell state stores long-term dependency information. S24: Output processing: The LSTM network finally outputs the classification result: normal shard or malicious shard, or a probability value indicating whether the shard sequence is abnormal. Based on the output result, it decides whether to cache the current shard sequence for subsequent processing, or whether to trigger the exception handling mechanism immediately. S3: Use the FCM algorithm to perform session reorganization and feature matching to restore the complete network session. The implementation process of feature matching using the FCM algorithm is as follows: S31: Behavior data collection: collect users' online behavior data, including access time, access duration, access frequency and access content type; S32: Feature construction: constructing user behavior feature vector based on behavior data; S33: FCM clustering: Use the FCM algorithm to cluster user behavior feature vectors and group users with similar behavior patterns into one category; S34: Abnormal behavior identification: Analyze clustering results and identify abnormal clusters that are significantly different from normal user behavior patterns; S4: Identify and handle malicious traffic through feature matching and automated handling technology; Adopt the predictive blocking technology based on probability model to accurately block and intercept malicious website traffic; use the rule engine technology to realize the automatic processing and implementation of disposal measures; the specific process of predictive blocking technology based on probability model is as follows: S41: Building a probabilistic prediction model 1) Collect historical traffic data, including normal traffic and known malicious traffic; 2) Extract features from these traffic data to extract packet size, access frequency, session duration, and specific byte patterns; 3) Use machine learning algorithms to train probabilistic prediction models to distinguish between normal and malicious traffic; S42: Real-time traffic prediction Perform the same feature extraction on the current real-time traffic data; input the extracted features into the trained probability prediction model to obtain the probability value of the current traffic being malicious traffic; S43: Threshold determination and ban Set a threshold to determine whether the traffic is malicious. The threshold is adjusted according to the actual application scenario and accuracy requirements. If the probability value exceeds the set threshold, the traffic is considered malicious and is immediately blocked. The blocking process includes blocking the connection, adding to the blacklist, and sending a warning message. S44: Model update and optimization Regularly collect new traffic data, including normal traffic and newly emerging malicious traffic; use new data to update and retrain the probability prediction model to improve the accuracy and adaptability of the model; adjust model parameters and thresholds based on actual application results to optimize the blocking effect; In step S4, the specific process of using rule engine technology to realize the automatic processing and implementation of disposal measures is as follows: 1) Define the rules Define a series of clear handling rules based on business needs and security policies. These rules describe the handling measures to be taken in different situations, including blocking access and sending warnings. 2) Rule engine configuration: Select and configure the rule engine system, which has the ability to handle complex logic, respond quickly and execute efficiently; import the defined disposal rules into the rule engine and make necessary configurations to ensure that the rules can be triggered and executed correctly; 3) Data input and processing Collect relevant event or behavior data from monitoring systems, log records, and sensor data sources in real time or periodically; pre-process the collected data to ensure data accuracy and consistency; 4) Rule matching: The rule engine matches the input data according to the imported rules; it involves conditional judgment and pattern recognition of the data; if the data meets the triggering conditions of a certain rule, the rule engine will trigger the corresponding disposal measures; 5) Implementation of disposal measures: According to the triggered rules, the rule engine automatically executes corresponding disposal measures, including sending warning messages and blocking network access processes; 6) Monitoring and feedback: Monitor the execution process of the rule engine in real time to ensure that the disposal measures are correctly executed; collect feedback information after the disposal measures are executed to make necessary adjustments and optimizations to the rules; 7) Rule optimization and update: Regularly review and optimize the rules based on monitoring results and feedback information; modify, add or delete rules.

2. The bad traffic monitoring method based on LSTM and clustering algorithm according to claim 1 is characterized in that: In step S33, the FCM algorithm is used to cluster the user behavior feature vectors, and users with similar behavior patterns are classified into one category. The specific implementation process is as follows: S331 to S334: S331: Data preprocessing and feature extraction In the session reassembly phase, the fragmented traffic in the cache needs to be reassembled to restore the complete network session. Then, key features are extracted from the reassembled session, including domain name, IP address, port number, URL path, and request method. S332: Initialization Determine the number of clusters c , randomly initialized c Cluster Centers v 1 ，v 2 ，...，v c ; Set the initial value of the membership matrix U, and set all elements to c A matrix of one-tenth of the clusters, representing all the clusters to which each data point may initially belong; setting the iteration stop conditions, including the maximum number of iterations or the objective function change being less than a certain threshold; S333: Iterative Optimization For each iteration, the following steps are performed: Update the membership matrix U: For each data point x j And each cluster center v i , update membership u ij , the formula is as follows: ； Among them, m represents the fuzzification parameter; x j Indicates j data points, which is a behavioral feature vector containing the eigenvalues ​​of each data point; v i Indicates i A cluster center, which is also a vector, indicating the center position of the cluster; v k Indicates k Cluster centers are used as comparison objects when calculating membership; Indicates j Data points x j To cluster center v i The Euclidean distance of Indicates j Data points x j To k Cluster Centers v k The Euclidean distance of Update cluster center V: For each cluster center v i , update its position according to the membership matrix, the cluster center feature vector v i The calculation formula is: ； in, u ij Indicates j Data points for i The membership degree of a cluster, n is the total number of users, express u ij of m Power; S334: Convergence judgment Check the change of the objective function or whether the maximum number of iterations has been reached. If the stopping condition is met, the iteration ends.

3. The bad traffic monitoring method based on LSTM and clustering algorithm according to claim 1 is characterized in that: In step S3, the decoupled feature library association is also used in the session reorganization and feature matching, and the specific contents include: (1) Feature library modularization First, different types of feature libraries are set as independent modules; each module contains a specific feature set and the matching logic associated with it; (2) Standardization of feature library interface Define unified interface standards for interaction between each feature library module and other parts of the system; these interfaces include feature loading, feature matching, and feature update functions; Loading interface: used to load feature data from feature library files; Matching interface: matches the provided network traffic features in the feature library and returns the matching results; Update interface: used to load new features into the system when the feature library is updated; (3) Feature matching process integration In the session reorganization and feature matching stage, the decoupled feature library association process is integrated into the system. The specific steps are as follows (31)~(34): (31) After the conversation is reorganized, extract the key features of the conversation; (32) According to the feature type, call the matching interface of the corresponding feature library module; (33) Collect the matching results of each feature library module and make a comprehensive judgment; if a session has matching items in multiple feature libraries, it may mean that the session involves multiple types of malicious behaviors; (34) Based on the matching results, execute corresponding disposal measures, including marking the session, blocking the connection and / or sending a warning; (4) Feature library configuration and management Configuration management tools allow system administrators to configure the use of signature libraries according to actual needs; Enable / disable signature library: The administrator can enable or disable specific signature library modules according to the current security threat situation; Priority setting: Set priorities for different signature library modules so that when multiple signature libraries have matching items, they will be processed in order of priority; Dynamic update: Supports dynamic update of feature libraries. When new malicious traffic features are discovered, these features are added to the corresponding feature libraries in a timely manner and loaded into the system through the update interface. (5) System monitoring and feedback Monitor the usage of the feature library, collect key indicators such as matching success rate and false alarm rate, and optimize and adjust the feature library based on these indicators; Performance monitoring: monitor the performance of feature matching to ensure that the system can maintain stable operation under high load conditions; Result feedback: The matching results and the effects of the disposal measures are fed back to the feature library maintenance team so as to optimize the quality and accuracy of the feature library based on the feedback.

4. A bad traffic monitoring system based on LSTM and clustering algorithm, realizing the bad traffic monitoring method based on LSTM and clustering algorithm described in any one of claims 1 to 3, characterized in that: It includes traffic aggregation module, fragmentation behavior identification and traffic caching module, session reorganization and feature matching module and disposal execution module; The traffic aggregation module collects website traffic data on the network server or proxy server, aggregates and analyzes the traffic data in real time using BPF, and provides a capability foundation for traffic segmentation and malicious feature behavior detection; The fragmentation behavior identification and traffic caching module: uses LSTM to filter out traffic with fragments according to the fragmentation identifier in TCP transmission and put it into the cache; The session reorganization and feature matching module: uses the FCM algorithm to reorganize the sharded sessions in the cache, and extracts the domain name, IP, port, and URL features, collides with the front-end feature library, and sends execution information to the execution unit when it is confirmed to meet the conditions; The processing execution module starts processing execution after receiving the object information and the execution action to be executed.

Images