A big data computer network security protection method

By segmenting and storing network traffic data and encoding its features, and combining Bloom filters and LSTM neural networks, the inefficiency of existing network security detection systems under large-scale traffic is solved, enabling efficient detection of complex attacks and zero-day vulnerabilities.

CN117938518BActive Publication Date: 2024-11-05SHUORUI (NANTONG) INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410118707.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-29
Publication Date
2024-11-05
Estimated Expiration
2044-01-29

AI Technical Summary

Technical Problem

Existing network security detection systems are inefficient when faced with large-scale network traffic, making it difficult to quickly identify complex attacks and zero-day vulnerabilities. Traditional rule-based matching methods suffer from delays in large-scale virus database queries and data transmission.

Method used

The network traffic data is fragmented and stored in the distributed file system Hadoop using a hash modulo algorithm. The hash algorithm is combined to generate unique identifier codes for source IP and destination IP, and a Bloom filter is used for fast IP filtering. Behavioral analysis is performed using an LSTM neural network, and traffic features are extracted through a sliding time window and feature encoding to build an efficient network security detection model.

Benefits of technology

It improves the efficiency and accuracy of network security detection, enables rapid identification and response to abnormal traffic, reduces computing resource requirements, enhances the ability to detect complex attack patterns, and improves the robustness and adaptability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117938518B_ABST
    Figure CN117938518B_ABST
Patent Text Reader

Abstract

The present application discloses a method for protecting the network security of a big data computer, which relates to the field of data security technology, including: obtaining network traffic data; slicing the obtained network traffic data using a hash modulo algorithm, and using the sliced ​​network traffic data as a first data set; copying the first data set to obtain a second data set; using a distributed file system to store the first data set, and using a database to store the second data set; extracting the source IP and destination IP from the first data set as traffic features; using a hash algorithm to generate a corresponding unique identification code, and adding the unique identification code to the first data set; signature matching based on black and white lists and behavior analysis based on LSTM neural networks; according to the detection results, querying the data records corresponding to the risk traffic data in the second data set, and performing TCP interception or source IP speed limit processing. In view of the problem of low network security detection efficiency existing in the prior art, the present application improves the detection efficiency of network security.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data security technology, and in particular to a method for protecting big data computer network security. Background Art

[0002] Early security systems relied heavily on signature-based detection, an approach that struggles to identify zero-day vulnerabilities and sophisticated attacks. Moving to behavior-based detection models allows for a more dynamic response to emerging threats. Incorporating machine learning algorithms into cybersecurity enables systems to learn from data patterns and predict or identify anomalies that indicate security breaches.

[0003] Traditional security systems often face challenges in detection efficiency, especially when scaling to accommodate large amounts of network traffic. Many existing solutions are not designed to handle the large and growing amounts of data generated by modern networks, resulting in slower analysis and response times. On-premises solutions require significant hardware investments and are limited by available computing resources, affecting their ability to efficiently process big data.

[0004] In the related art, for example, Chinese patent document CN117407879A provides a computer Internet information security control system based on big data analysis; the computer Internet information security control system based on big data analysis includes a central control system, which is used to transmit, manage and record the data in the computer Internet information security control system. The present invention collects virus data and defense data in the existing computer operation network through big data, and establishes a virus database and a defense database based on the data, and performs real-time comparison and detection on the network data, that is, comparing the data with the data in the virus database, and if there is a similarity, the corresponding defense data is immediately retrieved. However, this solution mainly relies on rule matching. In real-time comparison and detection, if it relies on a large-scale virus database for matching, delays are caused by database query and data transmission, especially when the database scale continues to grow, so the detection efficiency of this solution needs to be further improved. Summary of the invention

[0005] 1. Technical problems to be solved

[0006] In response to the problem of low network security detection efficiency in the prior art, the present application provides a big data computer network security protection method, which improves the network security detection efficiency through sharding storage and hierarchical detection.

[0007] 2. Technical solution

[0008] The purpose of this application is achieved through the following technical solutions.

[0009] The embodiment of the present specification provides a big data computer network security protection method, including: obtaining network traffic data, the network traffic data includes network logs, system logs and alarm information; using a hash modulo algorithm to slice the obtained network traffic data, and using the sliced ​​network traffic data as a first data set; copying the first data set to obtain a second data set; using a distributed file system to store the first data set, and using a database to store the second data set; extracting a source IP and a destination IP from the first data set as traffic features; using a hash algorithm to generate a corresponding unique identification code based on the extracted source IP and destination IP, and adding the unique identification code to the first data set; performing signature matching based on a black and white list and behavior analysis based on an LSTM neural network on the first data set after the unique identification code is added; according to the detection results, querying the data records corresponding to the risk traffic data in the second data set, and performing TCP interception or source IP speed limit processing.

[0010] Among them, the hash algorithm is an algorithm that maps an input of arbitrary length to an output of fixed length through a hash function. Common hash algorithms include MD5, SHA1, etc. Modulo operation, taking the remainder of a number, the result is the divisor and the remainder. Hash modulo: refers to first generating a hash value through a hash algorithm, and then performing a modulo operation on the hash value to map data of arbitrary length to a value within a specified range. In this application, network traffic data is obtained, including network logs, system logs, alarm information, etc. The hash modulo algorithm is applied to the obtained network traffic data. First, a hash value of a fixed length is generated through a hash function, and then the hash value is modulo to calculate the modulus, that is, the index value mapped to a fixed range. According to the index value, the traffic data is stored in shards. That is, the data mapped to the same index value is assigned to the same shard. The data after sharding is stored as the first data set for subsequent processing and analysis. The hash modulo algorithm can be used to evenly map network traffic data of variable length to a fixed number of buckets or shards, thereby realizing distributed storage and processing of traffic and fast sharding of large-scale network traffic data.

[0011] Among them, the distributed file system is a file system built on the network, which stores data distributed on multiple computer nodes, and each computer node stores part of the data. The distributed file system has the characteristics of high scalability and high fault tolerance. In this application, the distributed file system Hadoop is used to store the first data set. Hadoop adopts the HDFS architecture, which can store and manage a large amount of data on multiple physical machines and provide high-throughput data access. The data after network traffic slicing is used as the first data set and stored on HDFS. HDFS will automatically back up data blocks to multiple servers. Distributed storage improves the concurrent access capability of large-scale traffic data and supports parallel analysis such as traffic feature extraction and signature matching. At the same time, the second data set is stored in the form of a relational database MySQL. MySQL provides efficient data indexing and query functions. The database stores the SECOND data set to support fine-grained data access after signature matching, such as extracting information such as source IP and destination IP. Through the joint use of distributed file systems and databases, the solution realizes efficient storage, access and management of large-scale network traffic. The first data set uses HDFS to improve storage and processing capabilities, and the second data set uses MySQL to provide support for fast query and matching.

[0012] Among them, the source IP, the IP address of the host that sends the data packet, identifies the sender of the data packet. The destination IP, the IP address of the host that receives the data packet, identifies the receiver of the data packet. In the present application, traffic data packets are extracted from the first data set stored in the distributed file system HDFS. For each traffic data packet, the source IP and destination IP information contained therein are extracted. The source IP can identify the host or network source that generates the traffic. The destination IP can identify the receiver of the traffic. The source IP and destination IP are used as features of the traffic packet to indicate the source and destination of the traffic. In subsequent traffic analysis, the source IP and destination IP will be used to identify traffic, match black and white lists, perform behavioral analysis, etc., and are important features of network traffic analysis.

[0013] Furthermore, signature matching based on black and white lists includes the following steps: obtaining a locally stored IP address library, the IP address library containing a pre-marked normal IP address set and a malicious IP address set; encoding the IP address library using a double hash mapping function to generate a corresponding Bloom filter IP library; extracting the destination IP from the first data set; for the extracted destination IP, obtaining the corresponding first index value and second index value from the Bloom filter IP library using a double hash mapping function; querying in the Bloom filter IP library whether the index positions corresponding to the first index value and the second index value are marked at the same time, the mark indicating that the corresponding position in the Bloom filter IP library bit array of the index obtained by the double hash mapping function is set to 1; if the index positions corresponding to the first index value and the second index value are both marked as 1, searching the IP address library for the marking attribute of the corresponding IP; if any index corresponding position is not marked, marking the corresponding network traffic data as traffic to be detected.

[0014] Among them, the normal IP address set refers to the set of IP addresses that have been security verified and are trustworthy. The malicious IP address set refers to the set of IP addresses that are known or suspected to have malicious behavior (such as viruses, Trojans, botnets, etc.). In this application, a local IP address library is pre-established, which contains two IP address sets: normal and malicious. The normal IP address set can be obtained through the accumulation of whitelists, including reliable source IPs such as internal systems and partners. The malicious IP address set can be obtained through the shared blacklist of security vendors, attack source analysis, etc. When performing traffic analysis, the source / destination IP can be matched with the IP address library first to determine its classification. If the source / destination IP is in the whitelist, it is marked as normal traffic; if it is in the blacklist, it is marked as suspicious malicious traffic. The IP address library supports rapid filtering and classification of traffic, distinguishes normal and potentially malicious traffic, and is used for traffic assessment and risk control.

[0015] Among them, the double hash mapping function refers to the use of two different hash functions to perform hash calculations on the same input respectively to obtain two different hash values. Hash function: a function that maps input data of arbitrary length to a numerical output of fixed length. Mapping function: a function that defines the correspondence between input data and output data. In the present application, for each IP address in the IP address library, two different hash functions H1 and H2 are used to perform hash calculations. The two hash values ​​corresponding to the IP address are obtained to form a double hash mapping of the IP address. All IP addresses are mapped to hash value pairs using double hashing to generate a Bloom filter IP library. In subsequent traffic IP matching, double hash calculations are also performed on the query IP. If both hash values ​​can be found in the Bloom filter IP library, it means that the original IP is in the address library. Double hash mapping can greatly reduce conflicts and reduce the misjudgment rate.

[0016] Specifically, build a local IP address library, including all IP addresses that need to be matched, and mark the attributes of the IP addresses (such as normal or malicious). Select two different hash functions H1 (IP) and H2 (IP) to generate two different sets of hash values. For example, MD5 and SHA1. For each IP address IP in the IP address library, calculate: H1_Hash = H1 (IP), H2_Hash = H2 (IP), and determine the length L of the Bloom filter array according to the number of IP addresses N that need to be matched. Generally, L = N*10. Allocate an L-bit bit array and initialize all bits to 0. For each IP address IP, after calculating H1_Hash and H2_Hash: set the position 1 corresponding to H1_Hash in the bit array to 1. Set the position 1 corresponding to H2_Hash in the bit array to 1. Finally, the encoded Bloom filter IP library is obtained, which stores a compressed bit array after double hash encoding. It can be used to quickly query whether the IP is in the original address library.

[0017] Specifically, extract the destination IP from the network traffic, assuming it is DIP. Perform double hash calculation on DIP: H1_Hash=H1(DIP), H2_Hash=H2(DIP), where H1 and H2 are the same double hash functions used when constructing the Bloom filter. In the bit array of the Bloom filter: H1_Hash is the first index value Index1 corresponding to DIP; H2_Hash is the second index value Index2 corresponding to DIP. Determine whether the bits at the Index1 and Index2 positions in the bit array are 1. If the bits on Index1 and Index2 are 1 at the same time, it means that DIP is in the original IP address library and the match is successful. If the bit on any Index is 0, DIP is definitely not in the IP address library and the match fails. When both bits are 1, it is necessary to further search in the original IP library for an exact match. By calculating double hashes for the traffic IP, you can quickly query whether the IP is in the address library in the Bloom filter and filter out a large number of unmatched IPs. If both index bits hit, further precise matching is required to prevent misjudgment caused by hash conflicts.

[0018] The double hash index values ​​Index1 and Index2 of the traffic IP have been obtained. Search the bits corresponding to Index1 and Index2 in the bit array of the Bloom filter. Determine the values ​​of these two bits: If the bits on Index1 and Index2 are both 1, it means that the double hash of the IP has marked the bit array; if the bit corresponding to any index is 0, it is confirmed that the IP is definitely not in the original address library; only when both bits are 1, it indicates that the IP is in the address library. The reason is that when constructing the Bloom filter, the double hash is calculated for each address library IP, and the bits corresponding to the two indexes are set to 1. Therefore, as long as any index position is 0, it can be determined that the query IP is not in the address library. Only when both index positions are set to 1, further precise matching is required because there is a hash conflict. By querying the status of the two bits, the filtering of the traffic IP can be completed quickly to determine whether precise matching is required. By checking the marking status of the two hash indexes, fast IP filtering query based on Bloom filtering is realized, and query efficiency is improved.

[0019] The double hash index of the traffic IP has been queried, and the two bits Index1 and Index2 corresponding to the Bloom filter are checked. Judge the two bits on Index1 and Index2: If both bits are 1, it means that the double hash of the IP is hit, and further precise matching is required; find the IP in the original IP address library, obtain its tag attribute, and determine whether it is a normal IP or a malicious IP; according to the tag attribute, mark the traffic of the IP as known normal traffic or risk traffic; if the bit corresponding to any index is 0, it can be directly determined that the IP is not in the address library; directly mark the traffic of the IP as traffic to be detected, and further judgment is required through other models. The reason is that only when both hash bits are 1, it indicates a successful match, and further precise judgment is required. If any bit is 0, it directly indicates a mismatch, which can avoid unnecessary precise matching and improve efficiency. Mark the mismatched traffic as traffic to be detected for further security analysis. By combining double hash index query with precise matching, traffic IPs can be quickly filtered, and traffic can be classified and marked to provide input for subsequent detection and analysis.

[0020] Furthermore, when the index positions corresponding to the first index value and the second index value are both marked in the Bloom filter IP library, the marking attribute of the corresponding IP is searched in the local IP address library; if the IP marking attribute found is a malicious IP, the network traffic corresponding to the IP is marked as risk traffic; in the database of the second data set, the corresponding data record is queried according to the unique identification code of the risk traffic; for the queried data records of the risk traffic, the source IP and destination IP are extracted; for the extracted source IP, source IP speed limit or traffic isolation is performed; for the extracted destination IP, TCP interception is performed.

[0021] Among them, the double hash indexes Index1 and Index2 of the traffic IP that have been queried are both 1 in the corresponding bits of the Bloom filter. This means that the double hash of the IP is hit, and it is necessary to perform an exact match in the original IP address library to find the tag attribute of the IP. Search the IP in the IP address library and determine its tag attribute: if the IP is found and it is marked as a malicious IP or a blacklist IP, it can be determined that the IP is a risk source; at this time, the network traffic corresponding to the IP is marked as risk traffic and added to the result set. If the IP is found but marked as a normal IP or a whitelist IP, the traffic is kept in a normal state. If the IP is not in the address library, the traffic is kept in a pending detection state. By querying the attributes in the known IP address library, the risk level of the traffic can be determined and the risk traffic can be marked. The risk traffic will be added to the result set for further processing, including source IP speed limit, TCP interception, etc.

[0022] Among them, during traffic processing, a unique identification code has been generated for each traffic and stored in the first data set and the second data set. When a certain traffic is judged as risk traffic based on the IP address library, the unique identification code of the traffic is recorded. In the MySQL database of the second data set, the unique identification code is used as the query condition to retrieve the corresponding traffic data record. Since the first data set and the second data set correspond to the same traffic, the data record of the risk traffic in the second data set can be found. For the queried risk traffic record, the source IP and destination IP information are further extracted. The source IP represents the source address of the threat traffic, and the destination IP represents its attack target or victim address.

[0023] Among them, the following controls can be performed on the source IP: Source IP speed limit: Use network equipment to limit the speed of traffic containing the source IP to reduce the intensity of attack traffic; Source IP isolation: Isolate the port or VLAN corresponding to the source IP to prevent it from accessing key assets. For the destination IP, TCP interception can be performed: Obtain the gateway switch or firewall of the subnet where the destination IP is located; Configure the access control list on the device to discard the TCP traffic from the source IP to the destination IP; Intercept the TCP connection to prevent the attack connection from reaching the target server. Source IP control directly reduces the attack traffic. Destination IP interception prevents the attack connection from being successful. Depending on different situations, source IP speed limit, isolation or destination IP TCP interception can be used to process risky traffic. Effectively deal with malicious traffic and ensure network security. Through different control strategies for source and destination addresses, the harm of malicious traffic can be prevented and mitigated in a targeted manner to protect network and asset security.

[0024] Furthermore, the behavior analysis based on the LSTM neural network includes the following steps: obtaining the traffic to be detected; using a sliding time window to divide the traffic to be detected into sub-flow batches containing preset traffic data packets; extracting the source IP, destination IP, port number and timestamp of each sub-flow batch of the traffic data packet; encoding the extracted source IP and destination IP using a hash function to obtain a hash code; encoding the extracted port number using one-hot encoding to obtain a category vector; concatenating the obtained hash code, category vector and timestamp to obtain a coded digital vector of the traffic data packet; arranging the coded digital vectors of the traffic data packets in the time window in timestamp order to form a time series sample sequence as input data of the LSTM neural network;

[0025] Among them, the traffic to be detected is obtained by filtering out the network traffic. The length of the sliding time window is set, for example, 1 minute. The time window is smoothly slid, the traffic is intercepted in time order, and all traffic data packets in the current window are obtained. The traffic packets obtained in the current window are formed into a sub-flow batch. The sliding window is translated forward according to the time step (for example, 10 seconds). The traffic packet at the next window position is obtained to form a new sub-flow batch. Repeat this process until the sliding window covers all the traffic to be detected. Finally, the entire traffic to be detected is divided into multiple sub-flow batches with a fixed number of traffic packets in time order. Each sub-flow batch represents a set of traffic within a period of time, retaining the time characteristics of the traffic. Subsequently, the characteristics of the traffic packets can be extracted in units of sub-flow batches for model detection and analysis. Through sliding window division, the continuity of the time characteristics of the traffic is guaranteed, and the traffic processing can be better parallelized.

[0026] Among them, multiple sub-flow batches divided by time sliding windows have been obtained, and each sub-flow batch contains traffic packets within a period of time. For each traffic packet in each sub-flow batch, the four traffic features of source IP, destination IP, port number and timestamp are extracted. Encode the source IP and destination IP: set the hash function H (IP), which can be an algorithm such as MD5; calculate the source IP: source IP hash code = H (source IP); calculate the destination IP: destination IP hash code = H (destination IP); get the hash code of the source IP and destination IP. In this way, each traffic packet obtains 4 encoding features: source IP hash code, destination IP hash code, port number, timestamp; extract and encode features for all traffic packets in each sub-flow batch. Subsequently, based on these digital features, a detection model can be established to detect abnormal traffic. By extracting and encoding the key features of the traffic packet, the digital expression of the traffic can be obtained as the input for modeling and calculation.

[0027] Among them, the port number field is extracted from each traffic packet. The value range of the port number is analyzed. For example, there are 65536 values ​​from 1 to 65535. The vector dimension of the one-hot encoding is determined according to the number of values, which is 65535 bits in this case. For each port number value, the position corresponding to its value in the one-hot vector is set to 1, and the remaining positions are set to 0. For example, for port number 80, the 80th bit in the one-hot vector is set to 1, and the remaining 65534 bits are set to 0. In this way, each port number can be encoded into a 65535-dimensional one-hot vector. Only one bit in the vector is 1, and this position represents the value of the port number. The port number field of all traffic packets is encoded into a one-hot vector. Finally, each traffic packet obtains a port number encoding represented by an integer. The encoding method of different ports corresponding to different bits in the sparse vector is convenient for model processing. Through the one-hot technology, the category features of the port number are digitized and encoded into a vector expression that can be distinguished by the model. The field meaning of the port number is retained.

[0028] For each traffic data packet, the following have been extracted and encoded: hash codes of the source IP and destination IP, such as 64-bit length; one-hot codes of the port number, such as 65535-dimensional vectors; timestamps, such as 64-bit numbers. The above three parts are simply spliced ​​together to obtain a complete digital encoding vector for each traffic packet. Specifically, it can be: [source IP hash code, destination IP hash code, port number one-hot code, timestamp], so that each traffic packet is encoded as a fixed-length digital vector. The vector contains the source address, destination address, port and timestamp information of the traffic packet. The digital vectors of different traffic packets can be compared and modeled. The digital vectors of all traffic packets in the time window are organized to form a time series sample. Simple splicing codes combine different encoding features to form an overall digital expression of the traffic packet. By simply splicing the encoding of each feature, a digital vector containing complete traffic information is formed, laying the foundation for traffic detection modeling.

[0029] Perform feature extraction and encoding on all traffic packets in each time window to obtain source IP / destination IP hash codes, port number one-hot codes, and timestamps. Sort the traffic packets in the time window from small to large timestamps. Connect the encoding vectors of all traffic packets in the same window in the order of timestamps. Form a matrix in which each row is the encoding vector of a sample traffic packet. The number of rows in the matrix is ​​the total number of traffic packets in the time window, and the number of columns is the encoding vector dimension. This matrix is ​​the time series digital sample of the traffic in the entire time window. The order of the traffic packets in the matrix is ​​the actual time order. Use this time series sample matrix as the input of the LSTM model. LSTM can analyze the feature changes at different time steps in the matrix and detect the time correlation therein. Through the above transformation, the actual network traffic is encoded into time-correlated digital samples to be input into the LSTM model for time series modeling and analysis.

[0030] Furthermore, the behavior analysis based on the LSTM neural network includes the following steps: constructing an LSTM neural network including an input layer, a convolution layer, an LSTM layer and an output layer; setting a batch normalization layer after the input layer to linearly scale the input data obtained by the input layer to eliminate the mean shift and variance shift of the input data; setting a one-dimensional convolution layer after the batch normalization layer to perform one-dimensional convolution on the data output by the batch normalization layer to extract local features of the time series; setting a residual connection between adjacent LSTM layers to add the input features and output features of the previous LSTM layer as the input of the next LSTM layer; setting a self-attention layer after the last LSTM layer and before the output layer to learn the features of input data at different time steps; setting a dropout layer after the self-attention layer and before the output layer to randomly suppress the neurons in the input layer according to random probability;

[0031] Among them, mean shift refers to the degree to which the mean of each feature dimension of the input data deviates from 0. The less the mean of each dimension of the input data is close to 0, the greater the mean shift. Variance shift: the degree to which the variance of each feature dimension of the input data deviates from 1. The less the variance of each dimension of the input data is close to 1, the greater the variance shift. In the present application, the input layer obtains the encoded digital features of the traffic; there are obvious mean and variance shifts in different feature dimensions; in order to eliminate the impact of this shift on model training, a batch normalization layer is added after the input layer; the batch normalization layer performs mean-variance normalization on the input features of each batch, forcing the mean of the feature to be close to 0 and the variance to be close to 1; thereby reducing the interference of the feature shift on the model training and accelerating the convergence of the model. Through batch normalization processing, the shift of the input data in the mean and variance can be eliminated, making the model more stable and efficient to train.

[0032] Among them, local features: subsequence patterns formed by data points of adjacent time steps in the time series, reflecting local time correlation. In this application, the input layer inputs a digital sequence encoded from the traffic time series; the batch normalization layer outputs digital time series data; the convolution kernel of the one-dimensional convolution layer can slide and scan the time series to extract local patterns formed by adjacent data between time steps; different convolution kernels can extract local features of different ranges and semantics in the time series; these local features reflect the behavior pattern of traffic in the local time period; finally, the local features are combined through maximum pooling to obtain the overall traffic characteristics within the time window; these local and global features are used to represent the traffic time series for anomaly detection. By extracting time-related local features through one-dimensional convolution, the time series pattern of traffic can be better learned and represented, which is helpful for the detection of abnormal traffic.

[0033] Among them, the characteristics of input data at different time steps refer to the time series feature representation obtained after LSTM encodes the input sequence at different times. In this application, the LSTM layer will process the time series input in sequence and output the feature code at each moment. The feature codes at different moments contain different time step information; the self-attention layer can learn the importance weights of different time series codes; weight different time series features, emphasize important features, and filter out irrelevant features; obtain feature codes that are representative of the overall pattern of the time series; these encoding features reflect the correlation of inputs at different time steps and their contribution to the sequence as a whole; as the overall characteristics of the traffic time series, they are used for the final anomaly detection prediction.

[0034] Among them, random inhibition, during the training process of the neural network, randomly makes some neurons stop working according to a certain probability, and does not participate in model training and parameter updating. In this application, after the self-attention layer, the feature representation encoding the entire time series will be obtained; these feature codes will be used as the input of the output layer; a random drop layer is added between the two layers to randomly suppress the feature dimensions with a certain probability; that is, some features are not working during the training process; a certain random disturbance is achieved to enhance the generalization ability of the model; prevent the features learned by self-attention from being too close to the current traffic, and the adaptability to new traffic is reduced; improve the model's detection and generalization ability of external new traffic. By randomly inhibiting neurons, the model can be regularized to improve its detection and adaptability to unknown new traffic.

[0035] Furthermore, constructing a training set includes the following steps: extracting a set of IP addresses marked as normal from a local database as a first sample sequence; for the traffic sequence in the first sample sequence, using a random downsampling algorithm to generate a second sample sequence of time step sampling; for the traffic sequence in the second sample sequence, using a dynamic time folding algorithm to adjust the time step position to generate a third sample sequence; for the traffic sequence in the second sample sequence, adding random noise to simulate the jitter of normal traffic to generate a fourth sample sequence; combining the traffic sequences in the first sample sequence, the second sample sequence, the third sample sequence and the fourth sample sequence to form a sample sequence data set; encoding and converting the obtained sample sequence data set to obtain a digital vector sequence as a training set for the LSTM neural network.

[0036] Among them, the first sample sequence has been obtained, which contains encoded traffic data arranged in time order. Set the parameters of the random downsampling algorithm, including the sampling probability p and the minimum sampling ratio r. Traverse the first sample sequence, and randomly decide whether to sample the traffic data of each time step with probability p. Generate a binary random sequence, in which 1 indicates that the corresponding time step is sampled and 0 indicates that it is discarded. At least r proportion of data is guaranteed to be sampled, and no lower limit is set. According to the random binary sampling sequence, the sampled time steps are retained from the first sample sequence, and the discarded time steps are deleted. Obtain a second sample sequence with randomly sampled time steps. The second sample sequence retains part of the time data of the first sequence, but disrupts the original time order. Repeating random downsampling multiple times can obtain more samples with disturbed traffic time steps. These new samples can enhance the robustness of the model to time series anomalies and improve the detection ability. By disrupting the time order through the random downsampling algorithm, more diverse traffic time series samples can be obtained, which enhances the detection ability of the model.

[0037] Among them, the dynamic time folding algorithm is a data enhancement method that can randomly adjust the position of each time step in the time series sample and destroy the original time order. In this application, the second sample sequence is a sequence obtained by random downsampling; dynamic time folding is further used for the second sample sequence; the algorithm randomly generates a new position of the time step within the sequence length range; according to the generated new position, the data order of each time step in the sequence is adjusted; the third sample sequence with the time step order dynamically disrupted is obtained; multiple times of time folding are repeated to obtain more new samples with chaotic time steps; this further enhances the model's robustness in detecting abnormal time series. Through dynamic time folding, more traffic samples with destroyed time order can be obtained, improving the generalization ability of the model.

[0038] Among them, the second sample sequence obtained by random downsampling has been obtained. Set the distribution model of random noise, such as a normal distribution with a mean of 0 and a standard deviation of 0.01. According to the set noise distribution, add a random noise value to the traffic feature vector of each time step in the second sample sequence. Random noise can act on each dimension of the traffic vector to cause a certain disturbance in the value of each dimension. Repeat the adding process multiple times to obtain the fourth sample sequence in which the traffic feature vector is added with random noise. The added random noise can simulate various random jitters in normal network traffic. Make the model more accustomed to handling complex and changeable actual traffic environments. Improve the accuracy and generalization ability of the model to judge whether new traffic is normal or abnormal. The model obtained by training will be more robust and accurate. By adding random noise to simulate the dynamic changes of normal traffic, the model can make the judgment of normal traffic more accurate and improve the stability of abnormal traffic detection.

[0039] Among them, sample sequences generated by four processing methods have been obtained: the first sample sequence: the original time series, the second sample sequence: the random downsampling sequence, the third sample sequence: the dynamic time folding sequence, and the fourth sample sequence: the random noise added sequence. The above four sample sequences are mixed and combined in a certain proportion. For example, they are combined in a ratio of 2:1:1:1. Together, they constitute the final sample sequence data set as the model training set. Encode the sample sequence data set: Each sample sequence has been subjected to traffic feature extraction and encoding. Check the digital encoding format of the sample data again. Ensure that the training samples are represented in the form of digital vector sequences. The encoded sample data is directly used as the training set input of the LSTM model. Generate sample sequences through a variety of enhancement methods to construct a training set containing rich features. Improve the model's ability to learn and detect abnormal traffic. By combining different preprocessed sample sequences, a traffic training set containing complex and rich features is constructed for effective model training.

[0040] Furthermore, generating a third sample sequence includes the following steps: setting a maximum time distance threshold for merging traffic data packets; calculating the time step distance between each traffic data packet in the second sample sequence and other traffic data packets; merging multiple traffic data packets whose calculated time step distances are within the set maximum time distance threshold into one traffic data packet; inserting the merged traffic data packet into the time step position corresponding to the second sample sequence; repeating the above steps until the time step distances of all traffic data packets in the sample sequence are greater than the set maximum time distance threshold, and obtaining a sample sequence as the third sample sequence.

[0041] Among them, the maximum allowed merging time distance threshold is set, for example, set to 5. For each traffic data packet in the second sample sequence, calculate the time step distance between it and each other traffic packet in the sequence. For example, traffic packet A and traffic packet B are separated by 3 time steps, which is less than the threshold 5, and can be considered for merging. Each traffic packet and other traffic packets whose time step distance is within the threshold 5 are put into a set to be merged. For each set to be merged, multiple traffic packets therein are merged into a new traffic data packet. The merging method is to directly average the feature vectors of each traffic packet to obtain a new traffic packet vector. Replace each traffic packet in the original set to be merged with the new traffic packet generated after the merger. Finally, a new sample sequence is obtained in which the data packets with a time step distance within the threshold are merged. Merging time-related traffic packets can enhance the model's ability to learn the temporal relationship between traffic. By setting the maximum time distance threshold and merging traffic packets that are nearly related in time, the model's ability to learn traffic time-related features can be enhanced.

[0042] Among them, the traffic packet merging operation has been performed on the second sample sequence. Each merge will generate a new traffic packet, which contains the features of multiple original traffic packets. The generated new traffic packet is inserted back into the second sample sequence. The insertion position is the position of the traffic packet with the earliest time step in the original set to be merged. This ensures the consistency of the time order. Repeat the traffic packet merging and insertion operations. Until the time step distance between all traffic packets in the second sample sequence is greater than the set threshold. At this time, no more merging can be performed. Finally, the third sample sequence with a time step distance exceeding the threshold is obtained. It contains traffic packet features that merge time associations. These new training samples can better characterize the time correlation of traffic. By continuously merging and inserting traffic packets, a new sample sequence with enhanced time correlation can be obtained, which is used to train the model to learn traffic time characteristics.

[0043] Furthermore, the obtained training set is used to train the constructed LSTM neural network, including the following steps: using the generated training set as input, and inputting it into the constructed LSTM neural network for model training; calculating the predicted output value of each sample data and the loss function value of the corresponding label; using a chain algorithm, respectively calculating the partial derivatives of the weight parameters and the partial derivatives of the bias parameters of the loss function value of each sample data with respect to the sample passing through all nodes in the LSTM neural network; updating the node parameters in the LSTM neural network through a gradient descent algorithm based on the calculated partial derivatives of the weight parameters and the partial derivatives of the bias parameters; repeating the above steps until the loss function value on the training set is less than or equal to a threshold or reaches a preset number of training times.

[0044] Among them, the LSTM neural network model architecture has been constructed, including the input layer, LSTM layer, fully connected layer and output layer. The generated encoded digital vector sequence is used as the training set input. Each sample sequence has a normal or abnormal label. Through forward propagation, the model generates a predicted output value for each sample, such as a probability distribution. The cross entropy loss function is calculated by comparing the predicted output value with the true label of the sample. The cross entropy function measures the distance between the predicted output distribution and the true label distribution. The loss function value of each sample is calculated. The loss of all training set samples is summed and averaged to obtain the total loss function. The model parameters are updated through the back propagation algorithm to minimize the loss function. Repeat the forward propagation and back propagation processes to gradually optimize the model and improve the accuracy of traffic prediction. Until the network converges or reaches the preset number of times. The LSTM model is trained through the process of inputting training samples, forward propagation, calculating losses, and reverse optimization, and the model parameters are continuously optimized to improve the prediction ability of traffic sequences.

[0045] Among them, the chain rule, in the neural network, can be used to obtain the gradient of the loss function with respect to the weights and bias parameters of each layer of nodes. In this application, forward propagation calculates the loss function value of each sample; the model parameters need to be updated through back propagation to minimize the loss function; the application of the chain rule can efficiently calculate the derivative of the loss function with respect to the network parameters; that is, the parameter derivatives of each layer are calculated in turn, and the derivatives of each layer are chain-multiplied to obtain the final gradient; this avoids the high amount of calculation required to directly calculate the gradient of the entire network parameters; the chain derivative calculation can quickly and efficiently perform the back propagation of the loss function gradient; update the weights and biases of each layer of nodes in the LSTM model, and complete a parameter optimization. Through the gradient efficiency calculation of the chain rule, the training process of the LSTM model can be accelerated, so that it converges faster to obtain better parameters, thereby improving the effect of traffic detection.

[0046] Among them, the gradient of the loss function for each layer weight and bias parameter has been calculated by the chain rule. The gradient indicates the direction of minimizing the loss function. The network parameters are updated using the gradient descent algorithm: θ=θ-η*∇θ, where θ is the weight or bias parameter, η is the learning rate, and ∇θ is the parameter gradient. The weight parameters are updated by element-wise subtraction: W[i, j]=W[i, j]-η*∂Loss / ∂W[i, j], and the bias parameters are updated by element-wise subtraction: b[i]=b[i]-η*∂Loss / ∂b[i]. The amplitude of the parameter update is adjusted according to the learning rate η. A gradient descent parameter update is completed. Repeat multiple update iterations to gradually minimize the loss function and optimize the LSTM model. When the loss function converges or reaches the number of iterations, the model training is completed. By combining the gradient descent algorithm with the chain rule gradient, the LSTM model can be effectively trained to improve the prediction effect of the traffic time series.

[0047] Furthermore, the distributed file system for storing the first data set uses Hadoop; the database for storing the second data set uses the MySQL relational database. The first data set: large-scale raw traffic data. Considering the storage of massive data, the Hadoop distributed file system is selected. The HDFS component of Hadoop is used to provide highly fault-tolerant distributed data storage. The raw traffic data is stored on the data node in the form of HDFS files. The Name Node is responsible for the management and access of the file directory tree. Provides high-throughput parallel data reading. The second data set: the data set after traffic feature extraction. It is small in scale and needs to establish an index for fast query. Select the MySQL relational database for storage. Store the feature data in a table and establish an index. Use SQL to query and manage sample data. Provide low-latency feature data access. Hadoop and MySQL are used together to store large-scale raw traffic data and feature extraction data sets respectively, combining the advantages of both to build an efficient traffic analysis platform.

[0048] 3. Beneficial effects

[0049] Compared with the prior art, the advantages of this application are:

[0050] The network traffic data is sharded using a hash modulus algorithm, and the sharded data is stored in a distributed file system (Hadoop). This helps improve the efficiency of parallel processing and reading of data, and supports fast access and analysis of large-scale network traffic. The second data set is stored in a database (MySQL), which provides efficient data retrieval and query capabilities. The use of a database helps to quickly find and match the second data set to respond to real-time network security events.

[0051] The unique identification code of the source IP and the destination IP is generated by the hash algorithm, which helps to quickly identify and locate the traffic characteristics. The introduction of the unique identification code improves the efficiency of retrieval and matching in large-scale data sets; the Bloom filter is used in the signature matching based on the black and white lists, and the IP address library is encoded through the double hash mapping function, so as to quickly identify normal and malicious IPs. This improves the ability to quickly detect and filter IP addresses;

[0052] Dividing traffic data into sub-flow batches through sliding time windows helps preserve the temporal characteristics of traffic data; the use of sliding time windows enables the system to analyze traffic in real time and capture abnormal behaviors in the network in a timely manner;

[0053] The convolutional layer can effectively extract local features from time series data. This helps capture spatial locality in traffic data, such as traffic patterns of specific ports, common attack features, etc. The LSTM layer can learn the timing information in traffic data through long short-term memory units. This enables the neural network to identify patterns with time correlation, such as gradually evolving network attacks or long-standing abnormal traffic. The combination of convolutional layers and LSTM layers enables a comprehensive analysis of spatiotemporal correlations. This helps to fully understand the patterns in traffic data and improves the accuracy of analysis and detection of network behavior.

[0054] Extracting a set of normal IP addresses from the local database can improve query efficiency and quickly obtain samples of normal traffic by using technologies such as database indexing; random downsampling helps balance the number of normal and abnormal samples, prevents the problem of class imbalance in the training set, and improves the adaptability of the model to various situations; dynamic time folding is equivalent to data augmentation of the original time series, which increases the diversity of the training set and improves the robustness of the model; the introduction of random noise can improve the model's anti-attack ability, making it more vulnerable to adversarial attacks and enhancing the model's usability in actual network environments; the combination of different processing methods makes the model more robust and able to cope with different network environments and attack methods;

[0055] By using blacklists and whitelists for preliminary screening, the system can quickly exclude known safe and unsafe traffic. This can significantly reduce the amount of data that needs to be further analyzed, thereby saving computing resources; blacklists can quickly identify and intercept known threats, speed up response time, and immediately mitigate potential security incidents; LSTM can understand the long-term dependencies between data points distributed in time, which helps to identify complex attack patterns that are difficult to detect by simple rule-based systems; hierarchical detection allows the system to focus resources on traffic containing unknown threats rather than on a large amount of known benign or malicious traffic; by pre-excluding a large amount of harmless traffic, LSTM models can run on smaller data sets, reducing the computing resources required for model training and reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0057] Figure 1 is an exemplary flow chart of a big data computer network security protection method according to some embodiments of this specification;

[0058] Figure 2is an exemplary flow chart of obtaining traffic to be detected according to some embodiments of this specification;

[0059] Figure 3 is an exemplary flow chart of constructing a time series sample sequence according to some embodiments of this specification;

[0060] Figure 4 is an exemplary flowchart of constructing a training set according to some embodiments of the present specification. DETAILED DESCRIPTION

[0061] The method and system provided in the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0062] Example 1

[0063] Figure 1 This is an exemplary flow chart of a big data computer network security protection method shown in some embodiments of this specification, data collection and acquisition: collect network traffic data, including network logs, system logs and alarm information. Deploy network traffic monitoring equipment or use network traffic monitoring software to capture data transmitted in the network in real time. Use network traffic analysis tools, such as Wire shark, to capture and analyze traffic. Ensure that network equipment is configured to generate detailed network logs, system logs and alarm information. Standardize the format of the collected network traffic data for subsequent processing. You can use standard log formats such as JSON or CSV. Set up a secure data transmission channel to ensure that the collected data is not tampered with or leaked during transmission. Transmit the collected data to a data processing center or storage server. Perform real-time processing during data transmission, such as filtering out unnecessary information, performing data cleaning, etc. Perform basic statistical analysis on real-time traffic to detect abnormal situations in a timely manner. Store the real-time processed data in a distributed file system, such as Hadoop. Ensure that the storage system has high availability, fault tolerance and scalability to cope with the storage needs of large amounts of data.

[0064] Data Sharding and Storage: Use hash modulo algorithm to shard network traffic data. The sharded data is stored as the first data set in a distributed file system (such as Hadoop). Choose a suitable hash modulo algorithm, such as consistent hashing or ordinary hash modulo, to ensure the uniformity and distribution of the shards. Consistent hashing is often used to better maintain the consistency of data distribution when nodes are dynamically added or removed. Perform hash modulo operation on network traffic data to distribute the data into different shards. This can be hashed by information such as source IP address, destination IP address, timestamp, etc. Ensure that the output range of hash modulo is suitable for the number of nodes in the distributed file system to fully utilize system resources. Define the sharded data set as the first data set for subsequent processing and querying. When defining the data set, record the identifier, range and other meta information of each shard for subsequent data retrieval and management. Store the sharded data in a distributed file system, such as Hadoop's HDFS (Hadoop Distributed File System). Ensure the stability and reliability of the Hadoop cluster and improve the fault tolerance of the data through mechanisms such as data replication. Set up a suitable data backup strategy to ensure timely recovery in case of node failure or data corruption. Consider implementing a data redundancy mechanism so that data is backed up between different nodes to improve system reliability. Implement a data management system to monitor and manage sharded data sets. Provide query interfaces or tools so that users can easily query and retrieve network traffic data for a specific shard. Use a hash modulus algorithm to shard network traffic data and store the sharded data in a distributed file system to achieve efficient management and utilization of large-scale network traffic data.

[0065] Data replication and backup: Replicate the first data set to obtain the second data set. The second data set is stored in a relational database (such as MySQL) to ensure data backup and persistence. Develop a replication strategy to determine the time interval and frequency of replication. This can be real-time replication, periodic replication, or event-triggered replication. Use appropriate replication tools or mechanisms, such as database replication functions, snapshots of distributed storage systems, etc., to ensure data consistency and integrity. Generate the second data set during the replication process to ensure that the second data set is an accurate copy of the first data set. Record the replication meta information as required, such as the timestamp of the replication, the replication source and target, etc. Configure a relational database, such as MySQL, to store the second data set. Ensure that the performance and availability of the database meet the requirements. Create appropriate table structures and indexes to support efficient query and retrieval of data. Configure regular database backups to ensure rapid recovery in the event of data loss or corruption. Consider adopting a full backup and incremental backup strategy to reduce the impact of the backup process on system performance. Ensure the persistence of the database to maintain data consistency even in abnormal situations such as system failures or power outages. Configure appropriate transactions and logging mechanisms based on the characteristics of the relational database to support the persistence requirements of transactions. Deploy a monitoring system to monitor the replication status of the second data set, database performance and other indicators in real time. Set alarm rules to detect and solve potential problems in a timely manner to ensure the stability and reliability of the system.

[0066] Traffic feature extraction and encoding: Extract the source IP and destination IP from the first data set as traffic features. Use the hash algorithm to generate unique identification codes for the source IP and destination IP, and add them to the first data set. Traverse each data record in the first data set. Extract the source IP address and destination IP address from each record as traffic features. Select an applicable hash algorithm, such as SHA-256, MD5, etc., to generate a unique identification code. Ensure that the selected hash algorithm has sufficient hashability to avoid conflicts. Apply the selected hash algorithm to the extracted source IP and destination IP respectively to generate a hash value. Use the generated hash value as the unique identification code. Add the generated unique identification code to each data record in the first data set as a new traffic feature. Ensure that the addition of the unique identification code does not destroy the structure and integrity of the original data. If necessary, update the data format to adapt to the new traffic features to ensure that subsequent processing can correctly parse and utilize the newly added features. If the update of the data set requires transactional processing, ensure that the consistency of the data can be maintained during the update process. Consider using database transactions or other mechanisms to ensure the atomicity of the update operation. Consider the handling of abnormal situations, such as hash algorithm errors and data format update failures. Implement the corresponding error handling mechanism, record logs or send notifications for manual intervention. Considering the performance characteristics of the hash algorithm, consider using the optimized parameters of the hash algorithm or implementing parallel processing to improve the efficiency of traffic feature extraction and encoding.

[0067] Figure 2This is an exemplary flow chart for obtaining traffic to be detected according to some embodiments of this specification, based on signature matching of black and white lists: obtain a locally stored IP address library, including a set of normal and malicious IP addresses. Encode the IP address library using a double hash mapping function to generate a Bloom filter IP library. Perform double hash mapping on the destination IP and query the Bloom filter IP library. Mark the traffic data as traffic to be detected or normal traffic according to the query result. Signature matching based on black and white lists involves obtaining a locally stored IP address library, encoding the IP address library using a double hash mapping function to generate a Bloom filter IP library, and performing double hash mapping on the destination IP and querying the Bloom filter IP library. Obtain an IP address library containing a set of normal and malicious IP addresses from local storage. Select a double hash mapping function to ensure that hash conflicts are minimized. Apply a double hash mapping function to each IP address, generate a hash value, and encode it as a bitmap of a Bloom filter. Generate a bitmap of a Bloom filter using the encoded hash value. Save the Bloom filter IP library in a memory or other high-speed storage medium for quick query. Apply the same double hash mapping function to the destination IP in each piece of traffic data to generate a hash value. Query the Bloom filter IP library to check whether the hash value of the destination IP exists in the bitmap. If it exists, mark the traffic data as normal traffic; if not, mark the traffic data as traffic to be detected. Based on the query results, mark the traffic data as normal traffic or traffic to be detected. A new column or field can be added to the data set to record the marking status of the traffic data. Further analyze or process the data marked as traffic to be detected, such as deep packet inspection, behavioral analysis, etc. Regularly update the locally stored IP address library to ensure that the Bloom filter IP library contains the latest normal and malicious IP addresses. Update the Bloom filter to maintain its accuracy and effectiveness. Optimize performance based on the characteristics of the Bloom filter, such as adjusting the bitmap size, the choice of hash function, etc. Consider concurrent queries and caching mechanisms to improve query efficiency.

[0068] Risk traffic processing: For IPs marked in the Bloom filter, query the local IP address library. If the IP is malicious, mark the corresponding traffic as risk traffic. Query the data records corresponding to the risk traffic in the second data set. According to the records, perform TCP interception or source IP rate limit processing. First, use the Bloom filter to mark the IP in the network traffic. Bloom filter is a data structure used to quickly check whether an element belongs to a set. Here, it is used to quickly check whether the IP is marked as malicious. For IPs marked as malicious by the Bloom filter, query the local IP address library. The local IP address library contains a list of known malicious IP addresses, a blacklist, or other marking information. If the query finds that the IP is malicious, mark the corresponding traffic as risk traffic. This involves adding a mark or setting a corresponding flag bit in the traffic record. For IPs marked as risk traffic, further queries are performed in the second data set. The second data set contains more detailed traffic information, attack patterns, and other contextual information. According to the records in the second data set, take corresponding security measures. The operations include: TCP interception: Block TCP connections associated with malicious IPs to prevent further attacks. Source IP rate limiting: Limit the traffic of malicious IPs to slow down potential attacks and reduce their impact.

[0069] Figure 3This is an exemplary flowchart of constructing a time series sample sequence according to some embodiments of this specification, based on the behavioral analysis of the LSTM neural network: extract the traffic to be detected and divide it into sub-flow batches. Extract the source IP, destination IP, port number and timestamp for each sub-flow batch. Encode the source IP and destination IP, and perform one-hot encoding of the port number to form an encoded digital vector of the traffic data packet. Construct an LSTM neural network including an input layer, a convolutional layer, an LSTM layer and an output layer. Capture the network traffic data to be detected from the network, which is a network data packet or other form of network traffic record. Divide the traffic to be detected into sub-flow batches, each sub-flow batch containing network traffic within a certain time range. Perform feature extraction on each sub-flow batch. Extract features such as source IP, destination IP, port number and timestamp from each sub-flow batch. Encode the extracted source IP and destination IP. A common encoding method is to use one-hot encoding, in which each IP address is represented as a unique binary vector. One-hot encode the extracted port number and represent each port number as a binary vector. Combine the encoded source IP, destination IP and port number to form the encoded digital vector of each data packet. Construct an LSTM neural network consisting of an input layer, a convolution layer, an LSTM layer and an output layer. The specific network structure includes: Input layer: accepts the encoded digital vector as input. Convolution layer (optional): used to further extract spatiotemporal features. LSTM layer: used to learn sequence information and behavior patterns. Output layer: produces the results of behavior analysis, such as identifying whether it is normal traffic or potential attack. Use the labeled training data to train the constructed LSTM neural network and adjust the network parameters to adapt to the actual traffic pattern. Use the test data to test and evaluate the trained model to check its performance on unseen data.

[0070] Figure 4This is an exemplary flowchart for constructing a training set according to some embodiments of this specification, training set construction and code conversion: extract a normal IP address set as a first sample sequence. Random downsampling generates a second sample sequence of time step sampling. Dynamic time folding algorithm generates a third sample sequence. Add random noise to generate a fourth sample sequence. Combine sample sequence data sets and perform code conversion. Extract an IP address set from known normal network traffic to form a first sample sequence. For the first sample sequence, random downsampling is performed to generate a second sample sequence of time step sampling. In this way, some normal traffic data at different time points can be obtained. Using the dynamic time folding algorithm, the normal traffic data is time-folded to generate a third sample sequence. This helps to consider long-term dependencies in the sequence. Add some random noise to the normal traffic data to generate a fourth sample sequence. This helps to make the model more robust to some abnormal situations. Combine the first, second, third and fourth sample sequences into a training set. Ensure that the labels of the sample sequences are clear, for example, the first and second samples are marked as normal, and the third and fourth samples are marked as abnormal. Code conversion is performed on each sample sequence. Specific steps may include: encoding the IP address, which can use one-hot encoding or other encoding methods. Consider converting the sequence data into an appropriate time series format in the case of time step sampling and dynamic time folding. Consider preprocessing operations such as data normalization for samples with random noise. Use the constructed training set for model training. Choose an appropriate neural network structure and perform supervised learning to enable the model to learn the patterns of normal and abnormal traffic.

[0071] LSTM neural network training: Use the generated training set as input to train the LSTM neural network model. Calculate the loss function value and update the network parameters using the gradient descent algorithm. Repeat until the loss function value is less than or equal to the threshold or the preset number of training times is reached. Use the previously constructed training set, which includes normal and abnormal sample sequences. Perform necessary preprocessing on the generated training set, such as standardization, normalization, or other preprocessing steps specific to the LSTM network. Build an appropriate LSTM neural network model. This may include defining the number of LSTM layers, the number of hidden units, and other related parameters. Select an appropriate loss function to measure the performance of the model. In the context of anomaly detection, loss functions suitable for anomaly detection tasks, such as mean absolute error (MAE) or mean square error (MSE), are often used. Select a gradient descent optimization algorithm, such as Adam, RMS prop, etc., to update the network parameters. Input the preprocessed training set into the LSTM network. Calculate the loss function value. Update the network parameters using the gradient descent algorithm. Repeat the above steps until the loss function value is less than or equal to the preset threshold or the preset number of training times is reached. Use the validation set to validate the trained model to ensure the performance of the model on unseen data. Adjustments may be made based on the validation results. The network structure, hyperparameters, or other training-related parameters may need to be adjusted. Changes in the loss function are monitored during training to ensure that the model gradually converges during the learning process. The condition for terminating training can be when the loss function value is less than or equal to a threshold, or when the preset maximum number of training times is reached.

[0072] Storage system selection: Hadoop is used as the distributed file system for the first dataset. MySQL is used as the database for the second dataset. Hadoop is an open source distributed storage and computing framework suitable for distributed processing of large datasets. Hadoop provides a distributed file system (HDFS) and a distributed computing framework (Map Reduce), which are suitable for processing large datasets. Hadoop is highly fault-tolerant and can handle node failures without losing data. Hadoop clusters can be easily expanded to adapt to growing data needs. The Hadoop ecosystem includes multiple related projects, such as Hive, Pig, etc., which provide more data processing tools. By storing the first dataset in HDFS, tools in the Hadoop ecosystem can be used to perform distributed processing and analysis on the data. Data may exist in the form of files in HDFS, which is convenient for batch processing and analysis using the Hadoop ecosystem. MySQL is a popular open source relational database management system (RDBMS) suitable for processing structured data. MySQL is suitable for storing structured data and is a suitable choice for the database in the second dataset. MySQL provides strong transaction support and is suitable for application scenarios that need to ensure data consistency and integrity. MySQL has good performance when processing small to medium-sized relational datasets. The relational data in the second dataset is stored in a MySQL database. The design of data tables should take into account query efficiency and database normalization.

[0073] Example 2

[0074] Obtain raw traffic data in the network, including logs, system information, and alarms. Deploy network data collection probes to collect Net Flow traffic data from network devices, including source IP, destination IP, port number, traffic size, and other information. Configure the server log collection module to collect text logs such as access logs, authentication logs, and error logs of network servers. Enable the host log collection function to obtain information such as operating system kernel logs, security logs, and application logs. Connect various security devices to obtain security-related logs such as firewall logs, intrusion detection system logs, and VPN logs. Collect security event alarm information from the security information and event management system. Use tools such as log stash to parse, filter, and uniformly format each log data. Finally, aggregate the raw network traffic data from different sources and store it in Hadoop's HDFS as a comprehensive raw traffic data set. Subsequent modules extract traffic data from HDFS for detection and analysis. By collecting multi-source heterogeneous traffic logs and alarm information, a comprehensive network traffic data set can be established to provide data support for security monitoring and detection.

[0075] The original traffic data is sharded using the hash modulo algorithm to generate a first data set, which is stored in the Hadoop distributed file system. The first data set is copied to the MySQL database to form a second data set. For the collected original traffic data, the hash value of each data entry is calculated using the MD5 hash algorithm. The number of shards is set, for example, 1024. The MD5 hash value is modulo the number of shards, and sharding is performed based on the modulo result. The sharded data is saved as multiple files and uploaded to the Hadoop HDFS file system. A directory is created on HDFS for each shard file. The Map Reduce program can process different shards in parallel based on the shard value in the file name. The Map Reduce program that reads the shard file on HDFS writes the shard data to the corresponding table in the MySQL database. A database and table corresponding to the HDFS directory structure are created on MySQL. The data of each shard is inserted into the corresponding database table. Finally, the first data set stored by shard on HDFS is completely copied to the MySQL database to form the second data set. The traffic data in the MySQL database can be directly queried later, and the batch analysis of large data can be based on the HDFS data set. HDFS and MySQL are used together to achieve efficient storage and use of traffic data sets.

[0076] Extract the source IP and destination IP from the first data set as traffic features. In the Map Reduce program, set the Mapper task to read the shard file of the first data set on HDFS. In the map method of Mapper, parse each line of raw traffic log data read in. From the parsed log, use regular expressions or text parsing methods to extract the source IP and destination IP fields. Encapsulate the extracted source IP and destination IP into<key,value> Yes, the key is set as a constant, and the value is set as the combination of the source IP and the destination IP. In the setup method of Mapper, establish a Redis connection, and in the map method, extract<key,value> The Reducer receives the data sent by the Mapper.<key,value> Yes. In the reduce method of Reducer, each key corresponds to multiple values. For each key, merge the multiple values ​​corresponding to it to obtain the combined set of source IP and destination IP. Write the extracted source IP and destination IP feature data into the MySQL feature library table. Repeat the Map Reduce process to finally obtain all the source IP and destination IP features stored in the first data set. In this way, feature extraction can be completed efficiently through Map Reduce big data parallel processing.

[0077] Use the hash algorithm to encode the source IP and destination IP to generate a unique identification code. For each source IP and destination IP extracted from the first data set. Use the MD5 hash algorithm to perform hash operations on the source IP and destination IP respectively. Convert the obtained 128-bit MD5 hash value into a hexadecimal string. The hash value of the source IP is used as the first part, and the hash value of the destination IP is used as the second part. Concatenate the two into a 32-bit string to form the final unique identification code. For example, the MD5 code of the source IP "10.0.1.5" is "e4d909c290d0fb1ca068ffaddf22cbd0", and the MD5 code of the destination IP "110.44.23.1" is "639bae9ac6b3e1a84cebb7b403297b79". The unique identification code obtained by splicing the two is "e4d909c290d0fb1ca068ffaddf22cbd0639bae9ac6b3e1a84cebb7b403297b79". Use the Map Reduce program to generate a unique identification code for each traffic data in the first data set. Add the generated code to the log corresponding to the first data set. Finally, each traffic log in the first data set contains the corresponding unique identification code. The unique code can be used for query and association analysis later.

[0078] The code is added to the first data set for double hash matching and LSTM network behavior analysis. During the MapReduce processing of traffic data, a unique identification code is generated for each traffic log. The generated code is directly written back to the HDFS data set file where the traffic log data is located. In this way, each traffic log in the first data set is added with the corresponding unique code. Read the first data set with the code added and perform the double hash matching process: extract the destination IP in the first data set; perform double hash operation on the destination IP to obtain a double hash index; query the index in the double hash Bloom filter to see if it hits the malicious IP; if it matches the malicious IP, mark the traffic as risky traffic. Read the first data set with the code added and perform the LSTM neural network analysis process: encode the traffic log and convert it into a digital sequence sample; input the sample sequence, use the LSTM network to obtain the time feature, and determine whether it is abnormal traffic; if the LSTM determines it to be abnormal, also mark the traffic as risky traffic. Finally, through double hash matching and LSTM time series analysis, the risky traffic in the first data set can be effectively discovered.

[0079] Match and query the local IP address library to determine malicious IP. LSTM model determines abnormal traffic. The local IP address library contains a set of IP addresses marked with malicious tags and is stored in the database. Extract the destination IP from the first data set. Use the destination IP as a keyword to match and query the IP address library. If the destination IP is marked as a malicious IP in the address library, the traffic is determined to be risky traffic. Construct an LSTM model for the first data set and input the sample sequence. Input the sample sequence and perform time series analysis on the traffic through the LSTM model. The LSTM model outputs the judgment result. If it is judged to be abnormal traffic, it is marked as risky traffic. Deploy the LSTM model as an online prediction service. The traffic analysis module calls the LSTM prediction service interface and passes in the traffic sample to obtain the judgment. The final program integrates double hash matching and LSTM judgment. If any one is judged as risky traffic, it will be blocked. Combine static matching rules and dynamic deep learning technology to achieve high-accuracy network risk traffic detection.

[0080] If risky traffic is detected, the corresponding data record is searched in the second data set. In traffic detection, double hash matching and LSTM network have been used to determine that a certain traffic is risky traffic. Extract its unique identification code from the risky traffic log. In the MySQL database, the second data set is stored in the form of a table. According to the unique identification code, construct an SQL query statement and query the table in the second data set. For example: "SELECT*FROM flow_data_table WHERE uid='abcd1234'". Uid is the unique identification code field of the traffic log data table. This SQL query will retrieve the original data record that completely corresponds to the risky traffic. Get the query result, that is, the data entry of the risky traffic in the second data set. Contains all the original information of the traffic, including source IP, destination IP, port, etc. Subsequent bans and isolation can be directly performed based on the original data.

[0081] Limit the speed of the source IP of risky traffic, and perform TCP interception on the destination IP. The original data records of risky traffic have been queried from the second data set. The source IP and destination IP of risky traffic can be extracted from the records. Configure the speed limit policy and ACL rules on the network management and control device. For the source IP of risky traffic, add a rule to the speed limit policy. Specify the maximum inbound and outbound bandwidth values ​​of the source IP and enable speed limit. For the destination IP of risky traffic, add a deny rule to the ACL rules. Specify to intercept the TCP traffic of the destination IP. Send the speed limit policy and ACL rules to the core router of the network. After receiving the rules, the router implements bandwidth speed limit on the Specified source IP. At the same time, intercept the TCP traffic of the destination IP to prevent communication with it. Finally, use network equipment to implement traffic control and blocking on the source and destination IP of the detected risky traffic.

[0082] The model is trained through sample enhancement, LSTM network construction, loss calculation, gradient optimization and other processes. Normal traffic data is collected, and sample enhancement is performed by random downsampling, time folding and other methods to build a variety of training samples. A network structure containing an LSTM layer is constructed, and modules such as batch normalization layer, one-dimensional convolution layer, residual connection, and attention mechanism are added. The enhanced traffic sample is converted into a digital sequence and input into the LSTM network. Forward propagation is performed to calculate the loss function of the network output and the ground truth, such as cross entropy loss. The parameter gradient of each node is calculated by the chain rule. The gradient descent optimizer is applied to update the network parameters based on the calculated gradient. The sample input, forward propagation, back propagation, parameter update and other cyclic processes are repeated. After multiple iterations, the parameters of the network are gradually optimized, the loss function value is reduced, and the model effect is improved. When the early termination condition is met, the training is terminated to obtain the final optimized LSTM model. The trained LSTM model is deployed to the production environment for traffic anomaly detection. Through data enhancement and model iterative optimization, an LSTM deep learning model with better prediction effect on network traffic can be obtained.

[0083] Finally, the pre-trained double hash matching model and LSTM neural network model deployed in the network are obtained. The two models are connected in series to perform traffic detection, discover risk data, and perform source IP speed limit and TCP interception. Based on the data support provided by Hadoop and MySQL, a closed-loop network security protection solution of detection-query-disposal is formed. The training of the double hash matching model and the LSTM model is completed to obtain the optimized final model. The double hash matching model is encapsulated as a prediction service and published to the model prediction platform. The LSTM model is deployed through Tensor Flow Serving and the RESTAPI service is published. The traffic detection program is deployed in the network and the two model services are integrated and called. The traffic detection program receives network data and first calls the double hash matching service. If the match is judged as risky traffic, the source IP speed limit or destination IP blocking is directly performed. If the match does not hit, the LSTM model service is called and the traffic features are passed in. The LSTM service returns the judgment result. If it is judged as abnormal, the corresponding disposal is performed. In this way, the two model services are connected in series to perform multi-level detection of traffic. Finally, an online deployed traffic detection architecture is formed, which can monitor the real-time network security. Make full use of the advantages of the two models to improve the accuracy of detection. The original traffic data is stored on the Hadoop platform for offline batch analysis and model training. Traffic features and models are deployed in the MySQL database to provide fast query. The traffic detection program obtains real-time traffic from the network. The double hash matching model and LSTM model are called to detect the traffic. The model comes from Hadoop's offline training, and the parameters are stored in MySQL. If risky traffic is detected, a unique identification code query is performed. The corresponding detailed traffic records are queried in MySQL. According to the records, traffic control is performed on the network device. The entire closed-loop process from detection to disposal is completed. Based on Hadoop's distributed computing capabilities, massive data is stored and processed. MySQL provides fast query and service response. The combination of the two forms an end-to-end network security protection solution from data to model, from query to control.

[0084] The invention of the present application and its implementation methods are described schematically above. The description is not restrictive. Without departing from the spirit or basic features of the present application, the present application can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention of the present application. The actual structure is not limited to this. Any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the present invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all fall within the scope of protection of this patent. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

Claims

1. A big data computer network security protection method, comprising: Obtain network traffic data, which includes network logs, system logs and alarm information; The obtained network traffic data is fragmented by using a hash modulus algorithm, and the fragmented network traffic data is used as a first data set; Copy the first data set to obtain a second data set; Using a distributed file system to store the first data set, and using a database to store the second data set; Extract source IP and destination IP from the first data set as traffic features; According to the extracted source IP and destination IP, a corresponding unique identification code is generated using a hash algorithm, and the unique identification code is added to the first data set; For the first data set after adding the unique identification code, perform signature matching based on the black and white lists and behavior analysis based on the LSTM neural network; According to the detection results, the data records corresponding to the risk traffic data are queried in the second data set, and TCP interception or source IP speed limit processing is performed; Signature matching based on blacklist and whitelist includes the following steps: Obtain a locally stored IP address library, where the IP address library contains a pre-marked normal IP address set and a malicious IP address set; Encode the IP address library using a double hash mapping function to generate the corresponding Bloom filter IP library; Extract the destination IP from the first data set; For the extracted target IP, a double hash mapping function is used to obtain the corresponding first index value and second index value from the Bloom filter IP library; In the Bloom filter IP library, query whether the index positions corresponding to the first index value and the second index value are marked at the same time, where the marking indicates that the corresponding position of the index obtained by the IP according to the double hash mapping function in the Bloom filter IP library bit array is set to 1; If the index positions corresponding to the first index value and the second index value are both marked as 1, the marking attribute of the corresponding IP is searched in the IP address library; if any index corresponding position is not marked, the corresponding network traffic data is marked as traffic to be detected; Based on the detection results, TCP interception or source IP rate limiting is performed, including the following steps: When the index positions corresponding to the first index value and the second index value are both marked in the Bloom filter IP library, searching for a marking attribute of the corresponding IP in the local IP address library; If the IP tag attribute found is a malicious IP, the network traffic corresponding to the IP is marked as risky traffic; In the database of the second data set, query the corresponding data record according to the unique identification code of the risk flow; Based on the detection results, TCP interception or source IP rate limiting is performed, including the following steps: Extract the source IP and destination IP from the data records of the queried risk traffic; For the extracted source IP, limit the source IP speed or isolate the traffic; Perform TCP interception on the extracted destination IP; Behavior analysis based on LSTM neural network includes the following steps: Get the traffic to be tested; A sliding time window is used to divide the traffic to be detected into sub-flow batches containing preset traffic data packets; For each sub-flow batch of traffic packets, extract the source IP, destination IP, port number and timestamp; The extracted source IP and destination IP are encoded using a hash function to obtain a hash code; The extracted port number is encoded using one-hot encoding to obtain a category vector; The obtained hash code, category vector and timestamp are concatenated to obtain the encoded digital vector of the traffic data packet; The encoded digital vectors of the traffic data packets within the time window are arranged in timestamp order to form a time series sample sequence, which serves as the input data of the LSTM neural network.

2. The big data computer network security protection method according to claim 1 is characterized by: Behavior analysis based on LSTM neural network includes the following steps: Construct an LSTM neural network consisting of an input layer, a convolutional layer, an LSTM layer, and an output layer; A batch normalization layer is set after the input layer to linearly scale the input data obtained by the input layer to eliminate the mean shift and variance shift of the input data; A one-dimensional convolution layer is set after the batch normalization layer to perform one-dimensional convolution on the data output by the batch normalization layer to extract the local features of the time series; Residual connections are set between adjacent LSTM layers to add the input features and output features of the previous LSTM layer as the input of the next LSTM layer; After the last LSTM layer, a self-attention layer is set before the output layer to learn the features of input data at different time steps; A dropout layer is set after the self-attention layer and before the output layer to randomly suppress the input layer neurons according to random probability.

3. The big data computer network security protection method according to claim 2 is characterized by: Constructing a training set includes the following steps: Extracting a set of IP addresses marked as normal from a locally stored IP address library as a first sample sequence; For the traffic sequence in the first sample sequence, a random downsampling algorithm is used to generate a second sample sequence of time step sampling; For the flow sequence in the second sample sequence, a dynamic time folding algorithm is used to adjust the time step position to generate a third sample sequence; Adding random noise to the traffic sequence in the second sample sequence to simulate jitter of normal traffic, to generate a fourth sample sequence; Combining the traffic sequences in the first sample sequence, the second sample sequence, the third sample sequence, and the fourth sample sequence to form a sample sequence data set; The obtained sample sequence data set is encoded and converted to obtain a digital vector sequence as a training set for the LSTM neural network.

4. The big data computer network security protection method according to claim 3 is characterized by: Generating a third sample sequence includes the following steps: Set the maximum time distance threshold between traffic packets; Calculating the time step distance between each traffic data packet and other traffic data packets in the second sample sequence; Merge multiple traffic data packets whose calculated time step distances are within the set maximum time distance threshold into one traffic data packet; Inserting the merged traffic data packet into the time step position corresponding to the second sample sequence; Repeat the above steps until the time step distances of all traffic data packets in the sample sequence are greater than the set maximum time distance threshold, and the obtained sample sequence is taken as the third sample sequence.

5. The big data computer network security protection method according to claim 3 is characterized by: Using the obtained training set, the constructed LSTM neural network is trained, including the following steps: The generated training set is used as input to the constructed LSTM neural network for model training; Calculate the predicted output value of each sample data and the loss function value of the corresponding label; By chaining the algorithm, the partial derivatives of the loss function value of each sample data with respect to the weight parameters and the partial derivatives of the bias parameters of all nodes in the LSTM neural network are calculated respectively; According to the calculated partial derivatives of the weight parameters and the partial derivatives of the bias parameters, the node parameters in the LSTM neural network are updated by the gradient descent algorithm; Repeat the above steps until the loss function value on the training set is less than or equal to the threshold or the preset number of training times is reached.

6. The big data computer network security protection method according to any one of claims 1 to 5, characterized in that: The distributed file system for storing the first data set adopts Hadoop; The database storing the second data set adopts a MySQL relational database.