A high-speed network traffic anomaly detection method and device

Abstract

The application discloses a high-speed network flow anomaly detection method and device, belongs to the technical field of network data flow real-time detection, and determines the network characteristic base number estimation value of the storage position of the data flow through the data flow identifier of the network data packet in each data flow, determines the abnormal data flow in the fine-grained layer based on the network characteristic base number estimation value, and determines the abnormal subnet in the coarse-grained layer through the subnet identifier of the network data packet in the abnormal data flow, realizes the capture of individual abnormal behavior and group abnormal mode, can realize high-precision large-base flow detection and subnet-level aggregation analysis network flow anomaly detection under the condition of limited memory, and solves the problem that current high-speed network flow anomaly detection cannot consider light-weighted calculation and detection precision.

Description

Technical Field

[0001] This invention relates to a method and apparatus for detecting abnormal high-speed network traffic, belonging to the field of real-time network data stream detection technology. Background Technology

[0002] In the high-speed network environments of modern data centers, network link speeds can reach hundreds of Gbps or even Tbps. To ensure that monitoring performance matches the network line speed, traffic monitoring mechanisms typically need to be deployed on the network processing chips of switches or routers, and real-time data processing is performed based on on-chip caches. However, on-chip cache resources are very limited, with a capacity of only a few megabytes, and they need to share memory with other network functional modules. This limitation has become the main bottleneck for achieving high-precision traffic monitoring. Traditional monitoring methods that rely on precise counting require maintaining complete state information for each data stream, resulting in huge memory overhead, making it difficult to achieve real-time processing in high-speed networks. While sampling-based schemes can reduce resource consumption, they lose key traffic feature information, leading to a decrease in detection accuracy. At the same time, network traffic exhibits a long-tail distribution, with a few super-spreader nodes accounting for the majority of traffic, while the vast majority of ordinary hosts generate only a small number of connections, making it difficult for uniform sampling methods to effectively capture key anomalies. Furthermore, modern network attacks tend to be distributed and low-volume. Attackers often use a large number of controlled hosts to collaboratively send small-scale data packets to circumvent threshold detection mechanisms. Individual traffic may appear normal, but when aggregated, they can cause serious impacts.

[0003] Existing technologies typically introduce probabilistic data structures to reduce cache pressure, but they suffer from insufficient accuracy when handling hash collisions and fine-grained analysis. When multiple data streams are hashed to the same storage location, the estimation error increases significantly, making it difficult to meet the requirements for accurate identification of super-spreaders and distributed attacks. Furthermore, existing solutions mostly focus on single-stream level detection, lacking effective monitoring of aggregated traffic at the subnet level. This makes it difficult to simultaneously achieve multi-granularity monitoring targets at the host and subnet levels under limited memory conditions, thus limiting the system's comprehensive detection capabilities for complex distributed anomalies. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a high-speed network traffic anomaly detection method and device. This method and device can achieve high-precision large-radius flow detection and subnet-level aggregation analysis of network traffic anomaly detection under limited memory conditions, and solves the problem that current high-speed network traffic anomaly detection cannot balance lightweight computation and detection accuracy.

[0005] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:

[0006] This invention provides a method for detecting abnormal high-speed network traffic, comprising:

[0007] Obtain the data stream identifier to which the network packets in each data stream belong;

[0008] Based on the data stream identifier, determine the cardinality estimate of network features used to characterize the size of the data stream at the storage location;

[0009] Based on the estimated cardinality of network features, abnormal data streams are filtered out.

[0010] Obtain the subnet identifier to which the network packets in the abnormal data stream belong;

[0011] Based on the subnet identifier, identify the abnormal subnet corresponding to the abnormal data stream;

[0012] All abnormal data streams are correlated with abnormal subnets to identify the key source streams for each abnormal subnet.

[0013] Furthermore, the step of filtering out abnormal data streams based on network feature cardinality estimates includes comparing each network feature cardinality estimate with a preset abnormal traffic threshold. If the network feature cardinality estimate is greater than the preset abnormal traffic threshold, then the data stream corresponding to that network feature cardinality estimate is an abnormal data stream.

[0014] Furthermore, determining the abnormal subnet corresponding to the abnormal data stream based on the subnet identifier includes:

[0015] Based on the subnet identifier, determine the aggregate cardinality estimate of the network data packet corresponding to the subnet identifier in the storage location to characterize the importance of network features;

[0016] Each aggregate cardinality estimate is compared with a preset abnormal subnet threshold. If the aggregate cardinality estimate is greater than the preset abnormal subnet threshold, the subnet corresponding to the abnormal data stream is marked as an abnormal subnet.

[0017] Furthermore, the step of performing correlation analysis on all abnormal data streams and abnormal subnets to determine the key source streams of each abnormal subnet includes:

[0018] Extract the corresponding subnet identifier based on the source and destination addresses of each abnormal data stream to form a mapping from subnet to the abnormal data stream set;

[0019] The mapping from subnet to abnormal data stream set is compared with all abnormal subnets one by one to determine the abnormal data streams belonging to the abnormal subnets and the abnormal data streams belonging to the normal subnets.

[0020] Calculate the cardinality contribution of each abnormal data stream to the anomalies in its subnet;

[0021] Output the key source flows for each anomalous subnet, ranked from largest to smallest based on their contribution to the base population.

[0022] Furthermore, the storage location includes a first storage unit for storing and updating data flow state information and a second storage unit for storing and updating corresponding subnet aggregation feature information based on the data flow state information.

[0023] Furthermore, the method for storing and updating data stream state information in the first storage unit includes:

[0024] The data stream status information includes the data stream identifier, traffic growth frequency count value, and network characteristic cardinality estimate value;

[0025] Based on the source address, destination address, and protocol type characteristics of each arriving network data packet in the data stream, generate the current data stream identifier corresponding to the network data packet;

[0026] Based on the current data stream identifier, calculate the first index position corresponding to each network data packet in the storage location;

[0027] The first index position is calculated using the following formula:

[0028] ;

[0029] in, Indicates the first Each first storage unit corresponds to the first index position of the network data packet. Indicates the first hash function, Indicates the current data stream identifier. This represents the XOR operator. Indicates a random seed. This represents the modulo operator. Indicates the number of the first storage units;

[0030] If the first index position in the first storage unit is occupied, then the first storage unit contains a historical data stream identifier and the corresponding historical network feature cardinality estimate.

[0031] The historical data stream identifier is compared with the current data stream identifier. If the comparison results are consistent, the traffic growth frequency count is incremented and the corresponding network feature cardinality estimate is updated. Otherwise, the conflict handling mechanism is triggered.

[0032] Furthermore, the conflict handling mechanism includes:

[0033] Generate a random value in the range (0,1);

[0034] The replacement probability is calculated using the following formula:

[0035] ;in, Indicates the probability of replacement. This represents the count value indicating the frequency of traffic growth.

[0036] Compare the replacement probability with random values;

[0037] If the replacement probability is greater than the random value, then obtain the element identifier of the data packet corresponding to the current data stream identifier, and send the element identifier of the data packet corresponding to the current data stream identifier and the current data stream identifier to the second storage unit;

[0038] If the replacement probability is not greater than the random value, the historical data stream identifier and the corresponding historical network feature cardinality estimate are sent to the second storage unit.

[0039] Reset the first storage unit that triggered the conflict handling mechanism.

[0040] Furthermore, the second storage unit includes a first counting unit and a second counting unit;

[0041] The subnet aggregation feature information includes the subnet identifier, feature strength value, and estimated aggregation cardinality value;

[0042] The first counting unit stores the subnet identifier. First characteristic intensity value and estimated polymer basis ; Indicates the storage location number The subnet identifier stored in the first counting unit of the second storage unit. Indicates the storage location number The feature intensity value stored in the first counting unit of the second storage unit Indicates the storage location number The aggregate cardinality estimate stored in the first counting unit within each second storage unit;

[0043] The second counting unit stores the subnet identifier. Second characteristic intensity value and estimated polymer basis ; Indicates the storage location number The subnet identifier stored in the second counting unit of the second storage unit. Indicates the storage location number The feature intensity value stored in the second counting unit of each second storage unit Indicates the storage location number The estimated aggregation cardinality stored in the second counting unit of each second storage unit. .

[0044] Furthermore, the method for the second storage unit to store and update the corresponding subnet aggregation feature information based on the data flow state information includes:

[0045] If the data stream status information update triggers the conflict handling mechanism, then obtain the subnet identifier of the current data packet in the data stream;

[0046] Based on the subnet identifier, calculate the second index position of the current data packet in the storage location;

[0047] The second index position is calculated using the following formula:

[0048] ;

[0049] in, Indicates the first The second storage unit corresponds to the second index position of the current data packet. This represents the second hash function. Indicates the subnet identifier. This represents the XOR operator. Indicates a random seed. This represents the modulo operator. Indicates the number of second storage units in the storage location;

[0050] Based on the second index position, obtain the first feature strength value and the second feature strength value of the current data packet in the second storage unit;

[0051] Get the element identifier of the current data packet;

[0052] Calculate the real-time feature intensity value corresponding to the current data packet based on the element identifier of the current data packet;

[0053] The real-time feature intensity value is calculated using the following formula:

[0054] ;

[0055] in, This represents the real-time feature strength value corresponding to the current data packet. This represents the function for calculating the number of leading zeros. This represents the third hash function. The element identifier that represents the current data packet;

[0056] The real-time feature strength value is compared with the second feature strength value. If the real-time feature strength value is greater than the second feature strength value, the second feature strength value is replaced with the real-time feature strength value to obtain the updated second feature strength value; otherwise, the current data packet is discarded.

[0057] If the first feature strength value is less than the updated second feature strength value, then the two values ​​are swapped.

[0058] In another aspect, the present invention provides a high-speed network traffic anomaly detection device for implementing the above-described high-speed network traffic anomaly detection method, the device comprising:

[0059] The first acquisition module is used to acquire the data stream identifier to which the network data packets in each data stream belong;

[0060] The first calculation module is used to determine the estimated network feature cardinality of the data stream in the storage location, which is used to characterize the size of the data stream, based on the data stream identifier.

[0061] The processing module is used to filter out abnormal data streams based on the estimated cardinality of network features;

[0062] The second acquisition module is used to acquire the subnet identifier to which the network data packets in the abnormal data stream belong;

[0063] The second calculation module is used to determine the abnormal subnet corresponding to the abnormal data stream based on the subnet identifier;

[0064] The analysis module is used to perform correlation analysis on all abnormal data streams and abnormal subnets to determine the key source streams of each abnormal subnet.

[0065] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0066] 1. This invention determines the estimated network feature cardinality of a data stream at its storage location by identifying the data stream identifier to which the network data packets in each data stream belong. Based on the estimated network feature cardinality, it identifies anomalous data streams at a fine-grained level. Then, by identifying the subnet identifier to which the network data packets in the anomalous data stream belong, it identifies anomalous subnets at a coarse-grained level. This enables the capture of individual anomalous behaviors and group anomalous patterns. It can achieve high-precision large cardinality stream detection and subnet-level aggregation analysis of network traffic anomaly detection under limited memory conditions, solving the problem that current high-speed network traffic anomaly detection cannot balance lightweight computation and detection accuracy.

[0067] 2. This invention performs differentiated updates based on the usage status of the first storage unit. When the storage is idle, it directly initializes the storage to ensure timely updates of the data stream. When the storage is occupied and the data stream identifier comparison results are consistent, it updates the count and network feature cardinality estimation. When there is an identifier conflict, it triggers a conflict handling mechanism and ensures that the large flow exists in the first storage unit through probabilistic replacement, thus ensuring the effective utilization of the storage structure.

[0068] 3. When the conflict handling mechanism is triggered, the present invention determines whether to record in the second storage unit by comparing the real-time feature strength value corresponding to the current data packet in the data stream with the second feature strength value, thus avoiding invalid data replacement. Then, based on the relative size of the updated second feature strength value and the first feature strength value, selective updates are performed to ensure that important network features are preferentially retained, thereby improving the utilization efficiency of the storage structure. Attached Figure Description

[0069] Figure 1 This is a flowchart of a high-speed network traffic anomaly detection method provided in an embodiment of the present invention;

[0070] Figure 2 This is a flowchart of storing and updating data stream status information provided in an embodiment of the present invention;

[0071] Figure 3 This is a flowchart of storing and updating subnet aggregation feature information provided in an embodiment of the present invention;

[0072] Figure 4 This is a schematic diagram of the structure of a high-speed network traffic anomaly detection device provided in an embodiment of the present invention. Detailed Implementation

[0073] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0074] Example 1

[0075] like Figure 1 As shown, this embodiment 1 provides a method for detecting abnormal high-speed network traffic, including:

[0076] Obtain the data stream identifier to which the network packets in each data stream belong;

[0077] Based on the data stream identifier, determine the cardinality estimate of network features used to characterize the size of the data stream at the storage location. ;

[0078] Estimated cardinality of each network feature Compared with the preset abnormal traffic threshold To make a comparison, if If the estimated cardinality of the network features is then considered an anomalous data stream;

[0079] Obtain the subnet identifier to which the network packets in the abnormal data stream belong;

[0080] Based on the subnet identifier, the abnormal subnet corresponding to the abnormal data stream is determined, specifically:

[0081] Based on the subnet identifier, determine the aggregation cardinality estimate of the network data packet corresponding to the subnet identifier at the storage location, which is used to characterize the importance of network features. ;

[0082] Each aggregation basis estimate With preset abnormal subnet threshold Compare and mark all that meet the criteria. The subnet corresponding to the estimated aggregation cardinality is an anomalous subnet;

[0083] Perform correlation analysis on all abnormal data streams and abnormal subnets to identify the key source streams for each abnormal subnet. Specifically:

[0084] Extract the corresponding subnet identifier based on the source and destination addresses of each abnormal data stream to form a mapping from subnet to the abnormal data stream set;

[0085] The mapping from subnet to abnormal data stream set is compared with all abnormal subnets one by one to determine the abnormal data streams belonging to the abnormal subnets and the abnormal data streams belonging to the normal subnets.

[0086] Calculate the cardinality contribution of each anomalous data stream to the anomalous subnet to which it belongs; the cardinality contribution can be measured by the proportion of the estimated network feature cardinality of the stream to the estimated aggregate cardinality of the subnet, or by combining the traffic growth frequency count and feature intensity value for weighted sorting.

[0087] Based on the contribution of the base number from large to small, the key source flow of each abnormal subnet is output, which can distinguish whether a few large flows push up the subnet or a large number of dispersed flows raise it together. Abnormal large flows but normal subnets are marked separately as local hotspots or targeted anomalies for priority investigation.

[0088] In this embodiment, the storage location adopts a two-layer collaborative structure, including M first storage units storing data flow status information and N second storage units storing subnet aggregation feature information corresponding to the data flow information; wherein, the data flow status information includes data flow identifier, traffic growth frequency count value and network feature cardinality estimate value; the subnet aggregation feature information includes subnet identifier, feature strength value and aggregation cardinality estimate value.

[0089] Each first storage unit includes three fields: data stream identifier, traffic statistics counter, and network feature cardinality estimator. These correspond to the data stream identifier, traffic growth frequency count value, and network feature cardinality estimate value, which store data stream status information, respectively. It should be noted that the data stream identifier is used to uniquely identify a data stream, the traffic statistics counter records the activity level of the corresponding data stream, and the network feature cardinality estimator uses a probabilistic statistical algorithm to estimate the cardinality of the corresponding data stream.

[0090] When the target data stream is received, for each arriving network packet:

[0091] like Figure 2 As shown, based on the source address, destination address, and protocol type characteristics of the network data packet, a current data stream identifier corresponding to the network data packet is generated;

[0092] Based on the current data stream identifier, calculate the first index position corresponding to the first storage unit and each network data packet; the first index position is calculated using the following formula:

[0093] ;

[0094] in, Indicates the first Each first storage unit corresponds to the first index position of the network data packet. Indicates the first hash function, Indicates the current data stream identifier. This represents the XOR operator. Indicates a random seed. This represents the modulo operator. Indicates the number of the first storage units;

[0095] If the first index position in the first storage unit is free, the current data flow identifier is stored in the first index position in the free first storage unit, and the traffic growth frequency count value and network feature cardinality estimate value in the corresponding first storage unit are initialized.

[0096] If the first index position in the first storage unit is occupied, then the first storage unit contains a historical data stream identifier and the corresponding historical network feature cardinality estimate.

[0097] The historical data stream identifier is compared with the current data stream identifier. If the comparison results are consistent, the traffic growth frequency count is incremented and the corresponding network feature cardinality estimate is updated. Otherwise, the conflict handling mechanism is triggered.

[0098] The conflict resolution mechanism is triggered as follows:

[0099] Generate a random value in the range (0,1), which is obtained by mapping the output of the hash function;

[0100] The replacement probability is calculated using the following formula:

[0101] ;in, Indicates the probability of replacement. This represents the count value indicating the frequency of traffic growth.

[0102] Compare the replacement probability with random values;

[0103] If the replacement probability is greater than the random value, then obtain the element identifier of the data packet corresponding to the current data stream identifier, and send the element identifier of the data packet corresponding to the current data stream identifier and the current data stream identifier to the subnet aggregation analysis layer;

[0104] If the replacement probability is greater than the random value but not greater than the random value, then the historical data stream identifier and the corresponding historical network feature cardinality estimate are sent to the subnet aggregation analysis layer.

[0105] Reset the first storage unit that triggered the conflict handling mechanism.

[0106] It should be noted that each second storage unit includes two independent computing units, namely a first counting unit and a second counting unit; the first computing unit includes three fields, namely a subnet identifier, a first feature strength value, and a aggregation cardinality estimator; the second counting unit also includes three fields, namely a subnet identifier, a second feature strength value, and an aggregation cardinality estimator.

[0107] To distinguish the storage contents in the two computing units, the following representation is used:

[0108] The first counting unit stores the subnet identifier. First characteristic intensity value and estimated polymer basis ; Indicates the storage location number The subnet identifier stored in the first counting unit of the second storage unit. Indicates the storage location number The feature intensity value stored in the first counting unit of the second storage unit Indicates the storage location number The aggregate cardinality estimate stored in the first counting unit within each second storage unit;

[0109] The second counting unit stores the subnet identifier. Second characteristic intensity value and estimated polymer basis ; Indicates the storage location number The subnet identifier stored in the second counting unit of the second storage unit. Indicates the storage location number The feature intensity value stored in the second counting unit of each second storage unit Indicates the storage location number The estimated aggregation cardinality is stored in the second counting unit of each second storage unit;

[0110] in, The two counter units are arranged in order according to the magnitude of the feature intensity value to ensure that the first feature intensity value is always greater than the second feature intensity value.

[0111] like Figure 3 As shown, if the data flow status information update triggers the conflict handling mechanism, the subnet identifier of the current data packet in the data flow is obtained;

[0112] Based on the subnet identifier, determine the second index position of the current data packet in the storage location, including:

[0113] The second index position is calculated using the following formula:

[0114] ;

[0115] in, Indicates the first The second storage unit corresponds to the second index position of the current data packet. This represents the second hash function. Indicates the subnet identifier. This represents the XOR operator. Indicates a random seed. This represents the modulo operator. Indicates the number of second storage units in the storage location;

[0116] Based on the second index position, determine the second index position of the current data packet in the second storage unit;

[0117] Based on the second index position of the current data packet, obtain the first feature strength value and the second feature strength value of the second storage unit where the current data packet is located;

[0118] Get the element identifier of the current data packet;

[0119] Calculate the real-time feature intensity value corresponding to the current data packet based on the element identifier of the current data packet; the real-time feature intensity value is calculated using the following formula:

[0120] ;

[0121] in, This represents the real-time feature strength value corresponding to the current data packet. This represents the function for calculating the number of leading zeros. This represents the third hash function. The element identifier that represents the current data packet;

[0122] The real-time feature strength value is compared with the second feature strength value. If the real-time feature strength value is greater than the second feature strength value, the second feature strength value is replaced with the real-time feature strength value to obtain the updated second feature strength value; otherwise, the current data packet is discarded.

[0123] If the first feature strength value is less than the updated second feature strength value, then the two values ​​are swapped.

[0124] Example 2

[0125] like Figure 4 As shown, a high-speed network traffic anomaly detection device is used to implement the above-mentioned high-speed network traffic anomaly detection method. The device includes:

[0126] The first acquisition module is used to acquire the data stream identifier to which the network data packets in each data stream belong;

[0127] The first calculation module is used to determine the estimated network feature cardinality of the data stream in the storage location, which is used to characterize the size of the data stream, based on the data stream identifier.

[0128] The processing module is used to filter out abnormal data streams based on the estimated cardinality of network features;

[0129] The second acquisition module is used to acquire the subnet identifier to which the network data packets in the abnormal data stream belong;

[0130] The second calculation module is used to determine the abnormal subnet corresponding to the abnormal data stream based on the subnet identifier;

[0131] The analysis module is used to perform correlation analysis on all abnormal data streams and abnormal subnets to determine the key source streams of each abnormal subnet.

[0132] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0133] This application is described with reference to flowchart illustrations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 One or more processes or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.

[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0136] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for detecting abnormal high-speed network traffic, characterized in that, include: Obtain the data stream identifier to which the network packets in each data stream belong; Based on the data stream identifier, determine the cardinality estimate of network features used to characterize the size of the data stream at the storage location; The storage location includes a first storage unit for storing and updating data stream status information and a second storage unit for storing and updating corresponding subnet aggregation feature information based on the data stream status information; The method for storing and updating data stream state information in the first storage unit includes: The data stream status information includes the data stream identifier, traffic growth frequency count value, and network characteristic cardinality estimate value; Based on the source address, destination address, and protocol type characteristics of each arriving network data packet in the data stream, generate the current data stream identifier corresponding to the network data packet; Based on the current data stream identifier, calculate the first index position corresponding to each network data packet in the storage location; The first index position is calculated using the following formula: ； in, Indicates the first Each first storage unit corresponds to the first index position of the network data packet. Indicates the first hash function, Indicates the current data stream identifier. This represents the XOR operator. Indicates a random seed. This represents the modulo operator. Indicates the number of the first storage units; If the first index position in the first storage unit is occupied, then the first storage unit contains a historical data stream identifier and the corresponding historical network feature cardinality estimate. The historical data stream identifier is compared with the current data stream identifier. If the comparison results are consistent, the traffic growth frequency count is incremented and the corresponding network feature cardinality estimate is updated. Otherwise, the conflict handling mechanism is triggered. Based on the estimated cardinality of network features, abnormal data streams are filtered out. Obtain the subnet identifier to which the network packets in the abnormal data stream belong; Based on the subnet identifier, identify the abnormal subnet corresponding to the abnormal data stream; All abnormal data streams are correlated with abnormal subnets to identify the key source streams for each abnormal subnet.

2. The high-speed network traffic anomaly detection method according to claim 1, characterized in that, The process of filtering out abnormal data streams based on network feature cardinality estimates includes: Each network feature cardinality estimate is compared with a preset abnormal traffic threshold. If the network feature cardinality estimate is greater than the preset abnormal traffic threshold, the data stream corresponding to that network feature cardinality estimate is an abnormal data stream.

3. The high-speed network traffic anomaly detection method according to claim 1, characterized in that, The step of determining the abnormal subnet corresponding to the abnormal data stream based on the subnet identifier includes: Based on the subnet identifier, determine the aggregate cardinality estimate of the network data packet corresponding to the subnet identifier in the storage location to characterize the importance of network features; Each aggregate cardinality estimate is compared with a preset abnormal subnet threshold. If the aggregate cardinality estimate is greater than the preset abnormal subnet threshold, the subnet corresponding to the abnormal data stream is marked as an abnormal subnet.

4. The high-speed network traffic anomaly detection method according to claim 1, characterized in that, The process involves correlating all abnormal data streams with abnormal subnets to determine the key source streams for each abnormal subnet, including: Extract the corresponding subnet identifier based on the source and destination addresses of each abnormal data stream to form a mapping from subnet to the abnormal data stream set; The mapping from subnet to abnormal data stream set is compared with all abnormal subnets one by one to determine the abnormal data streams belonging to the abnormal subnets and the abnormal data streams belonging to the normal subnets. Calculate the cardinality contribution of each abnormal data stream to the anomalies in its subnet; Output the key source flows for each anomalous subnet, ranked from largest to smallest based on their contribution to the base population.

5. The high-speed network traffic anomaly detection method according to claim 1, characterized in that, The conflict handling mechanism includes: Generate a random value in the range (0,1); The replacement probability is calculated using the following formula: ;in, Indicates the probability of replacement. This represents the count value indicating the frequency of traffic growth. Compare the replacement probability with random values; If the replacement probability is greater than the random value, then obtain the element identifier of the data packet corresponding to the current data stream identifier, and send the element identifier of the data packet corresponding to the current data stream identifier and the current data stream identifier to the second storage unit; If the replacement probability is not greater than the random value, the historical data stream identifier and the corresponding historical network feature cardinality estimate are sent to the second storage unit. Reset the first storage unit that triggered the conflict handling mechanism.

6. The high-speed network traffic anomaly detection method according to claim 1, characterized in that, The second storage unit includes a first counting unit and a second counting unit; The subnet aggregation feature information includes the subnet identifier, feature strength value, and estimated aggregation cardinality value; The first counting unit stores the subnet identifier. First characteristic intensity value and estimated polymer basis ; Indicates the storage location number The subnet identifier stored in the first counting unit of the second storage unit. Indicates the storage location number The feature intensity value stored in the first counting unit of the second storage unit Indicates the storage location number The aggregate cardinality estimate stored in the first counting unit within each second storage unit; The second counting unit stores the subnet identifier. Second characteristic intensity value and estimated polymer basis ; Indicates the storage location number The subnet identifier stored in the second counting unit of the second storage unit. Indicates the storage location number The feature intensity value stored in the second counting unit of each second storage unit Indicates the storage location number The estimated aggregation cardinality stored in the second counting unit of each second storage unit. .

7. The high-speed network traffic anomaly detection method according to claim 6, characterized in that, The method for storing and updating the corresponding subnet aggregation feature information based on data stream state information in the second storage unit includes: If the data stream status information update triggers the conflict handling mechanism, then obtain the subnet identifier of the current data packet in the data stream; Based on the subnet identifier, calculate the second index position of the current data packet in the storage location; The second index position is calculated using the following formula: ； in, Indicates the first The second storage unit corresponds to the second index position of the current data packet. This represents the second hash function. Indicates the subnet identifier. This represents the XOR operator. Indicates a random seed. This represents the modulo operator. Indicates the number of second storage units in the storage location; Based on the second index position, obtain the first feature strength value and the second feature strength value of the current data packet in the second storage unit; Get the element identifier of the current data packet; Calculate the real-time feature intensity value corresponding to the current data packet based on the element identifier of the current data packet; The real-time feature intensity value is calculated using the following formula: ； in, This represents the real-time feature strength value corresponding to the current data packet. This represents the function for calculating the number of leading zeros. This represents the third hash function. The element identifier that represents the current data packet; The real-time feature strength value is compared with the second feature strength value. If the real-time feature strength value is greater than the second feature strength value, the second feature strength value is replaced with the real-time feature strength value to obtain the updated second feature strength value; otherwise, the current data packet is discarded. If the first feature strength value is less than the updated second feature strength value, then the two values ​​are swapped.

8. A high-speed network traffic anomaly detection device, characterized in that, For implementing the high-speed network traffic anomaly detection method according to claim 1, the apparatus includes: The first acquisition module is used to acquire the data stream identifier to which the network data packets in each data stream belong; The first calculation module is used to determine the estimated network feature cardinality of the data stream in the storage location, which is used to characterize the size of the data stream, based on the data stream identifier. The storage location includes a first storage unit for storing and updating data stream status information and a second storage unit for storing and updating corresponding subnet aggregation feature information based on the data stream status information; The method for storing and updating data stream state information in the first storage unit includes: The data stream status information includes the data stream identifier, traffic growth frequency count value, and network characteristic cardinality estimate value; Based on the source address, destination address, and protocol type characteristics of each arriving network data packet in the data stream, generate the current data stream identifier corresponding to the network data packet; Based on the current data stream identifier, calculate the first index position corresponding to each network data packet in the storage location; The first index position is calculated using the following formula: ； in, Indicates the first Each first storage unit corresponds to the first index position of the network data packet. Indicates the first hash function, Indicates the current data stream identifier. This represents the XOR operator. Indicates a random seed. This represents the modulo operator. Indicates the number of the first storage units; If the first index position in the first storage unit is occupied, then the first storage unit contains a historical data stream identifier and the corresponding historical network feature cardinality estimate. The historical data stream identifier is compared with the current data stream identifier. If the comparison results are consistent, the traffic growth frequency count is incremented and the corresponding network feature cardinality estimate is updated. Otherwise, the conflict handling mechanism is triggered. The processing module is used to filter out abnormal data streams based on the estimated cardinality of network features; The second acquisition module is used to acquire the subnet identifier to which the network data packets in the abnormal data stream belong; The second calculation module is used to determine the abnormal subnet corresponding to the abnormal data stream based on the subnet identifier; The analysis module is used to perform correlation analysis on all abnormal data streams and abnormal subnets to determine the key source streams of each abnormal subnet.

Images