A method for identifying abnormal network traffic based on online change point detection

Abstract

This invention discloses a method for identifying abnormal network traffic based on online change point detection, belonging to the field of network traffic monitoring and analysis technology. The method includes: S1, deploying a collaborative registration sliding HyperLogLog estimator group consisting of N sliding HyperLogLog estimators, with the resulting N-1 ratio components constituting the cardinality projection vector at the current observation time; S2, independently maintaining state variables including an expansion cumulative deviation variable, an expansion baseline variable, a contraction cumulative deviation variable, and a contraction baseline variable for each component of the cardinality projection vector, and performing a bidirectional decoupling Page-Hinkley test; S3, under the condition that the expansion direction drift and contraction direction drift are respectively greater than preset expansion direction trigger thresholds and contraction direction trigger thresholds, outputting the network flow records within the sliding period corresponding to the current observation time as abnormal candidate traffic. This invention is applicable to the long-term stable online identification of abnormal network traffic under high-throughput links.

Description

Technical Field

[0001] This invention relates to the field of network traffic monitoring and analysis technology, and in particular to a method for identifying abnormal network traffic based on online change point detection. Background Technology

[0002] Network anomaly traffic identification is a crucial foundation for network security and traffic governance, widely applied in scenarios such as distributed denial-of-service defense, port scanning detection, and lateral movement tracing within intranets. As backbone link speeds increase to tens of gigabits per second (TPS) or even terabits per second, real-time anomaly traffic identification places stringent constraints on computing power and memory. In terms of deduplication statistics, the HyperLogLog algorithm, due to its sublinear memory complexity, is widely used for cardinality estimation in large-scale network flow deduplication. In deduplication counting of network flow fields ranging from millions to hundreds of millions, it can achieve a relative standard error of approximately 1.625% using a kilobyte-level register array. In terms of change point detection, sequential testing methods such as the Page-Hinkley test and cumulative sum test, due to their simple structure and controllable detection latency, have become one of the mainstream technical paths for online anomaly identification in time series data, accumulating rich application experience in fields such as hydrological monitoring, industrial process control, and financial indicator monitoring.

[0003] In existing research combining the two aforementioned techniques for identifying abnormal network traffic, the common approach is to perform HyperLogLog deduplication cardinality estimation on a single network flow field dimension to obtain a single scalar cardinality sequence, followed by a one-way Page-Hinkley test to determine the occurrence of anomalies. This existing approach reveals three prominent problems in practical deployment. First, the single-dimensional cardinality sequence can only reflect the deduplication scale change of traffic in a single attribute dimension, failing to capture the extended topological information between multi-level network flow attribute dimensions. It is severely insensitive to abnormal traffic that only causes attribute structure rearrangement without significant overall scale change. Second, if the anomaly manifests only as divergence in some attribute dimensions while convergence in others, the single-dimensional scalar observation will cancel out due to the inverse changes in different dimensions, thus failing to trigger detection. Secondly, the cumulative deviation of the classic Page-Hinkley test is accumulated by sign-based binning without attenuation or baseline constraints. Under long-term operation, the cumulative deviation shows a monotonically increasing trend, and after several hours or even days, it will inevitably exceed any fixed threshold, thus generating systematic false alarms. In a network monitoring scenario with 24 / 7 continuous operation, it is difficult to maintain stability. Furthermore, the one-way Page-Hinkley test only focuses on whether the baseline has shifted significantly upward or downward, and lacks the ability to effectively distinguish between two types of heterogeneous false alarm sources: slow baseline drift and sparse jitter in individual sampling windows. In actual deployment, the false alarm rate and the missed alarm rate often increase simultaneously, making it difficult to balance sensitivity and stability. Summary of the Invention

[0004] In view of this, the present invention provides a network anomaly traffic identification method based on online change point detection. It elevates single-dimensional scalar observations to multi-level attribute-extended topology characterization through cardinality projection vectors, eliminates systematic false alarms caused by long-term monotonically increasing cumulative deviations through a historical minimum value tracking mechanism of benchmark variables, and suppresses isolated false alarms caused by unidirectional heterogeneous disturbances through bidirectional simultaneous triggering judgment conditions. It significantly reduces the false alarm rate without sacrificing the detection rate, and has low memory usage and computational overhead, making it suitable for long-term stable online identification of network anomaly traffic under high-throughput links.

[0005] The technical solution adopted in this invention is as follows:

[0006] The network anomaly traffic identification method based on online change point detection, executed by a data processing device, includes:

[0007] S1. Deploy a collaborative registration sliding HyperLogLog estimator group consisting of N sliding HyperLogLog estimators, where N is a positive integer greater than or equal to 3. The N sliding HyperLogLog estimators correspond to one combination of network flow attributes and are arranged in a preset fixed sequence number. Divide the observation time according to the preset sliding period length. At each observation time, the N sliding HyperLogLog estimators output a cardinality estimate. Take the quotient of two adjacent cardinality estimates along the fixed sequence number from small to large. The resulting N-1 ratio components constitute the cardinality projection vector for this observation time.

[0008] S2. For each component of the base projection vector, maintain independent state variables including the expansion cumulative deviation variable, expansion reference variable, contraction cumulative deviation variable and contraction reference variable, and perform a two-way decoupling Page-Hinkley test. The expansion reference variable and contraction reference variable take the historical minimum values ​​of the expansion cumulative deviation variable and contraction cumulative deviation variable respectively, and obtain the expansion direction drift and contraction direction drift at the current observation time.

[0009] S3. If the expansion direction drift and contraction direction drift are greater than the preset expansion direction trigger threshold and contraction direction trigger threshold respectively, it is determined that a network traffic change point has occurred at this observation time, and the network flow record within the sliding period corresponding to this observation time is output as an abnormal candidate flow.

[0010] Furthermore, the network flow attribute combination consists of one or more fields selected from the five network flow fields: source IP address, destination IP address, source port number, destination port number, and transport layer protocol number; all sliding HyperLogLog estimators in the collaborative registration sliding HyperLogLog estimator group are assigned fixed sequence numbers in a predefined order, and the value range of the fixed sequence number is a continuous integer from 1 to N.

[0011] Furthermore, the fixed arrangement number is assigned in ascending order of the number of network flow fields contained in the network flow attribute combination corresponding to each sliding HyperLogLog estimator. There is a field inclusion relationship between the network flow attribute combinations corresponding to two adjacent sliding HyperLogLog estimators with fixed arrangement numbers. The field inclusion relationship is as follows: the network flow attribute combination corresponding to the one with the larger fixed arrangement number completely includes all the fields of the network flow attribute combination corresponding to the one with the smaller fixed arrangement number, and adds one new network flow field to the aforementioned field composition.

[0012] Furthermore, in step S1, a sliding window length K, which is the number of sliding periods, is preset. K is a preset integer greater than or equal to 1. The observation time is divided into a continuous sequence of observation times according to the sliding period length. The time interval between two adjacent observation times is equal to the sliding period length. The cardinality estimate output by each sliding HyperLogLog estimator at the current observation time is the estimate of the number of deduplicated elements corresponding to all input items written to the corresponding sliding HyperLogLog estimator in the most recent K sliding periods up to the current observation time.

[0013] Furthermore, in step S1, within the sliding period corresponding to each observation time, the data processing device captures all network flow records arriving within the sliding period corresponding to the current observation time through the network traffic acquisition module; for each captured network flow record, the following processing is performed on each sliding HyperLogLog estimator in the collaborative registration sliding HyperLogLog estimator group: extract the values ​​that correspond one-to-one with all network flow fields contained in the network flow attribute combination corresponding to the corresponding sliding HyperLogLog estimator from the captured network flow record, concatenate them into an attribute value string according to the internal preset order of the corresponding network flow attribute combination, and write the attribute value string as an input item into the corresponding sliding HyperLogLog estimator.

[0014] Furthermore, in step S1, after the coordinated registration of the sliding HyperLogLog estimator group completes the cardinality estimate output at each observation time, the cardinality estimates output by two adjacent sliding HyperLogLog estimators at the current observation time are grouped into an ordered pair according to the fixed arrangement number in ascending order. For each ordered pair, the cardinality estimate output by the one with the smaller fixed arrangement number is used as the numerator, and the cardinality estimate output by the one with the larger fixed arrangement number is used as the initial denominator. The initial denominator is compared with a preset minimum positive number, and the larger of the two is used as the final denominator for this quotient calculation. The preset minimum positive number is a pre-set constant greater than 0. The numerator is divided by the final denominator to obtain a ratio component. The resulting N-1 ratio components are arranged in ascending order according to the fixed arrangement number to form the cardinality projection vector at the current observation time.

[0015] Furthermore, in step S2, the state variables also include one mean variable and one count variable. The state variables consist of six variables: one mean variable, one count variable, one expansion cumulative deviation variable, one expansion reference variable, one contraction cumulative deviation variable, and one contraction reference variable. Before executing step S2 at the first observation time, the initial storage values ​​of the mean variable, count variable, expansion cumulative deviation variable, expansion reference variable, contraction cumulative deviation variable, and contraction reference variable corresponding to all N-1 components of the cardinality projection vector are all set to 0.

[0016] Furthermore, in step S2, after obtaining the cardinality projection vector at each observation time by completing step S1, the mean update process is performed on all N-1 components of the cardinality projection vector in ascending order of fixed sequence numbers: For the component currently being processed, the value of the current component in the cardinality projection vector at this observation time is recorded as the current input value; the current stored value of the counter variable corresponding to the current component is incremented by 1, and the result after incrementing is written back to the counter variable corresponding to the current component, and the stored value after writing back is recorded as the updated counter value; the difference between the current input value and the current stored value of the mean variable corresponding to the current component is calculated, the difference is divided by the updated counter value to obtain a quotient value, the quotient value is added to the current stored value of the mean variable corresponding to the current component, and the result after adding is written back to the mean variable corresponding to the current component; the current stored value of the mean variable corresponding to the current component after writing back is recorded as the mean reference.

[0017] Furthermore, in step S2, for the currently processed component, after completing the mean update process: the difference between the current input value and the mean reference is accumulated into the expansion cumulative deviation variable corresponding to the current component, and the accumulated result is written back to the expansion cumulative deviation variable corresponding to the current component. The stored value of the expansion cumulative deviation variable corresponding to the current component after this write-back is compared with the current stored value of the expansion reference variable corresponding to the current component, and the smaller value is taken as the update value of the expansion reference variable corresponding to the current component, and the update value is written back to the expansion reference variable corresponding to the current component; the difference between the mean reference and the current input value is accumulated into the contraction cumulative deviation variable corresponding to the current component, and the accumulated result is written back to the contraction cumulative deviation variable corresponding to the current component. The stored value of the contraction cumulative deviation variable corresponding to the current component after this write-back is compared with the current stored value of the contraction reference variable corresponding to the current component, and the smaller value is taken as the update value of the contraction reference variable corresponding to the current component. The updated value of the contraction reference variable is written back to the contraction reference variable corresponding to the current component. After processing all N-1 components of the cardinal projection vector at this observation time, for each component of the cardinal projection vector, the stored value of its corresponding expansion cumulative deviation variable after processing at this observation time is subtracted from the stored value of its corresponding expansion reference variable after processing at this observation time, and the difference is taken as its expansion direction drift component at this observation time. The stored value of its corresponding contraction cumulative deviation variable after processing at this observation time is subtracted from the stored value of its corresponding contraction reference variable after processing at this observation time, and the difference is taken as its contraction direction drift component at this observation time. The largest value among the expansion direction drift components corresponding to all N-1 components of the cardinal projection vector is taken as the expansion direction drift amount at this observation time, and the largest value among the contraction direction drift components corresponding to all N-1 components of the cardinal projection vector is taken as the contraction direction drift amount at this observation time.

[0018] Furthermore, in step S3, an expansion direction trigger threshold and a contraction direction trigger threshold are preset in the data processing device. Both the expansion direction trigger threshold and the contraction direction trigger threshold are preset constants greater than 0. Under the condition that the expansion direction drift is greater than the expansion direction trigger threshold and the contraction direction drift is greater than the contraction direction trigger threshold at the current observation time, the sliding period corresponding to the current observation time is marked as an abnormal flow period. All network flow records captured by the network flow acquisition module within the most recent K sliding periods corresponding to the current observation time are output as abnormal candidate flow to the downstream security handling module of the data processing device. At the same time, the stored values ​​of the mean variable, count variable, expansion cumulative deviation variable, expansion reference variable, contraction cumulative deviation variable, and contraction reference variable corresponding to all N-1 components of the cardinality projection vector are all reset to 0. After completing all the processing of steps S1 to S3 at the current observation time, the data processing device enters the next observation time and continues to execute steps S1 to S3 in sequence.

[0019] By employing the above technical solutions, this invention achieves the following beneficial effects: By deploying multiple sliding HyperLogLog estimators in an incremental manner with strictly defined field inclusion relationships between adjacent attribute combinations, the single-dimensional cardinal scalar sequence is elevated into a cardinal projection vector carrying multi-level network attribute extension topological semantics. This allows divergent or convergent behavior on any attribute dimension to be recorded with clear physical meaning by the corresponding ratio component, fundamentally overcoming the deficiency of insufficient sensitivity of existing single-dimensional scalar observations to attribute structure rearrangement anomalies. Furthermore, by independently maintaining cumulative deviation variables and benchmark variables for each component of the cardinal projection vector in the expansion and contraction directions, and continuously tracking the historical minimum value of the cumulative deviation variable using the benchmark variable, the detection statistic remains bounded during the stationary period. Structurally, this eliminates the long-term systematic false alarms caused by the monotonically increasing cumulative deviation in the classic Page-Hinkley test, enabling the method to operate stably and continuously for extended periods. The cumulative deviation variables in the expansion and contraction directions are decoupled from the baseline variable and do not cancel each other out. Combined with the conjunctive judgment condition that the bidirectional drift amounts simultaneously exceed their respective trigger thresholds, this prevents two types of heterogeneous false alarms—slow baseline drift and sparse jitter in individual sampling windows—from triggering judgments independently. Meanwhile, true structural anomalies, due to their inherent bidirectional duality, can still be identified with high confidence, thus significantly reducing the false alarm rate without sacrificing the detection rate. Based on the sublinear memory characteristics of HyperLogLog and the recursive state update mechanism, this method requires relatively low memory and has low computational overhead per observation time. It can support real-time detection of high-throughput links on general-purpose servers and edge computing hardware, demonstrating good engineering versatility and scalability. Attached Figure Description

[0020] Figure 1This is a schematic diagram of the overall construction of a collaborative registration sliding HyperLogLog estimator group consisting of 5 sliding HyperLogLog estimators in an embodiment of the present invention, and the cardinality projection vector obtained by quotienting adjacent cardinality estimates.

[0021] Figure 2 This is a schematic diagram of the sliding mechanism of the circular queue of the sub-window register group inside a single sliding HyperLogLog estimator in this embodiment of the invention, and the core process of obtaining the cardinality estimate of the current observation time by taking the maximum value of the register level and merging them.

[0022] Figure 3 This is a schematic diagram of the measured evolution curves of the four components of the cardinal projection vector on the observation time axis in an embodiment of the present invention. Detailed Implementation

[0023] The network anomaly traffic identification method based on online change point detection, executed by a data processing device, includes:

[0024] S1. Deploy a collaborative registration sliding HyperLogLog estimator group consisting of N sliding HyperLogLog estimators, where N is a positive integer greater than or equal to 3. The N sliding HyperLogLog estimators correspond to one combination of network flow attributes and are arranged in a preset fixed sequence number. Divide the observation time according to the preset sliding period length. At each observation time, the N sliding HyperLogLog estimators output a cardinality estimate. Take the quotient of two adjacent cardinality estimates along the fixed sequence number from small to large. The resulting N-1 ratio components constitute the cardinality projection vector for this observation time.

[0025] S2. For each component of the base projection vector, maintain independent state variables including the expansion cumulative deviation variable, expansion reference variable, contraction cumulative deviation variable and contraction reference variable, and perform a two-way decoupling Page-Hinkley test. The expansion reference variable and contraction reference variable take the historical minimum values ​​of the expansion cumulative deviation variable and contraction cumulative deviation variable respectively, and obtain the expansion direction drift and contraction direction drift at the current observation time.

[0026] S3. If the expansion direction drift and contraction direction drift are greater than the preset expansion direction trigger threshold and contraction direction trigger threshold respectively, it is determined that a network traffic change point has occurred at this observation time, and the network flow record within the sliding period corresponding to this observation time is output as an abnormal candidate flow.

[0027] Before starting the method on the data processing device, the parameters of the collaborative registration sliding HyperLogLog estimator group are loaded offline. In this embodiment, the total number of estimators is... Taking 5 as an example, the five network flow attribute combinations corresponding to the five sliding HyperLogLog estimators are arranged in ascending order of field inclusion as follows: In a fixed order, sequence 1 corresponds to {source IP address}, sequence 2 corresponds to {source IP address, destination IP address}, sequence 3 corresponds to {source IP address, destination IP address, destination port number}, sequence 4 corresponds to {source IP address, destination IP address, destination port number, source port number}, and sequence 5 corresponds to {source IP address, destination IP address, destination port number, source port number, transport layer protocol number}. This chained arrangement, expanding from the source entity to the complete 5-tuple, ensures that the ratio of the cardinality estimates output by two adjacent estimators directly corresponds to "the reciprocal of the deduplication factor resulting from adding one network flow field to the existing attribute dimension".

[0028] For example, suppose the number of unique source IP addresses observed in the most recent sliding periods is . , That is, the cardinality estimate at fixed sequence number 1; the number of independent (source IP address, destination IP address) pairs observed within the same time period is , That is, the cardinal estimate at position 2 of the fixed arrangement. Ratio component. The value is equivalent to "the reciprocal of the number of unique source IP addresses corresponding to each unique (source IP address, destination IP address) pair". This directly represents the average number of different destination IPs that each independent source IP connected to during a given time period. When several source IPs suddenly initiate connections to a large number of different destination IPs, typical examples include port scanning or distributed scanning. Compared to Faster growth Consequently, the traffic contracts; conversely, when the traffic converges to at least the target IP, typically when a target experiences a centralized distributed denial-of-service attack. Compared to Slower growth This leads to expansion. Any chain-like expansion or contraction behavior in any attribute dimension will be characterized by the ratio component at the corresponding position in the form of "upward drift" or "downward drift". Here, "the smaller index as the numerator and the larger index as the denominator" is used as the basis for unifying the direction of quotient calculation - it uniformly maps "radical expansion" to "ratio component downward drift" and "radical convergence" to "ratio component upward drift", which is formally neat, and the bidirectional drift calculation in subsequent steps does not need to consider the sign convention.

[0029] The fixed permutation sequence number is assigned under the constraint that "the field composition of the larger sequence number completely includes the field composition of the smaller sequence number," and can be arranged in a specific order. In deployments that focus on scanning behavior, the destination port number can be prioritized to capture the rapid expansion of the destination port set early. For example, sequences 1 to 5 can be {source IP address}, {source IP address, destination port number}, {source IP address, destination port number, destination IP address}, {source IP address, destination port number, destination IP address, source port number}, and a complete 5-tuple, respectively. In deployments that focus on lateral movement within the internal network, the attribute combination with the earlier sequence number can be prioritized to include the destination IP address, so as to expose the divergence changes of each source IP to the internal network target earlier. The fixed permutation sequence number is determined at each method startup and remains unchanged throughout the entire lifecycle of the method to ensure that the cardinality projection vectors generated at different observation times are strictly comparable semantically.

[0030] The five estimators are configured to remain consistent. , The registers have identical bit widths, which reflects "co-registration" at the data structure level. A deeper level of this is reflected in the hash function: all five estimators use the same 64-bit unencrypted hash function. This embodiment employs a 64-bit variant of MurmurHash3 and fixes the same 64-bit seed constant for all five estimators. For example... Alternatively, a pre-agreed constant value can be used, and this seed value remains unchanged during method execution. Sharing the same hash function and the same seed ensures that any two estimators, within their respective attribute coverage ranges, will perform the same pseudo-random projection on the exact same string of attribute values. Therefore, the cardinality estimation biases of the two estimators are bound to the same approximate error curve. The ratio component obtained by their quotient calculation only reflects the true cardinality relationship of the samples statistically, without introducing systematic perturbations related to the specific hash configuration. If different estimators use different hash configurations, even with identical input data, the obtained cardinality estimates will carry a second-order bias independent of the specific input. This bias contaminates the physical semantics of the ratio component and amplifies the false alarm rate of the detection statistics in step S2. This is precisely the engineering pitfall that "cooperative registration" aims to avoid.

[0031] The sliding behavior is implemented through a sub-window register group queue. In this embodiment, the preset sliding period length is 1 second, and the sliding window length is... , Based on the number of sliding cycles, each estimator internally maintains a circular queue of sub-window register groups of length 60. Each slot in the queue corresponds to one sliding cycle and contains 4096 registers. Whenever a network stream record arrives, register updates are performed only on the tail of the queue, i.e., the most recent sub-window register group. Every second, i.e., one sliding cycle, the data processing unit removes the head of the queue, i.e., the earliest sub-window register group, clears the stored values ​​of all 4096 registers, and then re-attaches it to the tail of the queue as the new most recent sub-window register group. This circular layout of "clearing and recycling" ensures that the memory occupied by any one estimator remains constant at 60 × 3072 = 184320 bytes, or approximately 180 kilobytes, during long-term operation. The five estimators collectively occupy approximately 900 kilobytes, which can reside in the L2 or L3 cache of modern servers, thus supporting the writing of millions of network stream records per second.

[0032] At each observation time, i.e., at the end of each sliding cycle, when any estimator outputs the cardinality estimate for that observation time, it first takes the largest value among the stored values ​​with the same register number in its internal 60 sub-window register groups to obtain the merged register stored value. (Register numbering...) The merged stored values ​​satisfy:

[0033]

[0034] in This indicates the first [estimator] in the estimator queue. The sub-window register group is numbered The stored value on Indicates the numbering after the merger The maximum stored value. Based on all 4096 merged stored values, the cardinality estimate for this observation time is calculated using the HyperLogLog standard cardinality estimation formula:

[0035]

[0036] in For the debiasing correction constant, for The situation is taken Substitute get ; Indicates all Sum of the register numbers; The cardinality estimate is numerically equal to the estimated number of deduplicated elements corresponding to all input items written to the estimator in the most recent 60 sliding periods up to the current observation time.

[0037] The method of merging by taking the maximum value is not a temporary workaround in engineering, but rather stems from the fundamental semantics of the HyperLogLog register: register What is stored is "mapped to bucket" The maximum value of the leading zero count of the hash tail of all input items is incremented by 1. The HyperLogLog registers of the two sub-sample sets are maximized at each position, and the resulting register is exactly equivalent to the HyperLogLog register corresponding to the union of these two sub-sample sets. This union equivalence allows cardinality estimation across sub-windows to be naturally achieved by maximizing the register level without retaining any original network flow records. This is the foundation for HyperLogLog to support sliding window deployment in sublinear memory, and a key support for the collaborative registration sliding HyperLogLog estimator group to perform real-time processing of carrier-grade traffic on mainstream server hardware.

[0038] In the small base range, directly substituting into the above formula yields the result. There will be a significant positive deviation, therefore this embodiment uses linear counting correction: when And when there is a register that is currently zero, Replace the original Value, of which It represents the number of registers whose stored value is still 0 after merging. It is the natural logarithm. This correction ensures that the cardinality estimate remains stable during sparse periods with only tens to hundreds of deduplicated elements per second.

[0039] The input items are generated following a fixed byte order concatenation standard. Taking the attribute combination {source IP address, destination IP address, destination port number} corresponding to sequence number 3 as an example, after the data processing device captures a network flow record, it extracts 4 bytes of source IP address, 4 bytes of destination IP address, and 2 bytes of destination port number, and concatenates them into a 10-byte string in the order of "source IP address, destination IP address, destination port number". In the IPv4 scenario, this string is 10 bytes long; in the IPv6 scenario, the source and destination addresses are each 16 bytes, for a total string length of 34 bytes. All fields are concatenated in network byte order, i.e., big-endian. This convention avoids the problem of the same logical flow being mapped to different byte strings due to differences in the local byte order of different hosts. This byte string is the attribute value string corresponding to sequence number 3, and it serves as an input item in updating the sequence number 3 estimator. Similarly, the input item corresponding to sequence number 1 is the 4-byte or 16-byte representation of the source IP address, while the input item corresponding to sequence number 5 is a byte concatenation of a complete 5-tuple, with a length of 13 bytes in the IPv4 scenario. The same network flow record is synchronously dispatched to 5 estimators, generating 5 different attribute value strings and 5 independent write events. This "5-way parallel synchronous write" is a manifestation of collaborative registration in terms of execution rhythm, ensuring that the sample set seen by the 5 estimators at each observation time strictly corresponds to the same time period and the same batch of network flow records, so that the 5 cardinality estimates output have comparability due to being from the same source and at the same time.

[0040] Each time an input item is written, the following processing is performed: First, calculate the hash value of the input item using a 64-bit MurmurHash3 algorithm. , A 64-bit unsigned integer; truncation The lower 12 bits are used as the register number ,Right now , The value range is from 0 to 4095; then truncate. The remaining high 52 bits are denoted as ,statistics The leading zero count for this write is obtained by adding 1 to the number of consecutive 0s starting from the most significant bit in the binary representation.

[0041]

[0042] in Indicates to Find the number of consecutive zeros before the most significant bit of a 52-bit binary string. The value ranges from 1 to 53. At the hardware implementation level, This can be completed in a single cycle using the LZCNT instruction of a modern CPU or the CLZ instruction of an ARMv8, which is the engineering basis for using this statistic as the basic counting unit. Then... With the queue tail sub-window register group in number The current stored values ​​are compared, and the larger value is used as the value in the register group of that sub-window. The updated value. This "maximum value update" operation ensures that even if the same input item is written multiple times within a sliding cycle, the value stored in the register will not increase repeatedly. This is the fundamental mechanism by which cardinality estimation can correctly reflect the "number of deduplicated elements" rather than the "frequency of events".

[0043] After all five estimators have completed writing to each network flow record, at the exact moment the observation ends, each of the five estimators independently completes its own sub-window merging and cardinality estimation calculation, resulting in five cardinality estimates, denoted as follows: The subscript is the fixed sequence number.

[0044] Next, the ratio components are constructed. Following a fixed, ascending sequence of numbers, the quotients of adjacent cardinal estimates are calculated sequentially, for a total of [number] operations. The quotient operation is performed for the 1st time. Secondary quotient ( ), molecule take The initial denominator is taken as Before performing the division operation, first compare the initial denominator with the preset minimum positive number. We perform a numerical comparison and use the larger of the two values ​​as the final denominator for this quotient calculation:

[0045]

[0046] Indicates the first The final denominator of the quotient operation is then calculated. Then the... The ratio components are:

[0047]

[0048] That is, the cardinal projection vector in the ordinal number The component value at that location. In this embodiment... The value is set to 1, which can be understood as "acknowledging that there is at least one deduplicated element in any observation window". The reason for this is that the cardinality estimate may drop to 0 or approach 0 in a sliding window with extremely sparse or no samples. In this case, using it directly as the denominator will not only trigger numerical overflow or division by zero anomalies, but even if the above anomalies are avoided, the resulting ratio will also have a drastic order of magnitude jump in floating-point operations, thus injecting a large number of spurious drifts into the subsequent S2 step. The introduction of this is equivalent to setting a safety floor in the denominator direction, ensuring that the ratio component slowly tends to a small value during the sparse window period rather than producing a drastic jump.

[0049] The specific value can be adjusted according to the actual deployment: in low-throughput, sparse traffic scenarios, it can be 0.5 or even 0.1 to improve sensitivity to small-scale traffic anomalies; in backbone network-level high-throughput scenarios, it can be 5 or 10 to more proactively suppress small spurious drifts caused by noise estimated by HyperLogLog itself. Any The finite positive numbers do not affect the mathematical correctness of the change point detection in step S2, but only affect the sensitivity range of the method for extremely low flow rates.

[0050] After performing four quotient operations, the four ratio components are arranged in ascending order according to a fixed sequence number to obtain the cardinal projection vector at this observation time:

[0051]

[0052] That is, the current observation time The corresponding cardinal projection vector will be used as the input for the subsequent S2 step.

[0053] refer to Figure 1The five rounded rectangles arranged horizontally in the diagram represent the five components of this estimator group, labeled from left to right as Estimator 1, Estimator 2, Estimator 3, Estimator 4, and Estimator 5, strictly corresponding one-to-one with the fixed sequence numbers 1 to 5. The upper half of each rounded rectangle contains a list of fields for the network flow attribute combination corresponding to that estimator, while the lower half is separated from the field list by a horizontal line. Specifically, Estimator 1 corresponds to {source IP address}, Estimator 2 corresponds to {source IP address, destination IP address}, Estimator 3 corresponds to {source IP address, destination IP address, destination port number}, Estimator 4 corresponds to {source IP address, destination IP address, destination port number, source port number}, and Estimator 5 corresponds to {source IP address, destination IP address, destination port number, source port number, protocol number}. From estimator 1 to estimator 5, each pair of adjacent estimators is connected by a horizontal arrow with "+1 field" marked above the arrow. This visually represents the field inclusion relationship—the estimator with the larger fixed index completely includes all the fields of the smaller estimator and adds one new network flow field to the aforementioned field composition. This chain-like arrangement of fields appended hierarchically makes the ratio of the cardinality estimates output by two adjacent estimators physically equivalent to the reciprocal of the deduplication factor for each additional network flow field added to a certain attribute dimension of the network flow. This elevates the single-dimensional cardinality evolution to an extended topological characterization across multiple attribute dimensions. Below each rounded rectangle, a vertically downward-pointing arrow segment extends, with the cardinality estimate output by that estimator at the current observation time written at the end of the arrow. The five cardinality estimates are labeled from left to right as follows: , , , , The subscript indicates a fixed sequence number. The second row contains four division operation nodes, arranged in circles and written inside the circles. The circle's center x-coordinate is located slightly to the right of the midline between two adjacent estimators. Each division node receives cardinality estimates from two adjacent estimators, with the smaller of the fixed-order indices used as the numerator and the larger as the denominator. The names of the ratio components obtained from this quotient are labeled horizontally outside the circle on the right side, from left to right: , , , Below each division operation node, the vector is passed down again by vertical segments with arrows, eventually merging into the horizontal rectangle at the bottom of the graph. This rectangle is the container for the cardinality projection vector, and its interior is evenly divided into four cells by short vertical lines, with the code written in the center of each cell from left to right. , , , The complete mathematical expression of the cardinality projection vector is written centered above the container. This constitutes the core signal generation mechanism of this method: the observed object is no longer a single scalar cardinal sequence, but a 4-dimensional vector sequence carrying multi-level attribute incremental semantics. The divergent or convergent behavior of any network attribute dimension will be recorded by the ratio component at the corresponding position in the form of "drifting down" or "drifting up", providing a clearly physically meaningful input for the subsequent bidirectional decoupling Page-Hinkley test.

[0054] refer to Figure 2 At the top of the image is a horizontal arrow that runs across the entire map, pointing from left to right. The text above it reads "Earliest Time". "Latest Time" indicates the chronological order in which the sub-window register groups are arranged from left to right in the circular queue. Below it are eight rectangles of equal width and height, representing the length of the sliding window in this embodiment. The corresponding eight sub-window register groups are labeled "Number 1" to "Number 8" at the top center of each rectangle, and separated from the internal register list by a horizontal line at the bottom. Each rectangle contains a vertical list of the current stored values ​​of four registers, in a formatted way. Writing, among which Number the registers To store values, such as the values ​​listed from top to bottom inside the "first" rectangle. , , , The contents of the "8th" rectangle are listed from top to bottom. , , , It should be noted that this diagram, for illustrative purposes only, shows the stored values ​​of the first four registers within each sub-window register group. In actual deployment, each sub-window register group contains a total of [number missing]. One register, This is a precision parameter, typically with a value of 12. Each rectangle has a short vertical line extending from its lower edge. These eight lines converge below the rectangle into a horizontal beam running the entire width of the rectangle. From the center of the beam, they merge into a thick line segment with an arrow pointing downwards towards the central rounded rectangle. This central rounded rectangle is the register-level maximum value merging node. Inside, the title "Register-level Maximum Value Merging" is written on the upper line, and the formal definition is written on the lower line. ,in This indicates the number of estimators in the circular queue. The sub-window register group is numbered The stored value on Indicates the numbering after the merger The maximum stored value. This merging method of taking the maximum value originates from the union equivalence of the HyperLogLog register—taking the maximum value of the HyperLogLog register of two sub-sample sets at each position, the resulting register is exactly equivalent to the HyperLogLog register corresponding to the union of these two sub-sample sets. Therefore, cardinality estimation across sub-windows can naturally be achieved by taking the maximum value at the register level, without retaining any original network flow records. The rounded rectangle in the middle is then connected to the output rounded rectangle at the bottom of the figure by a vertical arrow segment pointing downwards. The output rounded rectangle has "cardinality estimate at this observation time" written on the top line and the complete estimation formula written on the bottom line. ,in For the debiasing correction constant, For the total number of registers, Indicates all Sum of register numbers, That is, the output cardinality estimate is numerically equal to the most recent estimate up to the current observation time. The estimated number of deduplicated elements corresponding to all input items written to the estimator within a sliding period. The figure clearly illustrates the sliding semantic implementation path of "parallel writing by time buckets, register-level merging at the current observation time, and unified estimation after merging".

[0055] In the data processing device, independent state variable storage areas are allocated for each of the four components of the radix projection vector. Each component corresponds to six floating-point variables: mean variable, count variable, expansion cumulative deviation variable, expansion base variable, contraction cumulative deviation variable, and contraction base variable. This embodiment uses double-precision floating-point numbers, i.e., 64-bit IEEE 754, with each variable occupying 8 bytes, and the state of a single component occupying 48 bytes, for a total of 192 bytes for the four components. Even if the number of components is expanded to 8 to 16, the total state size is only on the order of 0.4 to 0.8 kilobytes, which can reside entirely in the CPU's L1 data cache to ensure that the update latency remains stable within tens of nanoseconds.

[0056] Before the method starts, the data processing unit initializes all 24 variables to floating-point numbers 0.0. This initial value selection is not for convenience or convention, but rather to align with the semantics of the cardinal projection vector generated in step S1 at the first observation time—the first observation time has no prior information, all "historical minimums" have not yet been formed, the mean variable does not carry any samples, and the count variable represents the sample count, so starting from 0 is semantically appropriate. Any non-zero initial value would require additional justification to explain its relationship to the physical fact of the "blank first observation time," thus introducing additional bias.

[0057] Entering each observation time and delivering the cardinality projection vector by step S1. Then, the data processing device processes data in ascending order of a fixed sequence number—that is, it processes data first. Then process And so on—update the state of each of the four components one by one. Indicates the current observation time The output cardinality projection vector. This indicates that the vector is in a fixed sequence number. The component values ​​at the location. The "one-by-one" rather than "parallel" update order does not have a mandatory semantic impact in this embodiment, because the state variables of the four components are independent of each other and do not overlap; however, maintaining a definite order ensures that different implementation versions are strictly reproducible in numerical terms, which is very important in engineering scenarios such as debugging, playback, and grayscale verification.

[0058] For the component currently being processed, the data processing device first reads the value of that component at the current observation time and records it as the current input value. Then, increment the current stored value of the counter variable in memory by 1 and write it back. Let the counter variable for this component have a value of [value] before processing this input value. The value after processing is . This involves updating the count value. Immediately following this, an online update is performed on the mean variable—recording the mean variable value of this component before the current observation time as [value missing]. The updated value is Both conditions are met:

[0059]

[0060] in This is the input value for this time. It updates the count value. It is the stored value before the mean variable is updated. This is the updated stored value of the mean variable. This recursive form is the online mean algorithm proposed by Welford in 1962. Compared to the naive approach of "first summing all samples and then dividing by the total number of samples," the Welford form only needs to store the mean and sample count from the previous time step, without retaining any historical samples. Therefore, it does not accumulate floating-point rounding errors or occupy memory related to the total number of samples during long-term operation. When the current observation time is completed, the data processing device will... Write it back directly to the mean variable, and then... Recorded as mean reference This serves as the benchmark for this cumulative deviation update.

[0061] The reason for using the updated mean Instead of the mean before the update As a mean reference, considering the boundary behavior at the first observation time. If changed to use... For reference: Processing of the first observation time hour, , ,and Under normal network traffic conditions, the value typically falls within a non-zero positive value ranging from 0 to several orders of magnitude, such as 0.3, 0.5, or 1.5. This means that the inflated cumulative bias variable will abruptly change at the first observation time. This non-zero positive number indicates that the expansion baseline variable remains at 0, while the expansion direction drift component has already reached [value] at the first observation time. Even without any actual sudden changes in flow, this drift component will remain as a "pseudo-jump" in the state variable for multiple observation moments before being gradually absorbed by the mean convergence. (Change to...) This potential problem does not exist as a reference point—the initial observation time. , The cumulative deviation variable of expansion naturally remains at 0, the baseline variable of expansion also remains at 0, and the drift component of expansion direction is 0, which conforms to the setting that "there is no history to speak of at the first observation time and no drift signal should be given".

[0062] After completing the mean update, the data processing device sequentially updates the state of the expansion direction. The cumulative expansion deviation variable of this component before the current observation time is recorded as [value missing]. The processed value at this observation time is:

[0063]

[0064] in It is the new stored value of the cumulative expansion deviation variable at this observation time. This is the input value for this time. This is the average reference for this time. Then use... The value of this component relative to the expansion baseline variable prior to this observation time. Perform a numerical comparison, and write the smaller value back to the inflation baseline variable, denoted as:

[0065]

[0066] in This is the new stored value of the expanded baseline variable at the current observation time. The minimum-value update rule means that the expanded baseline variable always tracks the historical minimum value of the expanded cumulative deviation variable from the first observation time to the current observation time. Once the cumulative deviation variable reaches a lower value at a certain observation time than at any previous time, the baseline variable shifts down accordingly; once the cumulative deviation variable starts to drift upward, the baseline variable remains unchanged.

[0067] The updated structure in the contraction direction is strictly symmetrical with that in the expansion direction. Let the cumulative contraction deviation variable for this component before the current observation time be denoted as... The processed value at this observation time is:

[0068]

[0069] in It is a new stored value for shrinking the cumulative deviation variable. and The meaning is the same as above. Accordingly, the shrinking baseline variable is updated according to the same minimum tracking rule:

[0070]

[0071] in This is the new stored value of the contracted baseline variable at this observation time. Note that... and Each update proceeds synchronously with an increment of strictly opposite sign, i.e. This holds true at any observation time; however and Each variable independently tracks its own historical minimum value of the cumulative deviation variable, and the two are not opposites of each other—one tracks the "historical lowest point" of the cumulative deviation curve, and the other tracks the corresponding position of the "historical highest point" of the cumulative deviation curve after the sign is flipped. This is the purpose of maintaining the four variables independently: although the cumulative deviation variable pair can be folded into one, the benchmark variable pair must be maintained separately to handle the sensitivity of upward and downward drift respectively.

[0072] The design motivation for minimum tracking stems from a repeatedly overlooked yet fatal detail: the naive practice of "accumulating by sign"—that is, accumulating positive deviations to the expanding cumulative variable only when the current input value is greater than the mean reference, and accumulating negative deviations to the contracting cumulative variable otherwise—leads to both cumulative variables exhibiting monotonically increasing values. Under any normal fluctuation, given a sufficiently large number of observations, these two monotonically increasing cumulative values ​​will inevitably exceed any fixed finite threshold, thus triggering systematic false alarms. Minimum tracking uses "the current cumulative deviation value minus its historical minimum value" as the final test statistic, ensuring that this difference remains statistically bounded during stationary periods where there is no true distribution drift—more precisely, under conditions of independent and identically distributed samples with zero mean, the expected growth rate of the difference is... The iterative logarithmic law can be further tightened, resulting in a value far lower than the linear growth rate of the difference after the actual change point occurs. This asymptotic difference of "sublinearity during the stationary period and linearity after the turning point" is the core basis of the cumulative sum test proposed by Page in 1954 and the corresponding sequential test perfected by Hinkley in 1971.

[0073] After processing the four components at the current observation time, the data processing unit proceeds to the drift component extraction stage. For each component, its expanded cumulative deviation variable is processed and stored at the current observation time. Subtract the stored value of its expansion reference variable after processing at the current observation time. The difference obtained is recorded as the expansion direction drift component of that component at the current observation time. , Representing components At the observation time The expansion direction drift component, according to From the definition, we can know Similarly, the contraction direction drift component is obtained. .

[0074] Then, the largest value among the four dilation direction drift components is taken as the dilation direction drift at this observation time, i.e. Similarly, the contraction direction drift is obtained by taking the largest value from the contraction direction drift components corresponding to the four components. , and That is, the two directional drift values ​​output at this observation time are used as inputs for step S3.

[0075] The use of "maximum value" as the aggregation method, rather than "average" or "norm," is rooted in the sparsity of abnormal network traffic itself: true structural anomalies are often concentrated at only one or a few sequence positions at the network flow attribute level. For example, port scanning mainly causes a sharp increase in the cardinality estimate corresponding to attribute combinations with later sequence numbers covering the destination port number, while the cardinality estimates at the source IP and destination IP dimensions may not change synchronously. If the drift components of the four components are averaged, the component that truly carries the abnormal signal will be diluted by the other three undisturbed components, artificially weakening the detection sensitivity. Taking the maximum value, on the other hand, ensures that any significant drift of any component is fully propagated to the aggregated drift amount, neither missing any local anomalies in a single dimension nor sacrificing sensitivity due to averaging. Theoretically, it is also possible to use... While aggregation methods such as norms can be used, in sparse scenarios where typically only one or two of the four components carry effective signals, The detection capabilities of norm and maximum value are not significantly different, and the maximum value only requires 4 comparisons and assignments in hardware implementation, which is far lower than the implementation complexity of square root operation.

[0076] The following uses a set of specific numerical values ​​to demonstrate the state evolution of a certain component. Assume that this component acquires values ​​of 0.50, 0.52, 0.48, 0.51, 0.80, 0.82, and 0.50 at seven consecutive observation times. The first four values ​​fall within the steady-state noise range, the fifth and sixth values ​​reflect a structural abrupt change, and the seventh value returns to the original steady state. The process is as follows: At observation time 1... At that time, the count variable increases from 0 to 1, the mean variable is updated from 0 to 0.50, the mean reference is also 0.50, and the inflation cumulative deviation variable is accumulated. The value remains 0, and the inflation baseline variable is taken as... Expansion direction drift component The contraction direction also remains completely 0. Observation time 2 processing. At that time, the mean variable was updated to 0.51, the mean reference was 0.51, and the cumulative deviation variable was accumulated. The value is then 0.01, the expansion baseline variable remains 0, and the expansion direction drift component is 0.01; the contraction cumulative deviation variable is accumulated. Later The baseline variable was updated to shrink. The contraction direction drift component is 0. During observation times 3 to 4, the values ​​oscillate randomly within a small range. The expansion and contraction direction drift components always fall within the range of 0 to 0.02, reflecting that the statistics in both directions are effectively locked at a low level by the minimum value tracking mechanism of the benchmark variable under steady state.

[0077] Entering observation time 5, the current input value of 0.80 suddenly increases by approximately 0.30 compared to the previous steady-state mean of 0.5025. The count variable increases to 5, the mean variable is updated to 0.562, and the mean reference is 0.562. The cumulative deviation variable is accumulated. After that, from the time before this observation It jumped to 0.2355; the inflation baseline variable remained at its previous lowest value. The expansion direction drift component is This drift component is more than an order of magnitude larger than all the drift components in the previous steady state, clearly marking the occurrence of this structural abrupt change. Simultaneously, the cumulative deviation variable is shrunk. ,achieve The contraction of the benchmark variable synchronously sinks to The contraction direction drift component is 0, indicating that this mutation is a pure "upward drift" signal without any "downward drift" component. At observation time 6, the input value remained at 0.82, and the expansion direction drift component continued to accumulate upward to 0.4605. Even at observation time 7, when the input value dropped to 0.50, the expansion direction drift component remained around 0.37—this is a design characteristic of this statistic: it remembers the "maximum upward drift amplitude since the last reference point." Even if the instantaneous input value returns to normal, this memory will not fade on its own; it can only be cleared by the state reset performed in step S3 after the change point determination is established. This rhythm is consistent with the engineering logic of "determine first, then clear, then recount."

[0078] Regarding threshold settings, this embodiment sets both the expansion direction trigger threshold and the contraction direction trigger threshold to 0.15. This value is based on offline analysis of network traffic data from several enterprises: under steady-state conditions, the 99.9 percentiles of the drift components in both directions roughly fall between 0.05 and 0.08; when a true structural anomaly occurs, the drift component can cross the 0.20 to 1.0 range within 1 to 5 observation times. Therefore, the value of 0.15 simultaneously satisfies the requirements of "approximately twice the upper bound of the steady-state period to suppress false alarms" and "approximately 30% lower than the lower bound of the true anomaly to ensure no missed detections." The threshold can be adjusted under different deployment environments: for backbone link monitoring deployments with high noise levels, it can be increased to 0.30 to 0.50; for relatively clean internal server cluster monitoring deployments, it can be decreased to 0.08 to 0.10. The thresholds for the two directions do not need to be equal. In deployments where a certain type of anomaly is known to be dominated by unidirectional drift, a lower threshold can be set for that direction and a higher threshold for the other direction to balance sensitivity and false alarm rate. However, in the absence of prior information in a general deployment, taking symmetrical values ​​is the safest starting point.

[0079] Thus far, the work completed by step S2 at this observation time is as follows: receiving the 4-dimensional cardinal projection vector output by S1, updating the six state variables (count variable, mean variable, expansion cumulative deviation variable, expansion reference variable, contraction cumulative deviation variable, and contraction reference variable) for the four components in a fixed sequence, extracting the expansion direction drift component and contraction direction drift component of each of the four components, aggregating the maximum values ​​to obtain the expansion direction drift and contraction direction drift at this observation time, and delivering these two scalars to step S3 to complete the threshold determination and anomaly output.

[0080] The expansion direction drift at this observation time is completed in step S2. Shift in direction of contraction After calculating and delivering these two scalars, the data processing device enters the decision and output phase. This phase performs a seemingly simple but actually complex task that requires meticulous constraints: first, it determines whether the current observation time constitutes a network traffic inflection point; if the determination is correct, it completes four interconnected tasks at once: marking abnormal traffic cycles, outputting abnormal candidate traffic, resetting the state, and preparing for the next observation time.

[0081] Before proceeding to this step, two floating-point constants are pre-reserved on the data processing device: the expansion direction trigger threshold. Trigger threshold with contraction direction Both are preset constants greater than 0. In this embodiment, we take... The two thresholds are loaded into a read-only memory segment when the method starts and remain unchanged during runtime. Two auxiliary boolean flags are also maintained. and , respectively characterize whether the expansion direction drift and contraction direction drift exceed the threshold at the current observation time. Both are transient variables generated during the operation of the method and are recalculated at the beginning of each observation time, and do not constitute a persistent state across observation times.

[0082] The form of the decision logic is:

[0083]

[0084] Then perform the conjunction operation. If the sum of all the elements is true, then the current observation time is determined. A network traffic change point occurs at this location. The strict greater than / less than greater than or equal to is used here to avoid [the issue]. or Instability in decision-making arises when values ​​are accidentally equal to a threshold in floating-point operations: floating-point comparisons are easily affected by rounding errors when the values ​​are close to the boundary. Using a strict greater than threshold can keep the decision result stable under minor numerical perturbations. In deployments where numerical precision is more important, this can also be rewritten as... The form introduces a very small safety margin. , Take in to The magnitude is commensurate with the precision of floating-point representation. However, in this embodiment, since the threshold is 0.15 and the drift magnitude is between 0.0 and several units, the difference in magnitude between the two is significant, so there is no need to introduce additional [precision]. It already possesses sufficient stability.

[0085] The core difference between this method and the classic one-way Page-Hinkley test lies in using conjunction rather than disjunction—that is, requiring both directions to exceed the threshold simultaneously, rather than only one direction exceeding the threshold. The classic one-way test only concerns whether the baseline has shifted significantly upwards or downwards. However, in scenarios with abnormal network traffic, it faces two distinct sources of false alarms: first, the gradual change in the baseline itself, such as the slow migration of traffic structure during business periods, can cause the drift in either direction to occasionally emerge over a long period; second, data sparsity within individual sliding periods can cause the cardinality estimate to have an instantaneous deviation from the mean exceeding the threshold due to the statistical noise of HyperLogLog itself. Both of these false alarm sources exhibit a "one-way" characteristic in large-scale real-world deployments—gradual baseline changes can only cause drift in one direction, not both simultaneously, and instantaneous sparsity jitter only stands out in a specific direction. The requirement that both directions simultaneously exceed the threshold prevents false alarms from triggering a decision independently. True structural anomalies, whether port scanning, distributed denial-of-service, or lateral movement, simultaneously induce expansionary drift in some components and contractionary drift in others on the cardinality projection vector. The former is driven by imbalances such as "divergence of the destination attribute set" or "convergence of the source attribute set," while the latter appears synchronously as the dual of the former. Conjunctive decision captures the essential characteristic of the anomaly—a "holistic rearrangement of network traffic structure"—treating the presence or absence of duality as an observable signal, rather than relying on the absolute amplitude of either direction.

[0086] When conjunction does not hold, that is and If at least one of the observations is false, the data processing device does not perform any change-point related operations, but directly returns control to the main loop, waiting for the next observation moment. During this period, the network flow records captured within the sliding period corresponding to this observation moment expire and are eliminated in the network traffic acquisition module according to the normal lifecycle management process. They will not be output as abnormal candidate traffic, nor will they trigger any downstream security actions.

[0087] When the conjunction is valid, the data processing device enters the exception handling sub-process. This sub-process contains four transactions executed in a strict order. Failure of any transaction or disorder of the order will result in a semantic error in the overall method. Therefore, in this embodiment, it is completed serially in the form of a single atomic operation in engineering.

[0088] The first task is to mark the sliding period corresponding to the current observation time as an abnormal traffic period. This marking is physically implemented by flipping the "abnormal" bit on the sliding period entry corresponding to the current observation time from 0 to 1 in the sliding period metadata table maintained by the data processing device, and simultaneously writing a timestamp value, i.e., the absolute time corresponding to the current observation time. This marking itself does not change the content of any network flow records within the sliding period; it only provides downstream processing with metadata indicating "this period requires attention," so that subsequent actions such as auditing, backtracking, and review can accurately pinpoint the specific time period when the anomaly occurred.

[0089] The second task is to output abnormal candidate traffic. Specifically, this involves first determining the nearest [transaction / flow] corresponding to the current observation time. The network traffic acquisition module captures and temporarily stores all network flow records within the periodic buffer during the time window covered by each sliding cycle, and then delivers them to the downstream security processing module in the form of batch data streams via the internal data path. In this embodiment... That is, abnormal candidate traffic includes all network flow records within 60 seconds before the change point determination time.

[0090] The reason for using "recent" Using "the current sliding period" instead of "the current sliding period corresponding to this observation time" as the output range involves the inherent phenomenon of "post-hoc positioning bias" of the change point. The Page-Hinkley test is a sequential test, which essentially makes a judgment only when the accumulated bias reaches the threshold, rather than at the moment the anomaly occurs. In any sequential test, there is always a non-zero delay between the start of the accumulated bias and the time when the accumulated bias crosses the trigger threshold. This delay increases with the increase of the threshold and the increase of sample noise. This means that step S3 determines the current observation time. When a turning point occurs, the real structural anomaly may already be present. It started happening at this point in time. This indicates a detection delay, which, under the parameter configuration of this embodiment, typically falls within the range of 5 to 30 seconds after offline evaluation. If only the network flow record corresponding to the current observation time and a sliding period of 1 second is delivered downstream, the historical records that truly carry the abnormal behavior will be missed, and the downstream processing module will not be able to see the complete chain of evidence for the anomaly. Expanding the output range to the most recent... The 60-second time window covered by each sliding cycle is strictly the same as the sliding window length of the collaborative registration sliding HyperLogLog estimator group in step S1. This ensures that backtracking omissions caused by detection delays are completely covered—all network flow records involved in cardinality estimation and thus triggering this change point determination are included in the abnormal candidate traffic. This design of "using the sliding window length as the backtracking length" is, in engineering semantics, "all samples involved in this determination are included in the candidate output," which is consistent with the detection algorithm itself.

[0091] The periodic buffer is implemented using a circular buffer. The circular buffer stores the most recent data in chronological order. For each sliding cycle, the network flow records corresponding to the earliest sliding cycle are discarded in batches to make room for the new record of the current cycle. Each network flow record is typically stored in a compact binary format, with a typical size of 40 to 128 bytes in IPv4 scenarios and increasing to 60 to 160 bytes in IPv6 scenarios, depending on whether it includes extended fields such as transport layer flags, TTL, payload length, and application layer identifiers. Based on an enterprise-level backbone throughput of 1 million records per second, the periodic buffer occupies approximately 2.4 gigabytes to 7.7 gigabytes in IPv4 scenarios, which can easily reside in the main memory of modern servers; in edge deployments and scenarios with lower throughput, the required memory decreases linearly. This buffer is strictly aligned in length with the circular queue of the sub-window register group used by the estimator group in step S1, but the granularity of the information they carry is different—the sub-window register group of the estimator group only retains the HyperLogLog register required for cardinality estimation, while the periodic buffer retains each original network flow record. The former is used for change point detection, while the latter is used for evidence submission after a change point determination is established. The two types of data structures each perform their own functions and do not conflict with each other.

[0092] In this embodiment, the output format uses a streaming interface instead of batch file packaging to keep the downstream processing response latency within 100 milliseconds. The data processing device writes network flow records into the downstream processing receiving queue one by one in chronological order, and appends a header to the beginning of each batch of output. The header includes the timestamp of the current observation time corresponding to the abnormal traffic cycle and the expansion direction drift amount involved in this determination. Shift in direction of contraction The corresponding fixed sequence number positions, i.e., the component number contributing the largest expansion drift component and the component number contributing the largest contraction drift component, and the sliding window length. Core metadata such as sliding cycle length. This metadata provides downstream processing with the minimum set of information to trace the source of this anomaly without having to reverse-query the internal states of steps S1 and S2.

[0093] The third transaction is state reset. This involves overwriting the stored values ​​of the mean, count, expansion cumulative deviation, expansion baseline, contraction cumulative deviation, and contraction baseline variables maintained in step S2 for this component and all the other N-1 components with a value of 0.0. This action corresponds to "after determining a change point, immediately reset the detector to its initial state to prepare for the next round of detection" in the context of sequential change point detection. The reset is necessary because of the observation that the cumulative deviation variable in the Page-Hinkley test continues to monotonically accumulate after a change point occurs. If the reset is not performed after triggering a decision, the cumulative deviation variable will continue to carry the trace of the already processed anomaly, repeatedly triggering decisions in the following observation times, causing the same anomaly to be repeatedly output. The essence of the reset is "information consumption complete, counter reset to zero"—since the anomaly evidence corresponding to this change point has already been packaged and delivered downstream, continuing to retain its cumulative trace will only introduce noise without adding any new value.

[0094] The reset scope covers all N-1 components, not just the one or two that triggered the decision. This is because while the state variables of the components are mathematically independent, they physically originate from the same set of network traffic samples. The impact of any structural anomaly on network traffic is not limited to a single attribute dimension but leaves traces simultaneously in multiple dimensions, albeit with varying magnitudes. If only the component that triggered the decision is reset, retaining the accumulated state of the remaining components, the accumulated deviations of those components that, while not reaching the trigger threshold, have significantly deviated from the mean will continue to persist after the anomaly. In the following observation times, these deviations may interact with new samples, generating secondary triggers unrelated to the new samples, thus contaminating subsequent decisions. A complete reset eliminates this residual interference, allowing the next round of detection to begin afresh in a truly clean state space.

[0095] The fourth task is to advance the observation time pointer. This involves adjusting the time cursor within the data processing unit. Advance to The coordinated registration sliding HyperLogLog estimator group from step S1, all state variables from step S2, and the periodic buffer used in this step are all synchronously put into the waiting state for the next observation period. It is important to note that advancing the observation time does not mean that the circular queue of the estimator group's sub-window register group in step S1 is cleared—the estimator group is not touched during the state reset transaction. The contents of the registers maintained internally for the most recent K sliding periods continue to advance naturally according to the sliding rhythm specified in step S1. The sub-window register group corresponding to this observation time is removed from the queue tail, cleared, and then placed back at the head of the queue to serve as the new latest sub-window register group, regardless of whether a change point is determined. This is because the estimator group itself only undertakes the function of "objectively counting the number of duplicate elements." Whether the statistical result constitutes an anomaly is determined by steps S2 and S3. After an anomaly occurs, the estimator group can continue operating smoothly without any special action.

[0096] refer to Figure 3 The horizontal axis represents the observation time in seconds, labeled with Arabic numerals. to The interval, with a main scale interval of 10 seconds, corresponds to the sliding cycle length of 1 second and the sliding window length in this embodiment. The configuration matches; the vertical axis represents the ratio component values, using dimensionless numerical labels, with a coordinate range of 0.0 to 2.0 and a major scale interval of 0.2. Four curves are plotted in the figure, corresponding to the four components of the cardinality projection vector. , , , Different line types are used to distinguish them. Use solid lines with circular markers. Use dashed lines with square markings. Use dashed lines with triangle markers. The graph uses dotted lines with diamond-shaped markers, displaying four markers every five sampling points to balance clarity and simplicity. A legend is located in the upper right corner of the curve, with the same line type and markers corresponding to the component names. The entire graph at the observation time... to This section is shaded in light gray and marked in the center with a white border and the label "Structural Anomaly Period," visually marking periods of abrupt changes in network traffic structure. The overall shape of the curve shows that... During the steady-state period, the four curves fluctuate randomly within a small range around their respective steady-state baselines. Around 0.5, Around 1.2, Around 0.8, Around 0.3, the fluctuation range does not exceed This reflects the stability of the cardinal projection vector under normal conditions; after entering the abnormal period, It rose from 0.5 to around 0.82 within 3 to 4 observation periods. The value rose from 0.8 to around 1.25, showing an "upward drift" pattern, which corresponds to the cardinality convergence behavior in the network flow attribute dimension. It decreased from 1.2 to approximately 0.92. The values ​​dropped from 0.3 to around 0.12, exhibiting a "drifting" pattern, corresponding to cardinality divergence in the network flow attribute dimension. It's noteworthy that all four components simultaneously deviated significantly, but the directions of deviation were not consistent—some components drifted upwards, while others drifted downwards. This "simultaneous, cross-component, bidirectional" characteristic is a typical manifestation of the overall rearrangement of network traffic structure, and also the physical basis for the subsequent bidirectional decoupling Page-Hinkley test to identify structural anomalies with high confidence through the bidirectional simultaneous triggering condition. During the recovery period, all four curves regressed to their original steady-state baselines. Figure 3 This intuitively illustrates that the input signal processed by this method has the characteristics of "small amplitude during the stationary period and joint deviation during the changing period", providing a detection sequence with a significant signal-to-noise ratio for the evolution of the cumulative deviation variable and the benchmark variable in the subsequent S2 step.

[0097] The following numerical evolution over a series of consecutive observation times illustrates the entire process of step S3, which follows the sample sequence given in section S2: a certain component takes values ​​of 0.50, 0.52, 0.48, 0.51, 0.80, 0.82, and 0.50 at 7 consecutive observation times. Assume that the other 3 components exhibit the following value sequences within the same time period: 0.30, 0.30, 0.32, 0.30, 0.05, 0.04, 0.31; 1.20, 1.18, 1.22, 1.19; 2.00, 2.05, 1.18; 0.80, 0.78, 0.82, 0.79, 0.40, 0.38, and 0.81. After independently maintaining six state variables for each of the four components and updating them according to step S2, the drift components obtained at observation time 5 are approximately as follows: The first component has an expansion direction drift of approximately 0.25 and a contraction direction drift of approximately 0; the second component has an expansion direction drift of approximately 0 and a contraction direction drift of approximately 0.24; the third component has an expansion direction drift of approximately 0.78 and a contraction direction drift of approximately 0; and the fourth component has an expansion direction drift of approximately 0 and a contraction direction drift of approximately 0.38. The aggregation yields the expansion direction drift at this observation time. , amount of drift in contraction direction . and preset threshold In comparison, , If the sum of the values ​​is true, a network traffic change point is determined to have occurred at observation time 5. The data processing device then marks the sliding period corresponding to observation time 5 as an abnormal traffic period, delivers all network flow records cached by the network traffic acquisition module within the last 60 sliding periods to the downstream in batches, resets the six state variables of each of the four components to 0.0, and advances the time cursor to observation time 6.

[0098] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A method for identifying abnormal network traffic based on online change point detection, executed by a data processing device, characterized in that: include: S1. Deploy a collaborative registration sliding HyperLogLog estimator group consisting of N sliding HyperLogLog estimators, where N is a positive integer greater than or equal to 3. The N sliding HyperLogLog estimators correspond to one combination of network flow attributes and are arranged in a preset fixed sequence number. Divide the observation time according to the preset sliding period length. At each observation time, the N sliding HyperLogLog estimators output a cardinality estimate. Take the quotient of two adjacent cardinality estimates along the fixed sequence number from small to large. The resulting N-1 ratio components constitute the cardinality projection vector for this observation time. S2. For each component of the base projection vector, maintain independent state variables including the expansion cumulative deviation variable, expansion reference variable, contraction cumulative deviation variable and contraction reference variable, and perform a two-way decoupling Page-Hinkley test. The expansion reference variable and contraction reference variable take the historical minimum values ​​of the expansion cumulative deviation variable and contraction cumulative deviation variable respectively, and obtain the expansion direction drift and contraction direction drift at the current observation time. S3. If the expansion direction drift and contraction direction drift are greater than the preset expansion direction trigger threshold and contraction direction trigger threshold respectively, it is determined that a network traffic change point has occurred at this observation time, and the network flow record within the sliding period corresponding to this observation time is output as an abnormal candidate flow.

2. The method as described in claim 1, characterized in that, The network flow attribute combination consists of one or more fields selected from five network flow fields: source IP address, destination IP address, source port number, destination port number, and transport layer protocol number. All sliding HyperLogLog estimators in the collaborative registration sliding HyperLogLog estimator group are assigned fixed sequence numbers in a predefined order, and the value of the fixed sequence number is a continuous integer from 1 to N.

3. The method as described in claim 2, characterized in that, The fixed arrangement number is assigned in ascending order of the number of network flow fields contained in the network flow attribute combination corresponding to each sliding HyperLogLog estimator. There is a field inclusion relationship between the network flow attribute combinations corresponding to two adjacent sliding HyperLogLog estimators with fixed arrangement numbers. The field inclusion relationship is as follows: the network flow attribute combination corresponding to the one with the larger fixed arrangement number completely includes all the fields of the network flow attribute combination corresponding to the one with the smaller fixed arrangement number, and adds one new network flow field to the aforementioned field composition.

4. The method as described in claim 3, characterized in that, In step S1, a sliding window length K, which is the number of sliding periods, is preset. K is a preset integer greater than or equal to 1. The observation time is divided into a continuous sequence of observation times according to the sliding period length. The time interval between two adjacent observation times is equal to the sliding period length. The cardinality estimate output by each sliding HyperLogLog estimator at the current observation time is the estimate of the number of deduplicated elements corresponding to all input items written to the corresponding sliding HyperLogLog estimator in the most recent K sliding periods up to the current observation time.

5. The method as described in claim 4, characterized in that, In step S1, within the sliding period corresponding to each observation time, the data processing device captures all network flow records arriving within the sliding period corresponding to the current observation time through the network traffic acquisition module. For each captured network flow record, the following processing is performed on each sliding HyperLogLog estimator in the collaborative registration sliding HyperLogLog estimator group: extract the values ​​that correspond one-to-one with all network flow fields contained in the network flow attribute combination corresponding to the corresponding sliding HyperLogLog estimator from the captured network flow record, concatenate them into an attribute value string according to the internal preset order of the corresponding network flow attribute combination, and write the attribute value string as an input item into the corresponding sliding HyperLogLog estimator.

6. The method as described in claim 5, characterized in that, In step S1, after the coordinated registration of the sliding HyperLogLog estimator group completes the cardinality estimate output at each observation time, the cardinality estimates output by two adjacent sliding HyperLogLog estimators at the current observation time are grouped into an ordered pair according to the fixed arrangement number in ascending order. For each ordered pair, the cardinality estimate output by the one with the smaller fixed arrangement number is used as the numerator, and the cardinality estimate output by the one with the larger fixed arrangement number is used as the initial denominator. The initial denominator is compared with a preset minimum positive number, and the larger of the two is used as the final denominator for this quotient calculation. The preset minimum positive number is a pre-set constant greater than 0. The numerator is divided by the final denominator to obtain a ratio component. The resulting N-1 ratio components are arranged in ascending order according to the fixed arrangement number to form the cardinality projection vector at the current observation time.

7. The method as described in claim 6, characterized in that, In step S2, the state variables also include one mean variable and one count variable. The state variables consist of six variables: one mean variable, one count variable, one expansion cumulative deviation variable, one expansion reference variable, one contraction cumulative deviation variable, and one contraction reference variable. Before executing step S2 at the first observation time, the initial storage values ​​of the mean variable, count variable, expansion cumulative deviation variable, expansion reference variable, contraction cumulative deviation variable, and contraction reference variable corresponding to all N-1 components of the cardinality projection vector are set to 0.

8. The method as described in claim 7, characterized in that, In step S2, after obtaining the cardinality projection vector at each observation time by completing step S1, the mean update process is performed on all N-1 components of the cardinality projection vector in ascending order of fixed sequence numbers: For the component currently being processed, the value of the current component in the cardinality projection vector at this observation time is recorded as the current input value; the current stored value of the counter variable corresponding to the current component is incremented by 1, and the result after incrementing is written back to the counter variable corresponding to the current component, and the stored value after writing back is recorded as the updated counter value; the difference between the current input value and the current stored value of the mean variable corresponding to the current component is calculated, the difference is divided by the updated counter value to obtain a quotient value, the quotient value is added to the current stored value of the mean variable corresponding to the current component, and the result after adding is written back to the mean variable corresponding to the current component; the current stored value of the mean variable corresponding to the current component after writing back is recorded as the mean reference.

9. The method as described in claim 8, characterized in that, In step S2, for the component currently being processed, after completing the mean update process: the difference obtained by subtracting the mean reference from the current input value is accumulated into the expansion cumulative deviation variable corresponding to the current component, and the accumulated result is written back to the expansion cumulative deviation variable corresponding to the current component. The stored value of the expansion cumulative deviation variable corresponding to the current component after this write-back is compared with the current stored value of the expansion base variable corresponding to the current component, and the smaller of the two values ​​is taken as the update value of the expansion base variable corresponding to the current component. The update value is written back to the expansion base variable corresponding to the current component. The difference between the mean reference and the current input value is accumulated into the contraction cumulative deviation variable corresponding to the current component. The accumulated result is written back to the contraction cumulative deviation variable corresponding to the current component. The stored value of the contraction cumulative deviation variable corresponding to the current component after this write-back is compared with the current stored value of the contraction benchmark variable corresponding to the current component. The smaller of the two values ​​is taken as the updated value of the contraction benchmark variable corresponding to the current component. The updated value is written back to the contraction benchmark variable corresponding to the current component. After processing all N-1 components of the cardinal projection vector at the current observation time, for each component of the cardinal projection vector, the stored value of its corresponding expansion cumulative deviation variable after processing at the current observation time is subtracted from the stored value of its corresponding expansion reference variable after processing at the current observation time, and the difference is taken as its expansion direction drift component at the current observation time; the stored value of its corresponding contraction cumulative deviation variable after processing at the current observation time is subtracted from the stored value of its corresponding contraction reference variable after processing at the current observation time, and the difference is taken as its contraction direction drift component at the current observation time; the largest value among the expansion direction drift components corresponding to all N-1 components of the cardinal projection vector is taken as the expansion direction drift amount at the current observation time, and the largest value among the contraction direction drift components corresponding to all N-1 components of the cardinal projection vector is taken as the contraction direction drift amount at the current observation time.

10. The method as described in claim 9, characterized in that, In step S3, expansion direction trigger thresholds and contraction direction trigger thresholds are preset in the data processing device. Both expansion direction trigger thresholds and contraction direction trigger thresholds are preset constants greater than 0. Under the condition that the expansion direction drift is greater than the expansion direction trigger threshold and the contraction direction drift is greater than the contraction direction trigger threshold at the current observation time, the sliding period corresponding to the current observation time is marked as an abnormal flow period. All network flow records captured by the network flow acquisition module within the most recent K sliding periods corresponding to the current observation time are output as abnormal candidate flow to the downstream security handling module of the data processing device. At the same time, the stored values ​​of the mean variable, count variable, expansion cumulative deviation variable, expansion reference variable, contraction cumulative deviation variable, and contraction reference variable corresponding to all N-1 components of the cardinality projection vector are all reset to 0. After completing all the processing of steps S1 to S3 at the current observation time, the data processing device enters the next observation time and continues to execute steps S1 to S3 in sequence.

Images