System for detecting stable high-frequency flow in data flow

Abstract

The invention relates to the technical field of network measurement, and provides a system for detecting a stable high-frequency stream in a data stream, which is characterized by comprising a manifold projection module, a dual-mode counting module and a stability fusion device, the manifold projection module is used for receiving a network flow, extracting network flow characteristics, mapping the network flow characteristics to a preset stability reference coordinate system, and recording a quantitative stability value; the dual-mode counting module is used for recording a transient count value and an accumulated stable value; and the stability fusion device is used for comprehensively quantifying the stability value, the transient count value and the accumulated stability value and carrying out joint judgment on the network flow. According to the invention, through a three-stage processing architecture fusing manifold stability representation and a dual-mode counting mechanism, stable high-frequency streams in data streams are efficiently detected.

Description

Technical Field

[0001] The present invention belongs to the technical field of network measurement, and in particular relates to a system for detecting stable high-frequency flows in data flows. Background Art

[0002] With the development of network technology, especially the gradual implementation of 5G, network traffic has increased dramatically, making network behavior increasingly complex. Efficient network management has become a fundamental issue, and network measurement is an effective means of addressing this problem. Network measurement involves capturing, counting, analyzing, and processing all traffic passing through network devices to obtain basic characteristics of network behavior. This characteristic information is an important prerequisite for network performance analysis, network configuration optimization, and network anomaly detection. Furthermore, as a special type of streaming data, network traffic measurement technology can also be applied to other fields such as databases, data mining, and information security. Stable high-frequency flow detection is used to detect continuous, large, and relatively stable traffic in a data stream.

[0003] However, existing stable large flow detection has technical defects such as low efficiency of traditional stability detection, insufficient accuracy of high-frequency flow identification, unreasonable resource allocation and limited mathematical representation capabilities. These defects include the need to store multi-window historical data in existing variance-based methods, resulting in high memory access overhead, inability to distinguish between stable large flows and burst flows, the use of the same processing strategy for large flows and small flows, and difficulty in capturing the nonlinear stability characteristics of network flows. Summary of the Invention

[0004] In view of the above-mentioned defects of the prior art, the present invention proposes a system for detecting stable high-frequency flow in a data stream. The technical solution designed by the present invention includes: Manifold projection module, dual-mode counting module, stability fuser and window switching module; The manifold projection module is used to receive the network flow and extract the network flow features, map the network flow features to a preset stability reference coordinate system, and record the quantitative stability value; The dual-mode counting module is used to record the transient counting value and the accumulated stable value; The stability fusion device is used for comprehensively quantifying the stability value, the transient count value and the cumulative stability value, and making a joint judgment on the network flow.

[0005] Preferably, the network flow characteristics include: Packet number, duration, inter-arrival jitter, and quintuple information entropy.

[0006] Preferably, the recording of the quantitative stability value comprises: Four types of expert knowledge anchor points are preset, including highly stable large flow, moderately stable flow, fluctuating flow and unstable flow, to form a stability reference coordinate system. By calculating the similarity distance between the network flow characteristics and each anchor point, a multidimensional stability representation vector is generated. The multidimensional stability representation vector is weighted and summed, and the quantitative stability value in the range of 0-255 is output and recorded.

[0007] Preferably, the dual-mode counting module is used to record the transient counting value and the cumulative stable value, including: The dual-mode counting module includes a transient counter and an accumulation stabilizer; The transient counter is used to record the flow rate change within the current time window as a transient count value, and is automatically reset to zero when the window ends; The cumulative stabilizer is used to record the flow value of the stability weighted conversion across the time window as the cumulative stability value, so as to realize long-term stability memory.

[0008] Preferably, the dual-mode counting module is used to record the transient counting value and the cumulative stable value, and further includes: Generate a flow fingerprint for matching the counting bucket. Each counting bucket state is divided into three cases, including empty bucket, fingerprint match, and fingerprint conflict. The empty bucket case sets the initial stability value to 50, the original count to 1, and marks the bucket state as occupied. The fingerprint match adds 1 to the transient counter in the bucket and checks the transient count value. If it exceeds 240, a stability-aware conversion is performed, the conversion amount is added to the cumulative stabilizer, and the transient counter is set to 0. The fingerprint conflict checks the transient counter. If the transient count value is less than 10, high-frequency flows are retained first and low-frequency flows are discarded. If the transient count value is greater than 128, the high-frequency flow priority principle is triggered. Based on the operation results of the counting bucket state, the transient count value and the cumulative stability value are updated.

[0009] Preferably, the stability perception conversion is as follows:

[0010] Preferably, the stability fusion device is used to comprehensively quantify the stability value, the transient count value and the cumulative stability value to make a joint decision on the network flow, including: When the joint decision conditions are met at the same time, the network flow is determined to be a stable high-frequency flow, otherwise it is discarded; The joint decision conditions include the cumulative stability value exceeding the absolute flow threshold, the quantitative stability value exceeding the stability threshold, and the ratio of the cumulative stabilizer to the transient counter exceeding the steady-state transient ratio threshold.

[0011] Preferably, the system further comprises: Window switching module; The window switching module is used for transient counter reset, stability guided decay and low value bucket cleaning; The transient counter reset includes clearing all transient counters at the beginning of each time window and re-recording the flow change of the current window; The stability-guided attenuation includes dynamically adjusting the attenuation factor to perform stability-guided attenuation on the network flow; The low-value bucket cleaning includes regularly cleaning count buckets that have not been updated for a long time or have low quantitative stability values, thereby freeing up memory space and optimizing system performance.

[0012] Preferably, the dynamically adjusted attenuation factor performs stability-guided attenuation on the network flow, and the formula is as follows:

[0013] Beneficial effects: 1. This application accurately captures the nonlinear stability characteristics of network flows through manifold stability projection. The triple judgment mechanism comprehensively considers long-term stability and short-term fluctuations, significantly reducing the false positive rate. Stability perception conversion ensures efficient recording of truly stable large flows. 2. This application eliminates the need for multi-window historical storage through a dual-mode counting design, significantly reducing processing delays; 3. This application realizes intelligent resource recycling through stability-guided decay, and optimizes memory space utilization through adaptive counting conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a structural schematic diagram of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0015] The embodiments of the present invention are described in detail below. The following embodiments are implemented based on the technical solutions of the present invention, and provide detailed implementation methods and specific operating procedures. However, the protection scope of the present invention is not limited to the following embodiments.

[0016] The present invention designs a system for detecting stable high-frequency flow in data flow, and the technical solution includes the following steps: Figure 1 As shown, specifically including: Manifold projection module, dual-mode counting module, stability fuser and window switching module; The manifold projection module is used to receive network flow and extract network flow features, map the network flow features to a preset stability reference coordinate system, and record the quantitative stability value; The dual-mode counting module is used to record the transient counting value and the accumulated stable value; The stability fusion device is used to comprehensively quantify the stability value, transient count value and cumulative stability value to make a joint judgment on the network flow.

[0017] Preferably, the network flow characteristics include: Packet number, duration, inter-arrival jitter, and quintuple information entropy.

[0018] Preferably, the quantitative stability value is recorded, including: Four types of expert knowledge anchor points are preset, including highly stable large flow, moderately stable flow, fluctuating flow and unstable flow, to form a stability reference coordinate system. By calculating the similarity distance between the network flow characteristics and each anchor point, a multidimensional stability representation vector is generated. The multidimensional stability representation vector is weighted and summed, and the quantitative stability value in the range of 0-255 is output and recorded.

[0019] Preferably, the dual-mode counting module is used to record the transient counting value and the cumulative stable value, including: The dual-mode counting module includes a transient counter and an accumulation stabilizer; The transient counter is used to record the flow rate changes within the current time window as the transient count value, and is automatically reset to zero when the window ends; The cumulative stabilizer is used to record the flow value of the stability-weighted conversion across time windows as the cumulative stability value to achieve long-term stability memory.

[0020] Preferably, the dual-mode counting module is used to record the transient counting value and the cumulative stable value, and further includes: Generate a flow fingerprint for matching the counting bucket. Each counting bucket state is divided into three cases, including empty bucket, fingerprint match, and fingerprint conflict. The empty bucket case sets the initial stability value to 50, the original count to 1, and marks the bucket state as occupied. The fingerprint match adds 1 to the transient counter in the bucket and checks the transient count value. If it exceeds 240, a stability-aware conversion is performed, the conversion amount is added to the cumulative stabilizer, and the transient counter is set to 0. The fingerprint conflict checks the transient counter. If the transient count value is less than 10, high-frequency flows are retained first and low-frequency flows are discarded. If the transient count value is greater than 128, the high-frequency flow priority principle is triggered. Based on the operation results of the counting bucket state, the transient count value and the cumulative stability value are updated.

[0021] Preferably, the stability-aware conversion formula is as follows:

[0022] Specifically, for the dual-mode counting module, each counting bucket contains four fields: a 1-byte stream fingerprint, an 8-bit transient counter, a 16-bit cumulative stabilizer, and an 8-bit quantized stability value; and the data stream features are hashed to generate a 1-byte stream fingerprint for fast matching of the counting bucket.

[0023] Preferably, the stability fusion device is used to comprehensively quantify the stability value, the transient count value and the cumulative stability value to make a joint judgment on the network flow, including: When the joint decision conditions are met at the same time, the network flow is determined to be a stable high-frequency flow, otherwise it is discarded; The joint decision conditions include the cumulative stability value exceeding the absolute flow threshold, the quantitative stability value exceeding the stability threshold, and the ratio of the cumulative stabilizer to the transient counter exceeding the steady-state transient ratio threshold.

[0024] Preferably, the system further comprises: Window switching module; Window switching module for transient counter reset, stability-guided decay, and low-value bucket cleanup; Transient counter reset includes clearing all transient counters at the beginning of each time window and re-recording the traffic changes in the current window; Stability-guided attenuation includes dynamically adjusting the attenuation factor to perform stability-guided attenuation on network flows; Low-value bucket cleaning includes regularly cleaning count buckets that have not been updated for a long time or have low quantitative stability values, freeing up memory space and optimizing system performance.

[0025] Preferably, the attenuation factor is dynamically adjusted to guide the attenuation of the network flow in a stable manner. The formula is as follows:

[0026] The above describes in detail the preferred embodiments of the present invention. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible by those skilled in the art without inventive effort. Therefore, any technical solution that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A system for detecting stable high-frequency flows in data streams, characterized in that: include: Manifold projection module, dual-mode counting module and stability fuser; The manifold projection module is used to receive the network flow and extract the network flow features, map the network flow features to a preset stability reference coordinate system, and record the quantitative stability value; The dual-mode counting module is used to record the transient counting value and the accumulated stable value; The stability fusion device is used for comprehensively quantifying the stability value, the transient count value and the cumulative stability value, and making a joint judgment on the network flow.

2. A system for detecting stable high-frequency flow in a data stream according to claim 1, characterized in that: The network flow characteristics include: Packet number, duration, inter-arrival jitter, and quintuple information entropy.

3. A system for detecting stable high-frequency flow in a data stream according to claim 1, characterized in that: The recorded quantitative stability value includes: Four types of expert knowledge anchor points are preset, including highly stable large flow, moderately stable flow, fluctuating flow and unstable flow, to form a stability reference coordinate system. By calculating the similarity distance between the network flow characteristics and each anchor point, a multidimensional stability representation vector is generated. The multidimensional stability representation vector is weighted and summed, and the quantitative stability value in the range of 0-255 is output and recorded.

4. A system for detecting stable high-frequency flow in a data stream according to claim 1, characterized in that: The dual-mode counting module is used to record the transient counting value and the accumulated stable value, including: The dual-mode counting module includes a transient counter and an accumulation stabilizer; The transient counter is used to record the flow rate change within the current time window as a transient count value, and is automatically reset to zero when the window ends; The cumulative stabilizer is used to record the flow value of the stability weighted conversion across the time window as the cumulative stability value, so as to realize long-term stability memory.

5. A system for detecting stable high-frequency flow in a data stream according to claim 4, characterized in that: The dual-mode counting module is used to record the transient counting value and the accumulated stable value, and also includes: Generate a flow fingerprint for matching the counting bucket. Each counting bucket state is divided into three cases, including empty bucket, fingerprint match, and fingerprint conflict. The empty bucket case sets the initial stability value to 50, the original count to 1, and marks the bucket state as occupied. The fingerprint match adds 1 to the transient counter in the bucket and checks the transient count value. If it exceeds 240, a stability-aware conversion is performed, the conversion amount is added to the cumulative stabilizer, and the transient counter is set to 0. The fingerprint conflict checks the transient counter. If the transient count value is less than 10, high-frequency flows are retained first and low-frequency flows are discarded. If the transient count value is greater than 128, the high-frequency flow priority principle is triggered. Based on the operation results of the counting bucket state, the transient count value and the cumulative stability value are updated.

6. A system for detecting stable high-frequency flow in a data stream according to claim 5, characterized in that: The stability perception conversion formula is as follows:

7. A system for detecting stable high-frequency flow in a data stream according to claim 1, characterized in that: The stability fusion device is used to comprehensively quantify the stability value, the transient count value and the cumulative stability value, and make a joint judgment on the network flow, including: When the joint decision conditions are met at the same time, the network flow is determined to be a stable high-frequency flow, otherwise it is discarded; The joint decision conditions include the cumulative stability value exceeding the absolute flow threshold, the quantitative stability value exceeding the stability threshold, and the ratio of the cumulative stabilizer to the transient counter exceeding the steady-state transient ratio threshold.

8. A system for detecting stable high-frequency flow in a data stream according to claim 1, characterized in that: The system further comprises: Window switching module; The window switching module is used for transient counter reset, stability guided decay and low value bucket cleaning; The transient counter reset includes clearing all transient counters at the beginning of each time window and re-recording the flow change of the current window; The stability-guided attenuation includes dynamically adjusting the attenuation factor to perform stability-guided attenuation on the network flow; The low-value bucket cleaning includes regularly cleaning count buckets that have not been updated for a long time or have low quantitative stability values, thereby freeing up memory space and optimizing system performance.

9. A system for detecting stable high-frequency flow in a data stream according to claim 8, characterized in that: The dynamic adjustment attenuation factor is used to guide the attenuation of network flow stability. The formula is as follows: 。

Images