A system for detecting stable high frequency flow in a data stream

Abstract

The application relates to the technical field of network measurement, and provides a system for detecting stable high-frequency flow in a data flow, which is characterized by comprising a manifold projection module, a dual-mode counting module and a stability fusioner; the manifold projection module is used for receiving a network flow and extracting network flow features, mapping the network flow features to a preset stability reference coordinate system, and recording a quantitative stability value; the dual-mode counting module is used for recording a transient counting value and a cumulative stability value; and the stability fusioner is used for comprehensively quantifying the stability value, the transient counting value and the cumulative stability value, and jointly judging the network flow. The three-level processing architecture which fuses manifold stability representation and a dual-mode counting mechanism can efficiently detect stable high-frequency flow in a data flow.

Description

Technical Field

[0001] This invention belongs to the field of network measurement technology, and in particular relates to a system for detecting stable high-frequency streams in data streams. Background Technology

[0002] With the development of network technology, especially the gradual application of 5G technology, network traffic has surged, making network behavior increasingly complex. How to manage the network efficiently has become a fundamental issue, and network measurement is an effective means to solve this problem. Network measurement refers to the capture, statistics, analysis, and processing of all traffic passing through network devices to obtain basic characteristics of network behavior. This characteristic information is an important prerequisite for applying network performance analysis, network configuration optimization, and network anomaly detection. Furthermore, as a special type of streaming data, network traffic measurement techniques can also be applied to other fields such as databases, data mining, and information security. Stable high-frequency flow detection is used to detect continuously occurring, numerous, and relatively stable traffic within a data stream.

[0003] However, existing stable large flow detection methods suffer from technical defects such as low efficiency of traditional stability detection, insufficient accuracy of high-frequency flow identification, unreasonable resource allocation, and limited mathematical representation capabilities. These defects include the need to store multi-window historical data, resulting in high memory access overhead, inability to distinguish between stable large flows and burst flows, application of the same processing strategy to large and small flows, and difficulty in capturing the nonlinear stability characteristics of network flows. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the present invention proposes a system for detecting stable high-frequency streams in data streams. The technical solution designed by the present invention includes:

[0005] Manifold projection module, dual-mode counting module, stability fusion unit, and window switching module;

[0006] The manifold projection module is used to receive network streams and extract network stream features, map the network stream features to a preset stability reference coordinate system, and record the quantized stability value.

[0007] The dual-mode counting module is used to record transient count values ​​and cumulative stable values;

[0008] The stability fusion unit is used to combine the quantized stability value, transient count value and cumulative stability value to make a joint decision on the network flow.

[0009] Preferably, the network flow characteristics include:

[0010] Packet count, duration, arrival interval jitter, and quintuple information entropy.

[0011] Preferably, the recording of quantization stability values ​​includes:

[0012] Four types of expert knowledge anchors are preset, including high-stability large flow, medium-stability flow, fluctuating flow and unstable flow, forming a stability reference coordinate system. By calculating the similarity distance between network flow features and each anchor, a multi-dimensional stability representation vector is generated. The multi-dimensional stability representation vector is weighted and summed, and a quantitative stability value in the range of 0-255 is output and recorded.

[0013] Preferably, the dual-mode counting module is used to record transient count values ​​and cumulative stable values, including:

[0014] The dual-mode counting module includes a transient counter and a cumulative stabilizer;

[0015] The transient counter is used to record the flow rate changes within the current time window as a transient count value, and is automatically reset to zero when the window ends.

[0016] The cumulative stabilizer is used to record the flow value of the stability weighted transformation across time windows as a cumulative stable value, thereby achieving long-term stability memory.

[0017] Preferably, the dual-mode counting module is used to record transient count values ​​and cumulative stable values, and further includes:

[0018] Stream fingerprints are generated for matching counting buckets. Each counting bucket has three states: empty bucket, fingerprint matching, and fingerprint conflict. An empty bucket is characterized by an initial stability value of 50, an initial count of 1, and the bucket being marked as occupied. Fingerprint matching involves incrementing the transient counter within the bucket and checking the transient count. If the count exceeds 240, a stability-aware conversion is performed, adding the conversion value to the accumulator, and the transient counter is set to 0. Fingerprint conflict involves checking the transient counter. If the transient count is below 10, high-frequency streams are prioritized and low-frequency streams are discarded. If the transient count is above 128, the high-frequency stream priority principle is triggered. Based on the operation results of the counting bucket states, the transient count and the accumulator stability value are updated.

[0019] Preferably, the stability-aware conversion formula is as follows:

[0020]

[0021] Preferably, the stability fusion unit is used to comprehensively quantize the stability value, transient count value, and cumulative stability value to make a joint decision on the network flow, including:

[0022] When the joint decision conditions are met simultaneously, the network flow is determined to be a stable high-frequency flow; otherwise, it is discarded.

[0023] The joint decision conditions include the cumulative stable value exceeding the absolute flow threshold, the quantized stable value exceeding the stability threshold, and the ratio of the cumulative stabilizer to the transient counter exceeding the steady-state-transient ratio threshold.

[0024] Preferably, the system further includes:

[0025] Window switching module;

[0026] The window switching module is used for transient counter reset, stability-guided decay, and low-value bucket cleanup.

[0027] The transient counter reset includes clearing all transient counters to zero at the beginning of each time window and re-recording the flow changes of the current window;

[0028] The stability-guided attenuation includes dynamically adjusting the attenuation factor to guide the stability attenuation of the network flow.

[0029] The low-value bucket cleanup includes periodically cleaning up counting buckets that have not been updated for a long time or have low quantization stability values, thereby freeing up memory space and optimizing system performance.

[0030] Preferably, the dynamically adjusted attenuation factor guides the attenuation of network flow to improve stability, as shown in the following formula:

[0031]

[0032] Beneficial effects:

[0033] 1. This application accurately captures the nonlinear stability characteristics of network flow through manifold stability projection. The triple decision mechanism comprehensively considers long-term stability and short-term fluctuations, significantly reducing the false judgment rate. The stability-aware conversion ensures efficient recording of truly stable large flows.

[0034] 2. This application eliminates the need for multi-window historical storage through a dual-mode counting design, significantly reducing processing latency;

[0035] 3. This application achieves intelligent resource recycling through stability-guided decay and optimizes memory space utilization through adaptive counting conversion. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of a preferred embodiment of the present invention. Detailed Implementation

[0037] The embodiments of the present invention will be described in detail below. The embodiments described below are implemented based on the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the embodiments described below.

[0038] This invention designs a system for detecting stable high-frequency streams in a data stream. The technical solution includes the following steps, such as... Figure 1 As shown, it specifically includes:

[0039] Manifold projection module, dual-mode counting module, stability fusion unit, and window switching module;

[0040] The manifold projection module is used to receive network streams and extract network stream features, map the network stream features to a preset stability reference coordinate system, and record the quantized stability values.

[0041] The dual-mode counting module is used to record transient count values ​​and cumulative stable values;

[0042] The stability fusion unit is used to combine the quantized stability value, transient count value and cumulative stability value to make a joint decision on network flow.

[0043] Preferably, network flow characteristics include:

[0044] Packet count, duration, arrival interval jitter, and quintuple information entropy.

[0045] Preferably, recording quantization stability values ​​includes:

[0046] Four types of expert knowledge anchors are preset, including high-stability large flow, medium-stability flow, fluctuating flow and unstable flow, forming a stability reference coordinate system. By calculating the similarity distance between network flow features and each anchor, a multi-dimensional stability representation vector is generated. The multi-dimensional stability representation vector is weighted and summed, and a quantitative stability value in the range of 0-255 is output and recorded.

[0047] Preferably, the dual-mode counting module is used to record transient count values ​​and cumulative stable values, including:

[0048] The dual-mode counting module includes a transient counter and an accumulation stabilizer;

[0049] The transient counter is used to record the flow rate changes within the current time window as a transient count value, and is automatically reset to zero when the window ends;

[0050] The cumulative stabilizer is used to record the flow value of the stability weighted transformation across time windows as a cumulative stable value, thereby achieving long-term stability memory.

[0051] Preferably, the dual-mode counting module is used to record transient count values ​​and cumulative stable values, and further includes:

[0052] Stream fingerprints are generated for matching counting buckets. Each counting bucket has three states: empty bucket, fingerprint matching, and fingerprint conflict. An empty bucket is characterized by an initial stability value of 50, an initial count of 1, and the bucket being marked as occupied. Fingerprint matching involves incrementing the transient counter within the bucket and checking the transient count. If the count exceeds 240, a stability-aware conversion is performed, adding the conversion value to the accumulator, and the transient counter is set to 0. Fingerprint conflict involves checking the transient counter. If the transient count is below 10, high-frequency streams are prioritized and low-frequency streams are discarded. If the transient count is above 128, the high-frequency stream priority principle is triggered. Based on the operation results of the counting bucket states, the transient count and the accumulator stability value are updated.

[0053] Preferably, the stability-perceived conversion formula is as follows:

[0054]

[0055] Specifically, for the dual-mode counting module, each counting bucket contains four fields: a 1-byte stream fingerprint, an 8-bit transient counter, a 16-bit cumulative stabilizer, and an 8-bit quantization stability value; and the data stream characteristics are hashed to generate a 1-byte stream fingerprint for fast matching of counting buckets.

[0056] Preferably, the stability fusion unit is used to comprehensively quantize the stability value, transient count value, and cumulative stability value to make a joint decision on the network flow, including:

[0057] When the joint decision conditions are met simultaneously, the network flow is determined to be a stable high-frequency flow; otherwise, it is discarded.

[0058] The joint decision conditions include the cumulative stable value exceeding the absolute flow threshold, the quantized stable value exceeding the stability threshold, and the ratio of the cumulative stabilizer to the transient counter exceeding the steady-state-transient ratio threshold.

[0059] Preferably, the system further includes:

[0060] Window switching module;

[0061] The window switching module is used for transient counter reset, stability-guided decay, and low-value bucket cleanup;

[0062] Transient counter reset includes clearing all transient counters to zero at the beginning of each time window and re-recording the flow changes for the current window;

[0063] Stability-guided attenuation involves dynamically adjusting the attenuation factor to guide the stability attenuation of network flows.

[0064] Low-value bucket cleanup includes periodically cleaning up counting buckets that have not been updated for a long time or have low quantization stability values, freeing up memory space and optimizing system performance.

[0065] Preferably, the attenuation factor is dynamically adjusted to guide the stability attenuation of the network flow, as shown in the following formula:

[0066]

[0067] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A system for detecting stable high-frequency streams in a data stream, characterized in that, include: Manifold projection module, dual-mode counting module, and stability fusion unit; The manifold projection module is used to receive network streams and extract network stream features, map the network stream features to a preset stability reference coordinate system, and record the quantized stability value. The dual-mode counting module is used to record transient count values ​​and cumulative stable values; The stability fusion unit is used to combine the quantized stability value, transient count value and cumulative stability value to make a joint decision on the network flow. The dual-mode counting module is used to record transient count values ​​and cumulative stable values, including: The dual-mode counting module includes a transient counter and a cumulative stabilizer; The transient counter is used to record the flow rate changes within the current time window as a transient count value, and is automatically reset to zero when the window ends. The cumulative stabilizer is used to record the flow value of the stability weighted transformation across time windows as the cumulative stable value, thereby realizing long-term stability memory; The stability fusion unit is used to comprehensively quantize stability values, transient count values, and cumulative stability values ​​to make joint decisions on network flows, including: When the joint decision conditions are met simultaneously, the network flow is determined to be a stable high-frequency flow; otherwise, it is discarded. The joint decision conditions include the cumulative stable value exceeding the absolute flow threshold, the quantized stable value exceeding the stability threshold, and the ratio of the cumulative stabilizer to the transient counter exceeding the steady-state-transient ratio threshold.

2. The system for detecting stable high-frequency streams in a data stream according to claim 1, characterized in that, The network flow characteristics include: Packet count, duration, arrival interval jitter, and quintuple information entropy.

3. The system for detecting stable high-frequency streams in a data stream according to claim 1, characterized in that, The recorded quantization stability values ​​include: Four types of expert knowledge anchors are preset, including high-stability large flow, medium-stability flow, fluctuating flow and unstable flow, forming a stability reference coordinate system. By calculating the similarity distance between network flow features and each anchor, a multi-dimensional stability representation vector is generated. The multi-dimensional stability representation vector is weighted and summed, and a quantitative stability value in the range of 0-255 is output and recorded.

4. The system for detecting stable high-frequency streams in a data stream according to claim 1, characterized in that, The dual-mode counting module is used to record transient count values ​​and cumulative stable values, and also includes: Stream fingerprints are generated for matching counting buckets. Each counting bucket has three states: empty bucket, fingerprint matching, and fingerprint conflict. An empty bucket is characterized by an initial stability value of 50, an initial count of 1, and the bucket being marked as occupied. Fingerprint matching involves incrementing the transient counter within the bucket and checking the transient count. If the count exceeds 240, a stability-aware conversion is performed, adding the conversion value to the accumulator, and the transient counter is set to 0. Fingerprint conflict involves checking the transient counter. If the transient count is below 10, high-frequency streams are prioritized and low-frequency streams are discarded. If the transient count is above 128, the high-frequency stream priority principle is triggered. Based on the operation results of the counting bucket states, the transient count and the accumulator stability value are updated.

5. A system for detecting stable high-frequency streams in a data stream according to claim 4, characterized in that, The stability-aware conversion formula is as follows: 。 6. A system for detecting stable high-frequency streams in a data stream according to claim 1, characterized in that, The system also includes: Window switching module; The window switching module is used for transient counter reset, stability-guided decay, and low-value bucket cleanup. The transient counter reset includes clearing all transient counters to zero at the beginning of each time window and re-recording the flow changes of the current window; The stability-guided attenuation includes dynamically adjusting the attenuation factor to guide the stability attenuation of the network flow. The low-value bucket cleanup includes periodically cleaning up counting buckets that have not been updated for a long time or have low quantization stability values, thereby freeing up memory space and optimizing system performance.

7. A system for detecting stable high-frequency streams in a data stream according to claim 6, characterized in that, The dynamically adjusted attenuation factor guides the attenuation of network flow to ensure stability, and the formula is as follows: 。

Images