Flow mirror image data retention and threat tracing integrated device with built-in threat intelligence

Abstract

The invention discloses a flow mirror image data retention and threat tracing integrated device with built-in threat intelligence, and belongs to the field of network security. Aiming at the problems of low storage efficiency, threat detection lagging and the like of a traditional traffic mirroring system, the device adopts a three-stage assembly line architecture and comprises a traffic processing module, a threat intelligence fusion module and a traceability visual interface module. And dynamic traffic screening, efficient storage and accurate traceability are realized through a threat intelligence weighted traffic selection function, a space-time joint compression storage model and a multi-modal knowledge graph traceability reasoning framework. Through testing, the storage efficiency is improved by 4.8 times, the threat detection rate reaches 96.4% or above, the average evidence obtaining time is shortened to 9.2-15.8 minutes, and the network security threat response capability is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of network security technology, specifically to an integrated device for traffic mirroring data retention and threat tracing with built-in threat intelligence, used to solve the problems of efficient traffic storage, real-time threat detection and automated tracing in high-speed network environments. Background Technology

[0002] With the development of new cyber threats such as APT attacks and zero-day vulnerabilities, traditional rule-based network security devices suffer from shortcomings such as detection lag and insufficient attribution capabilities. According to Gartner, in 2023, the average cyber threat response time for enterprises worldwide reached 287 hours, with 78% of that time spent on threat forensics and correlation analysis. Currently, traffic mirroring systems (SPAN) and threat intelligence platforms (TIP) operate independently, exhibiting the following technical limitations:

[0003] Full backups result in low storage efficiency (industry average effective threat data percentage η < 12%).

[0004] Threat intelligence matching latency is exponentially related to traffic volume (Δt = k·e) αV (α=0.78);

[0005] Tracing the source requires cross-system data association, and the complete evidence collection process requires an average of 17.3 manual interventions.

[0006] Full names of terms and abbreviations

[0007] • FPGA: Field-Programmable Gate Array

[0008] • BERT: Bidirectional Encoder Representations from Transformers

[0009] • DCT: Discrete Cosine Transform

[0010] • LSTM: Long Short-Term Memory Network

[0011] • CVSS: Common Vulnerability Scoring System

[0012] • CISA: Cybersecurity and Infrastructure Security Agency (CISA)

[0013] OSINT: Open Source Intelligence

[0014] • NIST: National Institute of Standards and Technology

[0015] IEEE: Institute of Electrical and Electronics Engineers Summary of the Invention

[0016] This invention provides an integrated device for traffic mirroring data retention and threat tracing with built-in threat intelligence, addressing the problems of low traffic storage efficiency (industry average η < 12%) and high threat intelligence matching latency (Δt = k·e) in existing technologies. αV This addresses the issues of excessive manual intervention in tracing the source (averaging 17.3 times). A three-tier pipeline architecture is constructed, integrating a threat intelligence-weighted traffic selection function. Spatiotemporal joint compressed storage model (X=Φ·S·Ψ) T It utilizes a multimodal knowledge graph-based source tracing and reasoning framework (P(A|E)) to achieve dynamic traffic filtering (efficiency η≥67%), a 4.8x improvement in storage efficiency, and automated source tracing (F1-score 0.92). Testing shows a threat detection rate exceeding 96.4%, with the average forensic time reduced to 9.2-15.8 minutes, meeting 100Gbps line-speed processing requirements and providing an efficient solution for network security threat response.

[0017] Technical solution

[0018] To address the issues of low storage efficiency, high detection latency, and reliance on manual intervention in traditional traffic mirroring and threat attribution technologies, this invention utilizes threat intelligence-driven dynamic traffic mirroring, spatiotemporal joint compressed storage, and multimodal knowledge graph inference to achieve efficient traffic data retention, real-time threat detection, and automated attribution. This reduces the average forensic time from 287 hours to 9.2-15.8 minutes, improving the efficiency and accuracy of cybersecurity threat response. The specific implementation process is as follows:

[0019] 1. Device Architecture

[0020] This device innovatively adopts a three-stage pipeline architecture. Figure 1This system achieves a closed-loop processing from traffic capture to threat attribution, effectively improving the detection and response capabilities for cybersecurity threats. Its main technical details are as follows:

[0021] To address the issues of low storage efficiency and poor threat relevance in traditional traffic mirroring systems, this device proposes a traffic selection function based on threat intelligence weighting:

[0022]

[0023] in:

[0024] TI i (t) represents the activity score of the i-th type of threat intelligence at time t, which is used to measure the activity level of this type of threat at the current time.

[0025] Parameter confirmation method:

[0026] (1) Confidence weight β of threat intelligence i

[0027] A two-tier decision-making mechanism is adopted to determine:

[0028] Expert Experience Layer: Based on dimensions such as CVE vulnerability score (CVSS) and the authority of intelligence sources (e.g., CISA / OSINT), a beta layer is established. i Initial value matrix;

[0029] Machine learning layer: Trains a historical threat data using an LSTM model to calculate β. i Dynamic correction coefficient (correction formula: β') i =β i *(1+α*ΔFPR), where ΔFPR is the rate of change of false alarm rate).

[0030] (2) Time decay factor λ

[0031] Based on the exponential moving average algorithm:

[0032]

[0033] Where γ is the forgetting coefficient (value 0.75), T half Dynamically determined by the threat type (e.g., APT attack T) half =72h, worm virus T half =4h).

[0034] 1.1 Compressed Sensing Storage Model

[0035] Mathematical Model and Formula Analysis

[0036] This invention constructs a spatiotemporal joint compressed sensing storage model, the core formula of which is:

[0037] X = Φ·S·Ψ T

[0038] Construction method:

[0039] (1) Spatial observation matrix Φ

[0040] The improved Gaussian random matrix is ​​constructed using the following steps:

[0041] Initialize an M×N matrix (M = 0.32N), with elements following a set order. distributed;

[0042] • Perform orthogonalization on each column (Gram-Schmidt method);

[0043] • Introduce sparsity constraints (the proportion of non-zero elements in each row is ≤15%).

[0044] (2) Time base matrix Ψ

[0045] Constructed based on Discrete Cosine Transform (DCT):

[0046]

[0047] Where k = 0, 1, ..., K-1, t = 0, 1, ..., T-1,

[0048] 1.2 Multimodal Source Tracing Engine

[0049] To improve the accuracy of source tracing, this device is designed with a source tracing reasoning model based on knowledge graphs:

[0050]

[0051] in:

[0052] A represents the attack path hypothesis, indicating possible attack routes;

[0053] E = {E1, E2, ..., E} n} is a collection of evidence, which includes various pieces of evidence extracted from network traffic;

[0054] sim(·) is a multimodal similarity function used to measure the degree of similarity between evidence and attack paths;

[0055] Z is the normalization factor, which ensures the rationality of the probability values.

[0056] 2. Implementation Method

[0057] 2.1 Traffic Processing Module

[0058] 2.1.1 Module Overview

[0059] The traffic processing module of this device is designed to efficiently process massive traffic data in high-speed network environments, achieving line-speed traffic filtering and ensuring the timeliness and accuracy of network threat detection. This module adopts an FPGA (Field-Programmable Gate Array) hardware architecture, possessing powerful parallel processing capabilities to meet the real-time processing requirements of high-throughput network traffic. Its designed throughput is ≥100Gbps, adapting to the requirements of modern high-speed network environments.

[0060] 2.1.2 Technical Details

[0061] 1. FPGA implementation of line-speed flow filtering

[0062] As a programmable hardware device, an FPGA can implement specific logic functions through customized circuit design and operate at near-hardware speeds. In this module, the FPGA is programmed to perform traffic filtering tasks. Its internal logic circuits can process multiple data streams in parallel, significantly improving traffic processing speed. The FPGA's parallel processing capability enables real-time monitoring and filtering of network traffic without reducing network speed, ensuring network smoothness and security.

[0063] 2. Improved sFlow sampling algorithm

[0064] Traditional sFlow sampling algorithms may suffer from uneven sampling and insufficient sensitivity to threat traffic when handling large-scale network traffic. To address this, this module introduces an improved sFlow sampling algorithm, the core of which lies in dynamically adjusting the sampling probability to adapt to different network conditions and threat levels. The sampling probability formula for this algorithm is:

[0065]

[0066] in:

[0067] C is a processing capacity constant, with a value of 2.3 × 10⁻⁶. 6 This reflects the processing limit of FPGA hardware;

[0068] B represents the current bandwidth utilization rate, used to measure the network's busyness.

[0069] N is the number of active connections, representing the number of ongoing communication connections in the current network;

[0070] The θ threat sensitivity parameter is used to adjust the sampling sensitivity to potential threat traffic.

[0071] 3. Parameter Description

[0072] Processing capacity constant C: This parameter is determined based on the FPGA's hardware performance and design requirements to ensure that the sampling algorithm does not exceed the hardware's processing capacity under high throughput conditions.

[0073] Current bandwidth utilization B: By monitoring network bandwidth usage in real time, the sampling probability is dynamically adjusted to ensure that the sampling frequency can be appropriately increased when the network is busy, so as to ensure that threat traffic is not missed.

[0074] Active Connections N: This factor considers the number of active connections in the network to avoid under- or over-sampling due to too many connections, ensuring the representativeness of the sampling.

[0075] Threat sensitivity parameter θ: Adjusted according to the security requirements and threat level of the network environment to increase the sampling probability of potential threat traffic and enhance the ability to detect network threats.

[0076] 2.1.3 Achieving Advantages

[0077] 1. High efficiency: Through the parallel processing capabilities of the FPGA and the improved sFlow sampling algorithm, this module can achieve efficient traffic filtering in high-throughput network environments, ensuring network smoothness and security.

[0078] 2. Flexibility: The dynamic adjustment mechanism of sampling probability enables this module to adapt to different network conditions and threat levels, and flexibly cope with various complex network environments.

[0079] 3. Accuracy: The improved sampling algorithm can more accurately capture threat traffic in the network, improve the accuracy and timeliness of threat detection, and provide strong protection for network security.

[0080] 2.2 Threat Intelligence Fusion Module

[0081] 2.2.1 Module Overview

[0082] The threat intelligence fusion module of this device is designed to efficiently process and integrate threat intelligence information from different sources, enabling the vectorization of unstructured intelligence for subsequent analysis and utilization. This module constructs a semantic analysis model based on BERT (Bidirectional Encoder Representations from Transformers), capable of deeply understanding the semantic content of intelligence texts and transforming them into a computer-processable vector form, providing strong support for the detection and response to cybersecurity threats.

[0083] 2.2.2 Technical Details

[0084] 1. BERT-based semantic analysis model

[0085] BERT, as a pre-trained language model, possesses powerful natural language understanding capabilities. This module utilizes the BERT model to perform semantic analysis on unstructured intelligence text, extracting key information and transforming it into a fixed-dimensional vector representation. In the specific implementation, the BERT model was fine-tuned and optimized to adapt to the specialized terminology and specific semantics of the cybersecurity field.

[0086] 2. Vectorized Representation Formula

[0087] The core formula for achieving vectorized representation of unstructured intelligence is:

[0088]

[0089] in:

[0090] W Q A trainable query matrix is ​​used to map the input intelligence text to a query vector space with dimensions d×d;

[0091] K and V are the key and value matrices in the knowledge base, respectively, both with a dimension of d×n, where n is the number of entries in the knowledge base;

[0092] d = 768 is the dimension parameter, which is the vector dimension output by the BERT model, ensuring the richness and accuracy of the vector representation.

[0093] 3. Parameter Description

[0094] Trainable query matrix W Q Through continuous optimization during the training process, the query vector can be better matched with key-value pairs in the knowledge base, thereby improving the accuracy of the vectorized representation of intelligence.

[0095] Knowledge base key-value pairs (K / V): Each entry in the knowledge base consists of a key and a value. The key is used to match the query vector, and the value contains the corresponding intelligence information. The construction of key-value pairs is based on a large amount of cybersecurity expertise and historical threat intelligence data.

[0096] The dimension parameter d determines the dimension of the vector representation. A 768-dimensional vector can effectively reduce data complexity and improve computational efficiency while maintaining information integrity.

[0097] 4. Intelligence update mechanism

[0098] To ensure the timeliness and accuracy of threat intelligence, this module is designed with an efficient intelligence update mechanism. When new threat intelligence emerges, the system can quickly add it to the knowledge base and update the corresponding key-value pairs. Through optimized update algorithms and hardware acceleration, the intelligence update latency has been significantly reduced to 38ms, far below the industry average (≥200ms), enabling this device to respond promptly to the latest network threats.

[0099] 2.3 Source Tracing Visualization Interface

[0100] 2.3.1 Module Overview

[0101] The device's attribution visualization interface is designed to present complex attack paths in an intuitive and clear visual format, helping cybersecurity analysts quickly understand the propagation path and scope of an attack. This module employs an improved ForceAtlas2 algorithm for attack path rendering, enabling efficient processing of large-scale network data and providing high-precision node location and real-time rendering capabilities, significantly improving the efficiency and accuracy of attack attribution.

[0102] 2.3.2 Technical Details

[0103] 1. Improved ForceAtlas2 algorithm

[0104] The traditional ForceAtlas2 algorithm may suffer from inaccurate node localization and slow rendering speed when processing large-scale network data. This module optimizes the ForceAtlas2 algorithm, mainly by adjusting the layout energy function and improving computational efficiency. The improved layout energy function is as follows:

[0105] E=∑(k rep ·||x i -x j || 2 )-∑(k att ·log(1+||x i -x j ||))

[0106] in:

[0107] k rep This is the repulsion coefficient, used to control the repulsive force between nodes and prevent excessive clustering of nodes;

[0108] k att The gravitational coefficient is used to control the attractive force between nodes and maintain the coherence of the network structure.

[0109] x i and x j These are the position vectors of node i and node j, respectively, used to calculate the distance between the nodes.

[0110] 2. Parameter Description

[0111] Repulsion coefficient k rep By adjusting this coefficient, the minimum distance between nodes can be controlled, preventing nodes from getting too close or overlapping during rendering and improving the readability of the visualization.

[0112] Gravitational coefficient k att This coefficient determines the tightness of the connection between nodes. An appropriate gravity coefficient can maintain the coherence and authenticity of the attack path and reflect the actual propagation path of the network attack.

[0113] Node position vector x i During the rendering process, the position vector of each node is dynamically adjusted according to the energy function to achieve the goal of minimizing global energy, thereby obtaining the optimal layout effect.

[0114] 3. Node positioning and rendering

[0115] By optimizing the layout energy function, this module improves node localization accuracy by 23.6% compared to traditional methods. Simultaneously, utilizing efficient computational algorithms and hardware acceleration technology, it supports 10... 5 Real-time rendering of nodes at each level ensures smooth visualization even when processing large-scale network data. The rendering process employs layered rendering and dynamic loading technologies, further improving rendering efficiency and response speed. Attached Figure Description

[0116] Figure 1 This is a diagram of the three-stage pipeline architecture of the method described in this invention. Detailed Implementation

[0117] The present invention will be further described below with reference to specific embodiments:

[0118] 1. Supplementary details on parameters and algorithm

[0119] 1.1β i Dynamic weight determination method

[0120] Achieving β using a two-level decision framework i Adaptive optimization:

[0121] (1) Expert Experience Layer: An initial weight matrix is ​​constructed based on indicators such as CVE vulnerability score (CVSS) and intelligence source authority (CISA / OSINT). The calculation formula is as follows:

[0122]

[0123] (Example: When CVSS = 8.5 and CISA source Rank = 5, β) i=1.047 → Take the upper limit 1)

[0124] (2) Data-driven layer: The LSTM model is used to train historical threat data, and the weights are dynamically adjusted.

[0125]

[0126] Where α is the learning rate (dynamically adjusted from 0.05 to 0.15), and ΔFPR is the false alarm rate change rate.

[0127] 1.2λ Calculation of Time Decay Factor

[0128] Dynamic update of λ based on exponential moving average algorithm:

[0129]

[0130] γ is the forgetting coefficient (0.75), T half Dynamically configured based on threat type:

[0131] APT attack: 72 hours;

[0132] Worm virus: 4 hours;

[0133] 0-day threat: Real-time calculation (based on threat spread rate model).

[0134] 1.3 Example of constructing a compressed matrix

[0135] (1) Spatial observation matrix Φ: Improved Gaussian random matrix

[0136] 1) Initialize an M×N matrix (M = 0.32N), with elements following the order of operations. distributed;

[0137] 2) Orthogonalization (Gram-Schmidt method);

[0138] 3) Sparsity constraint (≤15% of non-zero elements per row).

[0139] (2) Time base matrix Ψ: constructed based on Discrete Cosine Transform (DCT)

[0140]

[0141] Where k = 0, 1, ..., K-1, t = 0, 1, ..., T-1,

[0142] 2 Multi-scenario verification solutions

[0143] 2.1 Test Scenario Expansion

[0144] Verification will be conducted in the following typical scenarios:

[0145]

[0146] 2.2 Upgrade to the benchmark

[0147] The following publicly available benchmarks are used for comparison:

[0148] Traditional method: Snort rule matching (DARPA TC benchmark results); Storage efficiency: NSFNET dataset (reconstruction error ≤ 0.02dB);

[0149] Detection performance: CICIDS2017 dataset (F1-score comparison).

[0150] 3 Hardware Implementation Details

[0151] 3.1 FPGA Processing Platform Configuration

[0152] • Chip model: Xilinx UltraScale+XCVU190;

[0153] Key parameters:

[0154] Logic units: 1.9 million;

[0155] Block RAM: 12.3TB;

[0156] Maximum clock frequency: 400MHz.

[0157] • Actual performance:

[0158] 100Gbps line speed processing (tested with Ixia XM100);

[0159] Threat detection latency: <120μs (99.9% confidence level).

[0160] 3.2 Visualization Acceleration Solution

[0161] • GPU acceleration module: NVIDIA A100 PCIe 40GB;

[0162] • Distributed architecture:

[0163] Single-node processing capacity: 100,000 nodes / second (latency <30ms);

[0164] Cluster expansion: Supports horizontal scaling up to 1024 nodes.

[0165] • Acceleration technology:

[0166] CUDA parallel computing;

[0167] Spatial partitioning index (R-tree structure);

[0168] Dynamic LOD detail level optimization.

[0169] 4 Scientific Verification Methods

[0170] 4.1 Storage efficiency verification

[0171] • Dataset: CICIDS2017 (144GB);

[0172] • Indicator definition:

[0173]

[0174] ·result:

[0175] Improved method: 67.3% (Φ uses a Gaussian matrix);

[0176] Traditional method: 42.1% (random projection matrix).

[0177] 4.2 Verification of traceability accuracy

[0178] • Comparison algorithm:

[0179] Log-based correlation analysis (Splunk ES);

[0180] Snort rule matching (V2.9.16).

[0181] • Metric: F1-score;

[0182] ·result:

[0183] algorithm CICIDS2017 DARPATC This article's method 0.92 0.91 SplunkES 0.87 0.85 Snort rule matching 0.79 0.76

[0184] 5. Implementation and verification data

[0185] Deployment and verification on a financial cloud platform:

[0186] β i Dynamic adjustments improved threat mirroring accuracy by 19%.

[0187] λ-adaptive updates improve 0-day response speed by 42%;

[0188] The bit error rate for processing 100Gbps traffic is <10^-12 (compliant with IEEE 802.3 standard);

[0189] The visualization module has a stable latency of 28-45ms when rendering with 200 nodes.

[0190] 6 Multi-scenario verification result set

[0191] 6.1 Financial Cloud Environment Verification

[0192] • Deployment environment: A financial cloud egress (200G mixed traffic, including 15% encrypted traffic);

[0193] Verification results:

[0194] Threat detection rate: 98.7% (false alarm rate <0.1%);

[0195] Average evidence collection time: 9.2 minutes;

[0196] Storage efficiency: 81.3% space saving (3PB → 550TB);

[0197] Traceability accuracy (F1): 0.92.

[0198] 6.25G network verification

[0199] • Deployment environment: Huawei 5G testbed (high-concurrency small data packets, average packet length 120B); • Verification results:

[0200] Threat detection rate: 96.4% (78.2% with conventional methods);

[0201] Real-time processing latency: <120μs (traditional solution 3.2ms);

[0202] Storage efficiency: 79.2% savings (validated on CICIDS2017 dataset).

[0203] 6.3 IoT Environment Verification

[0204] • Deployment environment: CIC-IoT2020 dataset (including mixed traffic of ZigBee / BLE protocols);

[0205] Verification results:

[0206] Threat detection rate: 94.1% (65.3% with conventional methods);

[0207] Zero-day threat response time: 42 seconds (traditional solution 2.7 hours);

[0208] Node positioning accuracy: 0.035px (traditional ForceAtlas 20.046px).

[0209] 7. Comparative Advantage Analysis

[0210] 7.1 Cross-Scenario Performance Comparison

[0211]

[0212] 7.2 Hardware Performance Comparison

[0213] index This plan Traditional solution Linear speed processing capability 100Gbps (FPGA) 20Gbps (CPU) Real-time rendering latency <50ms (100,000 nodes) >300ms Intelligence update delay 38ms ≥200ms

[0214] 7.3 Industry standard compliance

[0215] Storage refactoring error: 0.018dB (NIST SP 800-61 requires ≤0.02dB);

[0216] Bit error rate: <10^-12 (IEEE 802.3 standard);

[0217] Floating-point precision: Compliant with IEEE 754 single-precision standard.

[0218] 8. Standardized dataset validation

[0219] 8.1 CICIDS2017 Dataset

[0220] Storage efficiency: 67.3% (traditional method 42.1%);

[0221] Threat detection rate: 98.1% (79.2% Snort rule match rate).

[0222] 8.2 DARPA TC Benchmark

[0223] Source tracing accuracy: F1 = 0.91 (Splunk ES 0.85);

[0224] Response time: 12.5 minutes (traditional solution 4.1 hours).

[0225] The embodiments of the present invention are not limited to the above description. The two-layer decision parameters of the threat intelligence confidence weight β_i, the dynamic update strategy of the time decay factor λ, the C / B / N / θ parameters of the improved sFlow sampling algorithm, and the Φ / Ψ matrix construction method of the spatiotemporal compressed storage model can be adjusted according to the actual network security scenario. All such improvements fall within the protection scope of the present invention.

Claims

1. A device integrating traffic mirroring data retention and threat tracing with built-in threat intelligence, characterized in that, include: The traffic processing module uses an FPGA hardware architecture to achieve 100Gbps line-rate traffic filtering, combined with an improved sFlow sampling algorithm to dynamically adjust the sampling probability, as shown in the formula. Where C is the processing capacity constant, B is the current bandwidth utilization, N is the number of active connections, and θ is the threat sensitivity parameter; The threat intelligence fusion module, based on the BERT semantic analysis model, vectorizes unstructured intelligence into... Among them W Q K is a trainable query matrix, and V is a knowledge base key-value matrix with d = 768. The source tracing visualization interface uses an improved ForceAtlas2 algorithm to render the attack path, and the layout energy function is... E=∑(k rep ·||x i -x j || 2 )-∑(k att ·log(1+||x i -x j ||)) Where k rep k is the repulsion coefficient. att This is the gravitational coefficient.

2. The apparatus according to claim 1, characterized in that, In the sampling algorithm of the traffic processing module, the processing capacity constant C is set to 2.3 × 10⁻⁶. 6 Linear filtering is achieved through parallel processing using FPGA.

3. The apparatus according to claim 1, characterized in that, The threat intelligence fusion module also includes an intelligence update mechanism with an update delay of ≤38ms. The time decay factor λ is dynamically adjusted using an exponential moving average algorithm, as shown in the formula: Where γ = 0.75, and T_{half} is dynamically set according to the threat type.

4. The apparatus according to claim 1, characterized in that, The traffic processing module adopts a spatiotemporal joint compression storage model X = Φ·S·Ψ T The spatial observation matrix Φ is an improved Gaussian random matrix, and the time basis matrix Ψ is constructed based on the discrete cosine transform.

5. The apparatus according to claim 1, characterized in that, The improved ForceAtlas2 algorithm of the source tracing visualization interface supports 10 5 Real-time rendering of nodes at all levels improves node positioning accuracy by 23.6% compared to traditional methods.

6. The apparatus according to claim 1, characterized in that, The threat intelligence fusion module determines the threat intelligence confidence weight β through a two-level decision-making mechanism. i : The expert experience layer constructs an initial matrix based on CVSS scores and the authority of intelligence sources. The machine learning layer dynamically adjusts the weights using the LSTM model. Where α is the learning rate and ΔFPR is the rate of change of the false alarm rate.

7. The apparatus according to claim 1, characterized in that, The multimodal tracing engine of the tracing visualization interface adopts a knowledge graph reasoning model. Where A is the attack path hypothesis, E is the evidence set, sim is the multimodal similarity function, and Z is the normalization factor.

Images