Threat intelligence analysis method and system

Abstract

The invention discloses a threat intelligence analysis method and system. The threat intelligence analysis method comprises the steps of obtaining a full-flow mirror image copy and log data; obtaining each PCAP file according to the full-flow mirror image copy; preprocessing the log data to obtain a structured log stream; extracting the flow metadata of each PCAP file; obtaining a flow data set after layered sampling according to the flow metadata; carrying out feature extraction on the flow data set and the log data after layered sampling so as to obtain a high-order fusion feature vector; and inputting the high-order fusion feature vector into a trained gating cycle unit network so as to obtain an anomaly detection probability value. According to the method, a comprehensive complexity index is constructed through a spatial frequency feature (IP change frequency), a port turnover rate feature (port switching rate) and a protocol switching entropy feature (protocol diversity), and the traffic behavior abnormal degree is quantified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and in particular to a threat intelligence analysis method and a threat intelligence analysis system. BACKGROUND

[0002] The existing threat intelligence analysis method performs real-time analysis through full-flow mirroring. Although this method can retain complete context, it has two major defects:

[0003] Source consumption out of control: full-flow processing requires continuous occupation of high-bandwidth storage and computing resources. In the case of TB-level daily traffic, the hardware cost increases exponentially;

[0004] Significant detection delay: real-time analysis of full-quantity data packets results in an analysis delay of minutes, making it difficult to meet the rapid response requirements of low-speed threats such as APT attacks.

[0005] Some threat intelligence analysis methods use a static sampling analysis scheme. Specifically, to reduce costs, some schemes use fixed-rate random sampling or simple filtering based on ports. However, they have the following disadvantages:

[0006] Key traffic is missed: a fixed sampling rate cannot adapt to traffic burst scenarios. Low-complexity traffic (such as periodic heartbeat packets) is over-sampled, wasting resources, while high-complexity attack traffic (such as DNS tunnels) may be missed due to insufficient sampling rate;

[0007] Single feature expression: relying only on shallow features such as five-tuple, lacking the depiction of deep behavior patterns such as IP entropy and port turnover rate, resulting in a detection rate of less than 60% for advanced threats such as encrypted traffic.

[0008] In addition, for the current recognition model, the traditional machine learning detection scheme is generally based on SVM, random forest, etc. The detection method has the following disadvantages:

[0009] Weak time series modeling capability: unable to capture long-period dependencies across data packets, with a detection F1 value of less than 75% for phase-based attacks such as C2 communication;

[0010] Poor dynamic adaptability: model parameters are fixed, and the false positive rate may rise to more than 30% when facing holiday traffic surges or new protocol versions.

[0011] Therefore, it is desirable to have a technical solution to solve or at least alleviate the above-mentioned deficiencies of the prior art. SUMMARY

[0012] The purpose of the present application is to provide a threat intelligence analysis method to at least solve one of the above technical problems.

[0013] The present application provides the following scheme:

[0014] According to an aspect of the present application, there is provided a threat intelligence analysis method, comprising:

[0015] obtaining a full-flow mirror copy, log data;

[0016] obtaining each PCAP file according to the full-flow mirror copy;

[0017] preprocessing the log data to obtain a structured log stream;

[0018] extracting flow metadata of each PCAP file;

[0019] obtaining a hierarchically sampled flow dataset according to the flow metadata;

[0020] extracting features from the hierarchically sampled flow dataset and the log data to obtain a high-order fusion feature vector;

[0021] inputting the high-order fusion feature vector into a trained gated recurrent unit network to obtain an anomaly detection probability value.

[0022] Optionally, the obtaining of the hierarchically sampled flow dataset according to the flow metadata and the structured log stream comprises:

[0023] processing each flow metadata as follows:

[0024] extracting features to obtain a spatial frequency feature, a port turnover rate feature, and a protocol switching entropy feature;

[0025] obtaining a hierarchical probability according to the spatial frequency feature, the port turnover rate feature, and the protocol switching entropy feature;

[0026] obtaining spatial frequency feature weight information according to the spatial frequency feature, port turnover rate weight information according to the port turnover rate feature, and protocol switching entropy weight information according to the protocol switching entropy feature;

[0027] obtaining a comprehensive complexity index according to the spatial frequency feature, the port turnover rate feature, the protocol switching entropy feature, the spatial frequency feature weight information, the port turnover rate weight information, and the protocol switching entropy weight information;

[0028] obtaining final hierarchical information according to the comprehensive complexity index;

[0029] obtaining a self-catalytic sampling model;

[0030] obtaining a dynamic sampling rate according to the final hierarchical information and the self-catalytic sampling model;

[0031] obtaining a time decay weight;

[0032] obtaining a final sampling decision according to the dynamic sampling rate and the time decay weight;

[0033] sampling each flow metadata according to the final sampling decision, thereby obtaining a flow dataset after hierarchical sampling.

[0034] Optionally, the feature extraction, thereby obtaining the spatial frequency feature, the port turnover rate feature and the protocol switching entropy feature, comprises:

[0035] the spatial frequency feature is obtained by the following formula:

[0036]

[0037] wherein, Δt i is the time interval of the ith IP change; δ(·) is the Kronecker function; F s is the spatial frequency, measuring the frequency and pattern of IP address change; IP i is the source IP address of the ith packet; IP i-1 is the source IP address of the (i-1)th packet.

[0038] Optionally, the port turnover rate feature is obtained by the following formula:

[0039]

[0040] wherein, R p is the port turnover rate, measuring the frequency of port switching; Port i is the port number of the ith packet; Port i-1 is the port number of the (i-1)th packet; λ is the time decay coefficient (λ=0.1), controlling the weight of recent port change; τ is the feature time window (τ=60 seconds), defining the time range of short-term port change; π is the circular constant; Δt i is the time interval between the ith packet and the (i-1)th packet.

[0041] Optionally, the protocol switching entropy is obtained by the following formula:

[0042] H swith =

[0043] -∑ ij [β·p stat (i→j)+(1-β)·p dyn (i→j|t)]log2[β·p stat (i→j)+(1-β)·p dyn (i→j|t)];

[0044] wherein, Hswith Pi,j is the protocol switching entropy; p stat (i→j) is the static transition probability, representing the long-term historical switching frequency of protocol i to j; p dyn (i→j|t) is the dynamic transition probability, representing the short-term switching frequency of protocol i to j near time t; β is the static weight (β = 0.7), balancing the influence of static and dynamic transition probabilities.

[0045] Optionally, the obtaining of the spatial frequency feature weight information according to the spatial frequency feature, the port turnover rate weight information according to the port turnover rate feature, and the protocol switching entropy weight information according to the protocol switching entropy feature comprises:

[0046] Any one of the spatial frequency feature weight information, the port turnover rate weight information, and the protocol switching entropy weight information is calculated by the following formula:

[0047] Δω i = η·(F i · F j - γ·ω i );

[0048] wherein η is a learning rate, controlling the weight adjustment amplitude; γ is a forgetting rate, simulating the weight decay of unused features; Δω i is a feature weight change amount, representing the weight adjustment amplitude of the i-th feature; F i and F j represent the activation values of the presynaptic and postsynaptic neurons respectively.

[0049] Optionally, the comprehensive complexity index is calculated by the following formula:

[0050]

[0051] wherein k controls the steepness of the Sigmoid function, and the greater k is, the more sensitive the complexity change is; θ defines the reference point of the complexity evaluation, and the complexity growth slows down after exceeding the threshold; F i represents the activation values of the presynaptic and postsynaptic neurons; ω i is the feature weight of the i-th feature; and C neuro is the comprehensive complexity index.

[0052] Optionally, the feature extraction on the flow data set and the log data after hierarchical sampling, thereby obtaining a high-order fusion feature vector comprises:

[0053] extracting a spatio-temporal feature from the flow data set after hierarchical sampling, so as to obtain a spatial frequency feature after sampling, a port turnover rate feature after sampling and a protocol switching entropy feature after sampling, the spatial frequency feature after sampling, the port turnover rate feature after sampling and the protocol switching entropy feature after sampling forming a spatio-temporal feature vector;

[0054] generating a dynamic strategy feature vector according to the spatial frequency feature after sampling, the port turnover rate feature after sampling, the protocol switching entropy feature after sampling and a dynamic sampling rate and a time decay weight;

[0055] obtaining a hierarchical context feature vector according to the spatial frequency feature after sampling, the port turnover rate feature after sampling, the protocol switching entropy feature after sampling and a comprehensive complexity index;

[0056] performing a nonlinear interaction operation on the spatio-temporal feature vector and the dynamic strategy feature vector, so as to obtain an interaction feature matrix;

[0057] enhancing a nonlinear expression capability of the spatio-temporal feature vector and the interaction feature matrix by a kernel method, so as to obtain a kernelized feature matrix;

[0058] performing a time series convolution coding on the spatio-temporal feature vector, the interaction feature matrix and the kernelized feature matrix, so as to obtain a time series coding feature;

[0059] performing a dynamic attention fusion on the hierarchical context feature vector, the time series coding feature, the interaction feature matrix and the kernelized feature matrix, so as to obtain a weighted fusion feature.

[0060] The application further provides a threat intelligence analysis system, which comprises:

[0061] a data acquisition module, which is used for acquiring a full-flow mirror copy and log data;

[0062] a PCAP file acquisition module, which is used for acquiring each PCAP file according to the full-flow mirror copy;

[0063] a structured log stream acquisition module, which is used for preprocessing the log data, so as to obtain a structured log stream;

[0064] a flow metadata acquisition module, which is used for extracting flow metadata of each PCAP file;

[0065] a flow data set acquisition module, which is used for acquiring a flow data set after hierarchical sampling according to the flow metadata;

[0066] a high-order fusion feature vector acquisition module, configured to perform feature extraction on the flow data set after hierarchical sampling and the log data, so as to obtain a high-order fusion feature vector;

[0067] an anomaly detection probability value acquisition module, configured to input the high-order fusion feature vector into a trained gated recurrent unit network, so as to obtain an anomaly detection probability value.

[0068] The threat intelligence analysis method has the following advantages:

[0069] The application constructs a comprehensive complexity index by using a spatial frequency feature (IP change frequency), a port turnover rate feature (port switching rate) and a protocol switching entropy feature (protocol diversity), and quantifies the degree of flow behavior anomaly;

[0070] The application adopts a self-catalytic sampling model, dynamically adjusts the sampling rate according to the network load (Load) and the historical detection accuracy (Acc_prev): the sampling rate of high complexity flow is increased to 80% to 100%, and the sampling rate of low complexity flow is reduced to 10% to 20%.

[0071] The application constructs a nonlinear interaction matrix of the spatiotemporal feature vector (spatial frequency after sampling, port turnover rate, protocol switching entropy) and the dynamic strategy feature vector (sampling rate, time decay weight); introduces a kernel method to enhance the nonlinear expression ability of the features, and extracts long-period patterns across data packets through time convolution coding. In the encrypted flow detection scene, the model AUC value is increased from 0.74 to 0.89, and the recognition delay of Cobalt Strike communication is shortened to within 15 seconds.

[0072] The dynamic gating threshold modulation module of the application generates a third-order gating tensor according to the hierarchical context features (high / medium / low complexity probability), realizes the spatiotemporal adaptive adjustment of the gating value; the nonlinear dynamics coupling module models the change law of the gating value through a system of differential equations, enhances the response speed to burst traffic; the topology memory coding module extracts the topological features of the historical hidden state using the theory of persistent homology, and improves the memory retention ability for phased attacks.

[0073] The nonlinear feedback loop module of the application generates a feedback signal according to the detection loss change amount, dynamically adjusts the memory strength; the gating parameter adjustment module combines the feedback signal with gradient descent, realizes online optimization of parameters. In the flow pattern mutation scene, the model adaptive adjustment period is shortened from 24 hours to 5 minutes, and the false positive rate is reduced by 67%. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is a flowchart of the threat intelligence analysis method in an embodiment of the application. Detailed Implementation

[0075] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0076] like Figure 1 The threat intelligence analysis methods shown include:

[0077] Step 1: Obtain a full traffic mirror copy and log data;

[0078] Step 2: Obtain each PCAP file based on the full-traffic mirror copy;

[0079] Step 3: Preprocess the log data to obtain a structured log stream;

[0080] Step 4: Extract traffic metadata from each of the PCAP files; In this embodiment, the traffic metadata includes at least the following data: 5-tuple, timestamp, and initial hash value, wherein the 5-tuple includes source IP address, source port, destination IP address, destination port, and transport layer protocol;

[0081] Step 5: Obtain the hierarchically sampled traffic dataset based on the traffic metadata;

[0082] Step 6: Extract features from the hierarchically sampled traffic dataset and log data to obtain a high-order fusion feature vector;

[0083] Step 7: Input the high-order fusion feature vector into the trained gated recurrent unit network to obtain the anomaly detection probability value.

[0084] In this embodiment, the PCAP file can be obtained through one or both of the following methods:

[0085] The full traffic data is divided by time, and a PCAP file is generated every hour.

[0086] The entire data stream is split according to its size. For example, when a file reaches 1GB, it is automatically split and a new file is generated.

[0087] In this embodiment, a full traffic mirror copy is captured from the network splitter using the Tcpdump or Moloch tools.

[0088] In the embodiment, the log data can be obtained by deploying a Fluentd log collector, configuring a Syslog plug-in to listen to the log output of a security device (such as a firewall, an EDR, or a mail gateway), extracting key fields (such as a source IP and an event type), and outputting to a Kafka message queue.

[0089] In the embodiment, obtaining each PCAP file according to the full-flow mirror copy includes:

[0090] According to a preset initial filtering rule, the data in the full-flow mirror copy is initially filtered, and each PCAP file is obtained by automatically dividing the file according to time or flow size, wherein the initial filtering rule can include an IP black / white list, a port range filtering, a protocol type filtering, a packet length threshold, and the like.

[0091] In the embodiment, obtaining the flow data set after hierarchical sampling according to the flow metadata and the structured log stream includes:

[0092] Each flow metadata is processed as follows:

[0093] Feature extraction is performed to obtain a spatial frequency feature, a port turnover rate feature, and a protocol switching entropy feature;

[0094] A hierarchical probability is obtained according to the spatial frequency feature, the port turnover rate feature, and the protocol switching entropy feature.

[0095] Spatial frequency feature weight information is obtained according to the spatial frequency feature, port turnover rate weight information is obtained according to the port turnover rate feature, and protocol switching entropy weight information is obtained according to the protocol switching entropy feature.

[0096] A comprehensive complexity index is obtained according to the spatial frequency feature, the port turnover rate feature, the protocol switching entropy feature, the spatial frequency feature weight information, the port turnover rate weight information, and the protocol switching entropy weight information.

[0097] Final hierarchical information is obtained according to the comprehensive complexity index.

[0098] A self-catalytic sampling model is obtained.

[0099] A dynamic sampling rate is obtained according to the final hierarchical information and the self-catalytic sampling model.

[0100] A time decay weight is obtained.

[0101] A final sampling decision is obtained according to the dynamic sampling rate and the time decay weight.

[0102] Each flow metadata is sampled according to the final sampling decision to obtain the flow data set after hierarchical sampling.

[0103] In the embodiment, feature extraction is performed to obtain spatial frequency features, port turnover rate features and protocol switching entropy features, including:

[0104] The spatial frequency features are obtained by the following formula:

[0105]

[0106] Where Δt i is the time interval of the ith IP change; δ(·) is the Kronecker function; F s is the spatial frequency, measuring the frequency and pattern of IP address changes; IP i is the source IP address of the ith packet; IP i-1 is the source IP address of the (i-1)th packet;

[0107] In the embodiment, the port turnover rate features are obtained by the following formula:

[0108]

[0109] Where R p is the port turnover rate, measuring the frequency of port switching; Port i is the port number of the ith packet; Port i-1 is the port number of the (i-1)th packet; λ is the time decay coefficient (λ = 0.1), controlling the weight of recent port changes; τ is the feature time window (τ = 60 seconds), defining the time range of short-term port changes; π is the circular constant; Δt i is the time interval between the ith packet and the (i-1)th packet;

[0110] In the embodiment, the protocol switching entropy is obtained by the following formula:

[0111] H swith = -∑ ij [β·p stat (i→j)+(1-β)·p dyn (i→j|t)]log2[β·p stat (i→

[0112] j)+(1-β)·p dyn (i→j|t)];

[0113] Where H swith is the protocol switching entropy; p stat (i→j) is the static transition probability, representing the long-term historical switching frequency of protocol i to j; p dyn(i→j|t) dynamic transition probability, representing the short-term switching frequency of protocol i to j around time t; β static weight (β = 0.7), balancing the influence of static and dynamic transition probabilities.

[0114] In the embodiment, the space frequency feature weight information is acquired according to the space frequency feature, the port turnover rate weight information is acquired according to the port turnover rate feature, and the protocol switching entropy weight information is acquired according to the protocol switching entropy feature.

[0115] Any one of the space frequency feature weight information, the port turnover rate weight information and the protocol switching entropy weight information is calculated by the following formula:

[0116] Δω i = η · (F i · F j - γ · ω i ); wherein η is a learning rate, controlling the weight adjustment amplitude; γ is a forgetting rate, simulating the weight decay of unused features; Δω i is a feature weight change, representing the weight adjustment amplitude of the i th feature; F i and F j represent the activation values of presynaptic and postsynaptic neurons respectively. In the traffic layering scenario, it can be understood as the correlation between different features (such as space frequency, port turnover rate), if F i and F j are active at the same time, the weight is enhanced.

[0117] The space frequency feature weight information, the port turnover rate weight information and the protocol switching entropy weight information are the starting point of the dynamic traffic layering step, and the three feature values output are used for subsequent neural complexity evaluation. By adjusting the feature weight through the Hebbian learning rule, the traffic complexity can be quantified more accurately, providing strong support for layering decision. Therefore, the space frequency feature weight information, the port turnover rate weight information and the protocol switching entropy weight information play a crucial role in the whole dynamic traffic layering process.

[0118] In the embodiment, the comprehensive complexity index is calculated by the following formula:

[0119] Wherein, k controls the steepness of the Sigmoid function, the greater k, the more sensitive the complexity change is; F i represents the activation values of presynaptic and postsynaptic neurons; ω i is the feature weight of the i th feature; θ defines the reference point of complexity evaluation, and the complexity growth slows down after exceeding the threshold; C neuro is the comprehensive complexity index. In the embodiment, k is 5 and θ is 0.5.

[0120] In this embodiment, the comprehensive complexity indicator represents the complexity level of the traffic flow.

[0121] In this embodiment, obtaining the final hierarchical information according to the comprehensive complexity indicator comprises:

[0122] using fuzzy C-means clustering on C neuro to determine the probability that the traffic flow belongs to high, medium, and low complexity.

[0123] Fuzzy C-means assigns data points to different clusters and gives hierarchical probabilities (P high , P med , P low ) that belong to each cluster, i.e., final hierarchical information, to represent the probability that the traffic flow belongs to high, medium, and low complexity, respectively, by iteratively optimizing an objective function.

[0124] For example, assume there is a piece of traffic data, as follows:

[0125] Spatial frequency feature (F_s) = 0.8 (high frequency IP changes);

[0126] Port turnover rate feature (R_p) = 0.6 (moderate port switching);

[0127] Protocol switching entropy feature (H_switch) = 0.9 (highly random protocol switching);

[0128] After calculating the weights using the above formulas, the weights can be adjusted as follows:

[0129] Spatial frequency feature weight information w_s = 0.5 (strengthen the impact of IP changes);

[0130] Port turnover rate weight information w_p = 0.3 (moderately retain port information);

[0131] Protocol switching entropy weight information w_h = 0.2 (weaken the randomness of protocols);

[0132] Then the comprehensive complexity indicator is:

[0133]

[0134] Input C neuro = 0.70 into the fuzzy C-means clustering (assuming that three clustering centers have been trained through historical data: v_high = 0.85, v_med = 0.5, v_low = 0.15), and calculate the membership:

[0135] P_high = 1 / (1+(0.70-0.85) 2 / (0.7-0.5) 2) ≈ 0.3;

[0136] P_med = 1 / (1+(0.70-0.5) 2 / (0.70-0.85) 2)≈0.7;

[0137] P_low ≈ 0;

[0138] Then it means that the traffic has 70% probability of being medium complexity, 30% probability of being high complexity, and the probability of being low complexity can be ignored.

[0139] In this embodiment, the autocatalytic sampling model is as follows:

[0140]

[0141] where S is the sampling rate; k cat is the catalytic rate constant, controlling the speed of the sampling rate growth; S max is the maximum sampling rate, limiting the upper limit of the sampling rate; k inh is the inhibition rate constant, reflecting the inhibition of network load on the sampling rate; this equation describes how the sampling rate changes over time: in the absence of inhibition, the sampling rate will grow due to autocatalysis (the first term); while the network load will inhibit the growth of the sampling rate (the second term);

[0142] Solve the autocatalytic sampling model using numerical methods (such as the Runge-Kutta method) to obtain the dynamic sampling rate S(t); this sampling rate will change over time, reflecting the dynamic changes of network traffic complexity and network load.

[0143] In this embodiment, obtaining the dynamic sampling rate according to the final hierarchical information and the autocatalytic sampling model includes: obtaining the dynamic sampling rate using an enhanced adaptive sampling strategy, and the specific process is as follows:

[0144] In the training phase:

[0145] Initialize the Q table:

[0146] In this embodiment, the Q table structure is as follows: the dimension is |state space| x |action space|, that is, each state-action pair corresponds to a Q value.

[0147] The state space has 3 dimensions (Layer, Acc_prev, Load), and each dimension is discretized (such as Layer is divided into High / Medium / Low, Acc_prev and Load are discretized into 10 grades with an interval of 0.1).

[0148] The action space can include 10 discrete sampling rates (10%, 20%,..., 100%).

[0149] Initialization: all Q values are set to 0 or a small random value.

[0150] Randomly select action with probability ε (e.g. 0.1), explore unknown state-action pairs.

[0151] Select action with maximum current Q value with probability 1-ε.

[0152] For each training cycle (e.g. each sampling round), perform the following steps:

[0153] Observe current state S t = {Layer t , Acc prev,t , Load t}; Layer t is calculated by the multi-dimensional entropy-weighted layering module; Acc prev,t is the detection accuracy after the last sampling round; Load t is the current network load (e.g. obtained through SNMP monitoring).

[0154] Select action A t = Sample_Rate, according to the ε-greedy strategy;

[0155] Perform action: sample traffic at the selected sampling rate;

[0156] Observe reward R t and next state S t+1 ;

[0157] Update Q value: Q(S t , A t ) ← Q(S t , A t ) + α[R t + γ · max A′ Q(S t+1 , A′) - Q(S t , A t )]; α = 0.1, α is the learning rate, controlling the update step size. γ = 0.9, γ is the discount factor, valuing long-term rewards. State transition: S t ← S t+1 , enter the next cycle.

[0158] Terminate when one of the following termination conditions is met:

[0159] Pre-set training rounds (e.g. 1000 iterations) are reached; Q values converge (change less than a threshold);

[0160] Detection performance (e.g. accuracy, F1 score) is stable.

[0161] Execution phase:

[0162] Load the trained Q table;

[0163] Inference using the Q-value table optimized in the training phase.

[0164] Real-time decision-making process: observe the current state S t : Layer t Through real-time traffic feature calculation, Acc prev,t Detection results after the last round of sampling, Load t Real-time network load;

[0165] Select the optimal action:

[0166] A t = arg max A Q(S t , A);

[0167] Directly select the sampling rate with the maximum Q value, with A t probability to sample the traffic.

[0168] In this embodiment, the state space is the core input of reinforcement learning decision-making, which is used to describe the running state of the current system. In the scheme in this paper, the state space is defined as S = {Layer, Acc prev , Load}, and the meaning and calculation method of each dimension are as follows:

[0169] Layer (comprehensive complexity index), which serves to quantify the complexity of traffic flow and guide the allocation of differentiated resources by sampling strategy for traffic of different complexity.

[0170] Acc prev (Historical detection accuracy), which serves to reflect the accuracy of model detection of anomalies after the last round of sampling, and is used to balance the sampling rate and detection performance. It is obtained directly through model evaluation, and the value range is [0, 1].

[0171] Load (current network load), which serves to represent the network resource occupancy rate to avoid overloading in the sampling process. It is calculated by monitoring network bandwidth, CPU / memory usage, etc., and the value range is [0, 1].

[0172] In this embodiment, the action space defines the set of operations that can be executed by the sampling strategy. In this paper, the action space is A = {Sample_Rate}, i.e., the sampling rate, with a discrete value range (such as 10%, 20%,..., 100%). The sampling rate is dynamically adjusted through the Q-learning algorithm.

[0173] In this embodiment, feature extraction is performed on the traffic data set after hierarchical sampling and log data, thereby obtaining a high-order fusion feature vector, including:

[0174] extracting a space-time feature from the flow data set after the hierarchical sampling, so as to obtain a spatial frequency feature after sampling, a port turnover rate feature after sampling and a protocol switching entropy feature after sampling, the spatial frequency feature after sampling, the port turnover rate feature after sampling and the protocol switching entropy feature after sampling forming a space-time feature vector;

[0175] generating a dynamic strategy feature vector according to the spatial frequency feature after sampling, the port turnover rate feature after sampling, the protocol switching entropy feature after sampling and a dynamic sampling rate and a time decay weight;

[0176] obtaining a hierarchical context feature vector according to the spatial frequency feature after sampling, the port turnover rate feature after sampling, the protocol switching entropy feature after sampling and a comprehensive complexity index;

[0177] performing a nonlinear interaction operation on the space-time feature vector and the dynamic strategy feature vector, so as to obtain an interaction feature matrix;

[0178] enhancing a nonlinear expression capability of the space-time feature vector and the interaction feature matrix by a kernel method, so as to obtain a kernelized feature matrix;

[0179] performing a time sequence convolution coding on the space-time feature vector, the interaction feature matrix and the kernelized feature matrix, so as to obtain a time sequence coding feature;

[0180] performing a dynamic attention fusion on the hierarchical context feature vector, the time sequence coding feature, the interaction feature matrix and the kernelized feature matrix, so as to obtain a weighted fusion feature.

[0181] In the embodiment, generating a dynamic strategy feature vector according to the spatial frequency feature after sampling, the port turnover rate feature after sampling, the protocol switching entropy feature after sampling and a dynamic sampling rate and a time decay weight comprises:

[0182] Model initialization:

[0183] Setting a catalytic rate constant k cat , a maximum sampling rate S max , an inhibition rate constant k inh ;

[0184] Solving the following equation by the Runge-Kutta method to obtain a dynamic sampling rate S(t):

[0185]

[0186] Mapping S(t) to the interval [0, 1] as a feature Sdynamic collected by a dynamic sampling rate;

[0187] Time decay weight calculation:

[0188] Obtain the data packet timestamp sequence {Ti}i=1N, the current time Tcurrent, the Gaussian mixture model parameters (mean μk, variance σk2, mixing weight πk);

[0189] Time difference calculation:

[0190] For each data packet, calculate its difference with the current time:

[0191] Δt i = T current - T i ;

[0192] Gaussian mixture decay:

[0193] Simulate the importance decay of recent, medium-term and long-term traffic through K Gaussian components:

[0194]

[0195] μ k is the mean of the kth Gaussian component, reflecting the decay center of different time scales (such as short-term, medium-term and long-term), σk2 k is the variance of the kth Gaussian component, controlling the decay rate, π

[0196] Divide W(Δt) by the maximum value to ensure the range is between [0, 1], and generate the time decay weight feature Wtime;

[0197] The hierarchical context feature vector is obtained according to the sampled spatial frequency feature, the sampled port turnover rate feature, the sampled protocol switching entropy feature and the comprehensive complexity index, and comprises:

[0198] Obtain the hierarchical sampled traffic data set (containing the high / medium / low complexity membership matrix U∈R N×C , N: number of data packets, C: number of cluster centers (corresponding to high / medium / low complexity, C=3);

[0199] Normalize and smooth the membership matrix U output by the fuzzy C-means.

[0200] Convert the normalized membership to a probability form to generate a complexity probability vector;

[0201] Introduce non-linear correction through the logistic function to enhance the significance of high / low complexity differentiation, thereby obtaining the corrected complexity probability vector;

[0202] Combine the corrected probability into a hierarchical context feature vector.

[0203] The nonlinear interaction of the spatio-temporal feature vector and the dynamic strategy feature vector obtains an interaction feature matrix, including:

[0204] The Cartesian product operation of the spatial frequency (F_s) and the port turnover rate (F_p) generates a first interaction matrix;

[0205] The Cartesian product operation of the dynamic sampling rate (S_dynamic) and the time attenuation weight (W_time) generates a second interaction matrix;

[0206] Element-wise multiplication of the protocol switching entropy (F_e) and the time attenuation weight (W_time) generates a third interaction matrix, and the first interaction matrix, the second interaction matrix and the third interaction matrix constitute the interaction feature matrix.

[0207] The nonlinear expression ability of the kernel method is enhanced for the spatio-temporal feature vector and the interaction feature matrix, so as to obtain a kernelized feature matrix, including:

[0208] The Gaussian similarity of each feature vector with a random anchor point is calculated:

[0209] Wherein, γ=0.5 is a kernel width parameter, and c is a randomly selected anchor point.

[0210] The spatio-temporal feature vector, the interaction feature matrix and the kernelized feature matrix are sequentially convolutional coded, so as to obtain a time sequence coding feature, including:

[0211] Parallel application of multiple groups of different time window convolution kernels extracts multi-scale time sequence patterns:

[0212]

[0213] Wherein, W (k) represents the kth group of convolution kernel, and the time window length T k ; T1=3, T2=5, T3=10 respectively correspond to short, medium and long time scales; σ(·) is an activation function; F input is an input feature (basic spatio-temporal feature vector, interaction feature matrix and kernelized feature matrix);

[0214] The dynamic attention fusion of the hierarchical context feature vector, the time sequence coding feature, the interaction feature matrix and the kernelized feature matrix obtains a weighted fusion feature, including:

[0215] The dynamic attention weight is generated according to the hierarchical context feature by the following formula, and the feature channel of high complexity traffic is strengthened:

[0216]

[0217] wherein λ: attention sharpening coefficient (λ>0), controls the contribution intensity of complexity probability to weight; α l is the attention weight corresponding to the lth complexity (α high +α med +α low =1); P l ′ is the corrected complexity probability of the lth complexity.

[0218] The channel-level weighting is performed on the time-series convolution feature, the interaction feature matrix and the kernelized feature matrix by the following formula:

[0219]

[0220] wherein, is the input feature subset associated with the lth complexity; is the weighted feature corresponding to the lth complexity;

[0221] The weighted features are spliced by modal to generate a fusion feature vector.

[0222] Specifically, the splicing is performed by the following formula:

[0223]

[0224] wherein F fused is the fusion feature vector; is the weighted time-series convolution feature corresponding to the lth complexity; is the weighted interaction feature matrix corresponding to the lth complexity; is the weighted kernelized feature matrix corresponding to the lth complexity.

[0225] In the embodiment, the control cycle unit network includes a dynamic gate threshold modulation module, a nonlinear dynamics coupling module, a high-order memory injection module, a topological memory encoding module and a gate fusion module.

[0226] The input of the dynamic gate threshold modulation module is a hierarchical context feature vector, which is used to construct a third-order gate tensor, generate a sampling vector and calculate a gate threshold;

[0227] The input of the nonlinear dynamics coupling module is an updated gate threshold θ update and a reset gate threshold θ reset ; a differential equation set is constructed to describe the change rule of the dynamic gate value Θ, a numerical method is used to solve the differential equation to obtain the dynamically adjusted gate value Θ(t);

[0228] The input of the high-order memory injection module is a high-order fusion feature vector X t, the history hidden state sequence, construct a third-order Volterra kernel K(τ1, τ2, τ3), combine the Gaussian basis function and the nonlinear mixing coefficient, perform the memory convolution operation, and convolve the history hidden state with the Volterra kernel to obtain the candidate hidden state h after short-term memory injection

[0229] The input of the topological memory encoding module is a history hidden state sequence, specifically, the persistence homology features of the history hidden state sequence are calculated to obtain the Betti number, and the Betti number is mapped to the topologically encoded hidden state through linear transformation

[0230] The gating fusion module is used for the following processing:

[0231] According to the update gate threshold and the reset gate threshold, the candidate hidden state and the history hidden state are fused to obtain the fused hidden state h t 。;

[0232] The nonlinear feedback loop module is used to build a self-excited oscillator to describe the change rule of the feedback signal f, and generate the feedback signal according to the loss change and the output of the self-excited oscillator.

[0233] The adaptive memory curve module updates the memory strength according to the time encoding function and the memory half-life period; and adjusts the memory strength to reflect the importance of the current detection result in combination with the learning rate and the feedback signal.

[0234] The gating parameter adjustment module adjusts the gating parameter to optimize the detection performance according to the feedback strength coefficient and the gradient of the loss function on the gating parameter.

[0235] The tensor probability mapping module is used to construct a fourth-order probability tensor P, which is obtained through singular value decomposition and orthogonal basis vectors; and performs multiple tensor contraction operations to map the logit to a probability value.

[0236] The nonlinear probability calibration module is used to convert the logit to the final anomaly detection probability value Pfinal according to the piecewise polynomial calibration function.

[0237] The tensor gating routing module is used to construct a query tensor Q and a key tensor K, calculate a routing weight tensor R through a Softmax function and a tensor contraction operation, and perform an information routing operation to output the routed hidden state

[0238] The memory emergence network module is used to build a memory graph, which is used to describe the association relationship between the history hidden states, and propagate the memory information in the memory graph through a graph attention mechanism to obtain the updated hidden state.

[0239] In this embodiment, the third-order gated tensor is constructed using the following formula:

[0240]

[0241] Among them, T gate (i,j,k): The value of the third-order gated tensor at position (i,j,k). σ: The Sigmoid activation function. W pqr : Trainable tensor kernel with dimensions P×Q×R. The i-th statistical moment of the p-th input feature, where i∈{1,2,3} represents the moment order. P,Q,R: the dimensions of the tensor kernel.

[0242] The generated sampling vector is constructed using the following formula:

[0243] u = Softmax(W u C), v = Softmax(W) v h t-1 );

[0244] Where u is a sampling vector generated based on the hierarchical context feature vector C, used to capture the importance of patterns with different complexities; v is a sampling vector based on the historical hidden state h. t-1 The generated sampling vectors are used to capture the importance of temporal dependencies; W u W v The trainable weight matrix maps the hierarchical context feature vector and the historical hidden state to the sampling space, respectively; C is the hierarchical context feature vector, containing high, medium, and low complexity probability values ​​C1, C2, and C3; h t-1 It represents the hidden state from the previous moment, carrying temporal information;

[0245] The Softmax normalization exponential function ensures that the sum of the elements of the sampled vector is 1, reflecting the importance weights of different modes.

[0246] w = Sigmoid(W) w [C;h] t-1 ]); where w is a joint sampling vector based on hierarchical context feature vectors and historical hidden states, used to capture the joint effects of spatiotemporal complexity. W w Given a trainable weight matrix, the concatenated vector is mapped to the sampling space. [C;h] t-1 To represent the vector concatenation operation, the hierarchical context feature vector C and the historical hidden state h are concatenated. t-1 The elements of the sampled vector are connected to form a longer vector; the igmoid function is a sigmoid function that maps the elements of the sampled vector to the (0,1) interval, reflecting the influence intensity of different spatiotemporal patterns.

[0247] The gating threshold is calculated using the following formula:

[0248] θ update = T gate x1u x2v x3T; where, θ update : update gate threshold. T gate : third-order gating tensor. xn denotes the contraction operation of tensor along the nth dimension. u, v, T: sampling vectors.

[0249] In the embodiment, the third-order Volterra kernel K is constructed by the following formula:

[0250] K (τ1, τ2, τ3) = φ (τ1) φ (τ2) φ (τ3) + λ∑ perm φ (τ1) φ (τ2) φ (τ3) ;

[0251] Where, K (τ1, τ2, τ3) : third-order Volterra kernel. φ (τ) : Gaussian kernel, φ (τ) = e-τ2 / σ2. λ: nonlinear mixing coefficient. σ: width parameter of Gaussian kernel.

[0252] In the embodiment, the candidate hidden state after short-term memory injection is obtained by the following formula:

[0253]

[0254] Where, is the candidate hidden state after short-term memory injection. h t-τ is the history hidden state. ⊙ is Hadamard product.

[0255] In the embodiment, the persistent homology feature PH is obtained by the following formula:

[0256] PH (H t-T:t ) = {β0, β1, β2} ; Where, PH (H t-T:t ) : persistent homology feature of history hidden state sequence. β0, β1, β2: 0-dimensional, 1-dimensional, 2-dimensional Betti number. H t-T:t : history hidden state sequence.

[0257] The hidden state after topological memory encoding by mapping Betti number to topological memory encoding by linear transformation is obtained by the following formula:

[0258] Where, is the hidden state after topological memory encoding. W topo , b topo : trainable parameters.

[0259] In the embodiment, the candidate hidden state after short-term memory injection is obtained by the following formula:

[0260]

[0261] wherein, candidate hidden state. x W h : trainable weight matrix. t : high-order fusion feature vector. t : reset gate output. h : bias vector.

[0262] In this embodiment, the fused hidden state is obtained according to the update gate threshold and the reset gate threshold, and the fusion of the candidate hidden state and the historical hidden state, and is obtained by the following formula:

[0263]

[0264] wherein, h t is the fused hidden state. t is the update gate output; and is the Hadamard product.

[0265] In this embodiment, the feedback signal is obtained by the following formula:

[0266]

[0267] wherein, f is the feedback signal; δ is the damping coefficient; ω0 is the natural frequency; ∈ is the nonlinearity strength; A is the excitation amplitude; ω is the forced frequency;

[0268] f t = (1 + ηf osc (t)) · sign(ΔL t ); wherein f t is the feedback signal at time “t”, which is used to adjust the gating parameter to optimize the model performance; η is the feedback modulation coefficient, which controls the influence degree of the non-linear feedback loop output on the feedback signal; f osc (t) is the output of the self-excited oscillator, which represents the oscillation state of the non-linear feedback loop at time “t”. ΔL t is the loss change at time “t”, which represents the accuracy change of the model detection result; sign() is the sign function, which is used to extract the direction information (positive or negative) of the loss change;

[0269] In this embodiment, the gating parameter is adjusted to optimize the detection performance according to the feedback strength coefficient and the gradient of the loss function on the gating parameter, and is obtained by the following formula:

[0270]

[0271] wherein, wherein k is a feedback strength coefficient; w is a feedback weight; and gate L t is a loss function gradient of the gating parameter.

[0272] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A threat intelligence analysis method, characterized in that, The threat intelligence analysis method comprises: acquiring full-flow mirror copies, log data; acquiring each PCAP file according to the full-flow mirror copies; preprocessing the log data to acquire a structured log stream; extracting traffic metadata of each PCAP file; acquiring a hierarchically sampled traffic dataset according to the traffic metadata; extracting features from the hierarchically sampled traffic dataset and the log data to acquire a high-order fusion feature vector; inputting the high-order fusion feature vector into a trained gated recurrent unit network to acquire an anomaly detection probability value.

2. The threat intelligence analysis method of claim 1, wherein, The acquisition of the hierarchically sampled traffic dataset according to the traffic metadata and the structured log stream comprises: processing each traffic metadata as follows: extracting features to acquire spatial frequency features, port turnover rate features, and protocol switching entropy features; acquiring hierarchical probabilities according to the spatial frequency features, the port turnover rate features, and the protocol switching entropy features; acquiring spatial frequency feature weight information according to the spatial frequency features, port turnover rate weight information according to the port turnover rate features, and protocol switching entropy weight information according to the protocol switching entropy features; acquiring a comprehensive complexity index according to the spatial frequency features, the port turnover rate features, the protocol switching entropy features, the spatial frequency feature weight information, the port turnover rate weight information, and the protocol switching entropy weight information; acquiring final hierarchical information according to the comprehensive complexity index; acquiring a self-catalytic sampling model; acquiring a dynamic sampling rate according to the final hierarchical information and the self-catalytic sampling model; acquiring a time decay weight; acquiring a final sampling decision according to the dynamic sampling rate and the time decay weight; sampling each traffic metadata according to the final sampling decision to acquire the hierarchically sampled traffic dataset.

3. The threat intelligence analysis method of claim 2, wherein, The extraction of features to acquire spatial frequency features, port turnover rate features, and protocol switching entropy features comprises: acquiring spatial frequency features by the following formula: where Δt i is the time interval of the ith IP change; δ(·) is the Kronecker function; F s is the spatial frequency, which measures the frequency and pattern of IP address changes; IP i is the source IP address of the ith packet, IP i-1 is the source IP address of the (i-1)th packet.

4. The threat intelligence analysis method of claim 3, wherein, acquiring port turnover rate features by the following formula: where R p is the port turnover rate, measuring the frequency of port switching; Port i is the port number of the i-th packet; Port i-1 is the port number of the i-1-th packet; λ is the time decay coefficient, controlling the weight of recent port changes, τ is the characteristic time window, defining the time range of short-term port changes; π is the circular constant; Δt i is the time interval between the i-th packet and the i-1-th packet.

5. The threat intelligence analysis method of claim 4, wherein, acquiring protocol switching entropy by the following formula: H swith = -∑ ij [β·p stat (i→j)+(1-β)·p dyn (i→j|t)]log2[β·p stat (i→j)+(1-β)·p dyn (i→j|t)] where H swith is the protocol switching entropy; p stat (i→j) is the static transition probability, representing the long-term historical switching frequency from protocol i to j; p dyn (i→j | t) is the dynamic transition probability, representing the short-term switching frequency from protocol i to j around time t; β is the static weight (β = 0.7), balancing the influence of static and dynamic transition probabilities.

6. The threat intelligence analysis method of claim 5, wherein, The acquisition of spatial frequency feature weight information according to the spatial frequency features, port turnover rate weight information according to the port turnover rate features, and protocol switching entropy weight information according to the protocol switching entropy features comprises: calculating any one of the spatial frequency feature weight information, the port turnover rate weight information, and the protocol switching entropy weight information by the following formula: Δω i = η · (F i · F j - γ · ω i ); where η is the learning rate, controlling the weight adjustment amplitude; γ is the forgetting rate, simulating the weight decay of unused features; Δω i is the feature weight change, representing the weight adjustment amplitude of the i-th feature; F i and F j represent the activation values of presynaptic and postsynaptic neurons, respectively.

7. The threat intelligence analysis method of claim 6, wherein, calculating the comprehensive complexity index by the following formula: where k controls the steepness of the sigmoid function, the greater k is, the more sensitive the complexity change is; F i represent the activation values of presynaptic and postsynaptic neurons; ω i is the feature weight of the ith feature; θ defines the reference point of the complexity evaluation, and the complexity growth slows down after exceeding the threshold; C neuro is the comprehensive complexity index.

8. The threat intelligence analysis method of claim 7, wherein, The extraction of features from the hierarchically sampled traffic dataset and the log data to acquire a high-order fusion feature vector comprises: The time-space feature vector is extracted from the flow data set after hierarchical sampling, so as to obtain a spatial frequency feature after sampling, a port turnover rate feature after sampling and a protocol switching entropy feature after sampling, wherein the spatial frequency feature after sampling, the port turnover rate feature after sampling and the protocol switching entropy feature after sampling constitute the time-space feature vector; The dynamic strategy feature vector is generated according to the spatial frequency feature after sampling, the port turnover rate feature after sampling, the protocol switching entropy feature after sampling and a dynamic sampling rate and a time decay weight; The hierarchical context feature vector is obtained according to the spatial frequency feature after sampling, the port turnover rate feature after sampling, the protocol switching entropy feature after sampling and a comprehensive complexity index; The time-space feature vector and the dynamic strategy feature vector are subjected to nonlinear interaction operation, so as to obtain an interaction feature matrix; The time-space feature vector and the interaction feature matrix are subjected to kernel method to enhance the nonlinear expression ability of the features, so as to obtain a kernelized feature matrix; The time-space feature vector, the interaction feature matrix and the kernelized feature matrix are subjected to time sequence convolution coding, so as to obtain time sequence coding features; The hierarchical context feature vector, the time sequence coding features, the interaction feature matrix and the kernelized feature matrix are subjected to dynamic attention fusion, so as to obtain weighted fusion features.

9. A threat intelligence analysis system characterized by, The threat intelligence analysis system comprises: A data acquisition module, which is configured to acquire a full-flow mirror copy and log data; A PCAP file acquisition module, which is configured to acquire each PCAP file according to the full-flow mirror copy; A structured log stream acquisition module, which is configured to preprocess the log data, so as to acquire a structured log stream; A flow metadata acquisition module, which is configured to extract flow metadata of each PCAP file; A flow data set acquisition module, which is configured to acquire a flow data set after hierarchical sampling according to the flow metadata; A high-order fusion feature vector acquisition module, which is configured to extract features from the flow data set after hierarchical sampling and the log data, so as to acquire a high-order fusion feature vector; An anomaly detection probability value acquisition module, which is configured to input the high-order fusion feature vector into a trained gated recurrent unit network, so as to acquire an anomaly detection probability value.

Images