Abnormal traffic detection and attack identification method and system based on deep learning

Abstract

The invention belongs to the technical field of network security, and provides an abnormal traffic detection and attack recognition method and system based on deep learning, and the method comprises the steps: data preprocessing and feature extraction, cross-modal semantic alignment and knowledge graph construction, causal enhancement association reasoning, intelligent engine optimization, cloud edge collaborative resource scheduling, and result output. According to the method, statistical features and signature features are mapped to a unified semantic space through a cross-modal semantic alignment and knowledge graph construction module, a semantic barrier between heterogeneous features is broken through, time sequence causal discovery and transfer entropy calculation are introduced, a simple correlation and a reliable causal can be distinguished, and the method has a good application prospect. According to the method, the accuracy and credibility of attack chain reasoning are improved, the false alarm rate is reduced, online self-evolution of a detection model and dynamic optimal allocation of system resources are realized through intelligent engine optimization and cloud edge collaborative resource scheduling modules, and the overall adaptability, robustness and practicability of the system are enhanced.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to a method and system for abnormal traffic detection and attack identification based on deep learning. Background Technology

[0002] Anomaly traffic detection and attack identification are core tasks of network security protection. Anomaly traffic detection refers to identifying abnormal data that deviates from normal traffic patterns by analyzing traffic data transmitted in the network. Attack identification is based on anomaly detection, further matching attack characteristics to determine the attack type.

[0003] Existing abnormal traffic detection and attack identification technologies can be mainly divided into three categories: signature-based detection, statistical anomaly detection technology, and detection based on traditional deep learning.

[0004] However, existing technologies have some shortcomings, specifically as follows: Existing methods typically process numerical statistical features and symbolic signature features in parallel or independently. These two feature modalities are quite different: statistical features are continuous numerical vectors, while signature features are discrete symbolic identifiers. Due to the lack of cross-modal semantic alignment mechanisms, these two features are mapped to completely different semantic spaces, forming a semantic gap. This fundamental heterogeneity prevents the system from effectively associating traffic anomalies with specific attack behaviors, allowing only isolated judgments and making it difficult to construct a complete attack chain view, resulting in false positives and false negatives. Existing methods based on knowledge graphs or association rules mostly construct association relationships based on static presets or simple co-occurrence statistics. They lack the ability to analyze causal relationships and temporal evolution, and cannot distinguish between simple correlations and reliable causal relationships between features. For example, they cannot determine whether port scanning caused data leakage or whether the two simply occurred simultaneously by chance. Furthermore, their knowledge systems cannot be updated online, lacking incremental learning and federated learning mechanisms to integrate newly emerging attack patterns on edge nodes. This results in the knowledge graph becoming outdated quickly and its adaptive capabilities being severely insufficient when the system faces attackers fine-tuning their attack methods or launching entirely new zero-day attacks. Summary of the Invention

[0005] The purpose of this application is to propose a method and system for abnormal traffic detection and attack identification based on deep learning, so as to solve at least one technical problem mentioned in the background art.

[0006] To address the aforementioned technical problems, this application provides a deep learning-based method for abnormal traffic detection and attack identification, employing the following technical solution: A deep learning-based method for anomaly traffic detection and attack identification includes the following steps: Raw traffic data is collected from network devices, and statistical and signature features are extracted in parallel. The statistical features are then standardized to obtain standardized statistical features. These standardized statistical and signature features are encoded into a set of vectors in a unified semantic space to construct an initial knowledge graph. A graph attention network is used to calculate the attention weights between nodes in the initial knowledge graph and to select candidate node pairs with high similarity. A temporal causal relationship algorithm is used to distinguish the causal and correlation relationships between candidate node pairs. The causal strength is quantified by combining transitive entropy, and the final edge weights are obtained by fusing semantic similarity and causal credibility. The knowledge graph is dynamically updated through incremental learning and federated learning. Anomaly detection and attack identification engines are constructed and optimized. Reinforcement learning is used to dynamically schedule signature subsets, and a generative adversarial network is used to achieve co-evolution of the two engines. A game model is built based on the two engines, and a particle swarm optimization algorithm is used to optimize the resource allocation of the two engines. Lightweight tasks and computationally intensive tasks are divided according to the GPU acceleration ratio, and cloud-edge collaborative offloading is performed. The resource priority of the two engines is adjusted based on the attack severity using a deep Q-network. Attack links are traced according to high-weight edges, and hierarchical alarm information and resource allocation reports are output.

[0007] Preferably, the statistical feature extraction adopts the sliding window method, with the window size consistent with the sampling frequency, to extract the number of data packets per second, average packet length, retransmission rate, connection duration, and connection establishment success rate; the signature feature extraction uses regular expression matching and ngram methods to obtain attack signature codes, abnormal ports, and abnormal protocols from a public attack rule base; the standardization process performs Z-score normalization based on the mean and standard deviation of the training set.

[0008] Preferably, the statistical feature encoding uses the TrafficBERT model to output a 768-dimensional semantic vector, and the signature feature encoding uses the Sig2Vec model to map the discrete signature into a continuous vector that is in the same space as the statistical feature semantic vector. The initial knowledge graph is constructed using the k-nearest neighbor algorithm, and the semantic similarity of nodes is calculated using cosine similarity. The nodes with the highest similarity are selected to construct the initial edge with the current node.

[0009] Preferably, the temporal causal discovery algorithm filters potential causal relationships through conditional independence tests and determines reliable causal pairs by combining statistical significance p-values; the transfer entropy is normalized and then fused with statistical significance weights to calculate causal credibility, and the final edge weights are obtained by weighted fusion of semantic similarity and causal credibility, with the weight coefficients optimized by minimizing attack chain inference errors; the incremental learning uses an incremental graph neural network to update graph parameters, and federated learning is implemented by updating parameters locally at edge nodes and aggregating global parameters in the cloud, protecting the data privacy of edge nodes while adapting to new attack modes.

[0010] Preferably, the anomaly detection engine employs a variational autoencoder, and the attack identification engine employs a convolutional neural network; the reinforcement learning signature scheduling uses a deep Q-network, with detection accuracy and resource consumption as reward functions. The deep Q-network models signature selection as a Markov decision process, with states including attack type, traffic intensity, and CPU utilization, and actions including selecting a subset of signatures for different attack types. The reward function prioritizes detection accuracy and outputs the optimal subset of signatures with the largest Q value.

[0011] Preferably, in the generative adversarial network, the attack identification engine acts as the generator, and the anomaly detection engine acts as the discriminator. Unknown anomalies trigger the generation of new signatures. The generative adversarial network constructs a feedback loop of generating simulated attacks, detecting unknown anomalies, and updating signatures. The method is as follows: the attack identification engine acts as the generator, inputting random noise and attack statistical features to generate simulated attack vectors; the anomaly detection engine acts as the discriminator, distinguishing between real traffic and simulated attack vectors; adversarial training improves the discriminator's ability to detect unknown anomalies and the authenticity of the simulated attacks generated by the generator; after detecting unknown anomalies, a new signature is generated to update the signature library of the attack identification engine.

[0012] Preferably, the constraint condition of the game model is that the sum of the CPU ratios of the two engines is ≤0.5, and 50% of the CPU resources are reserved for other system tasks. The payoff function of the game model is the weighted sum of detection accuracy and resource consumption. The particle swarm optimization algorithm aims to maximize the total payoff of the two engines and converges to the Nash equilibrium point through iterative updates of particle velocity and position.

[0013] Preferably, the lightweight tasks are executed at the edge, and the computationally intensive tasks are executed in the cloud. The method for offloading cloud-edge tasks is as follows: the lightweight tasks, including traffic collection, feature encoding, and real-time detection, are processed at the edge, and only the input data of the intensive tasks is uploaded to the cloud; the intensive tasks such as knowledge graph updates and global parameter synchronization are processed by GPUs at the cloud, and only the processing results are sent to the edge.

[0014] To address the aforementioned technical issues, this application also provides a deep learning-based abnormal traffic detection and attack identification system, comprising: a data acquisition module for real-time acquisition of raw traffic data from network devices; a preprocessing module for cleaning, normalizing, and formatting the raw traffic data to generate a structured dataset; a feature extraction module for parallel extraction of statistical features and signature features from the structured dataset, where statistical features are used to calculate traffic distribution indicators through a time-series window, and signature features are used to generate behavioral fingerprints based on an attack rule base, while the statistical features are standardized to eliminate the influence of dimensions, outputting standardized statistical features and signature features; a cross-modal semantic alignment module for deploying a bidirectional traffic encoder representation model and a signature feature vector embedding model, where the former encodes standardized statistical features into statistical feature semantic vectors, and the latter encodes signature features into signature feature semantic vectors, placing the two types of vectors in the same semantic space to eliminate the semantic gap; a knowledge graph construction module for using the two types of semantic vectors as graph nodes, calculating the semantic similarity between nodes using the k-nearest neighbor algorithm, and selecting high-similarity nodes to construct an initial knowledge graph; and causal enhanced association reasoning. The knowledge graph is divided into three modules: a knowledge graph module and an intelligent engine optimization module. The intelligent engine optimization module includes an unsupervised learning-based anomaly detection engine and a supervised learning-based attack identification engine. The intelligent engine optimizes the system resource allocation by dynamically scheduling attack signature subsets through reinforcement learning and constructing an adversarial feedback loop through generative adversarial networks to generate simulated attacks, detect unknown anomalies, and update the signature database, achieving the co-evolution of the two engines. The cloud-edge collaborative resource scheduling module models system resource allocation as a Nash equilibrium problem through game theory and uses optimization algorithms to solve for the optimal resource allocation strategy. It also performs intelligent task offloading and result synchronization between edge nodes and the cloud based on the computational intensity of the task. The result output module performs attack chain inference based on the final knowledge graph and generates a visualization report. It outputs graded alarm information based on association weights and attack severity levels and generates a system resource utilization report.

[0015] Preferably, the feature extraction module uses the sliding window statistical method to extract the number of data packets per second, average packet length, retransmission rate, average connection duration, and connection establishment success rate as statistical features; and extracts attack signature codes, abnormal ports, and abnormal protocols as signature features from a public attack rule base using regular expression matching. The cross-modal semantic alignment module uses the TrafficBERT model to encode statistical features and the Sig2Vec model to embed signature features. The parameters of the TrafficBERT model are obtained through pre-training on normal traffic, and the parameters of the Sig2Vec model are obtained through training on the Snort rule base. The knowledge graph construction module uses cosine similarity to calculate the semantic similarity between nodes. During initialization, it first constructs a borderless graph, and then selects the node with the highest similarity for each node to construct an edge, thus forming an initial knowledge graph. The causal enhanced association reasoning module uses the PCMCI algorithm to perform conditional independence tests and significance calculations to discover reliable causal relationships, and uses transitive entropy to quantify causal strength. Finally, it weights and fuses semantic similarity weights with causal credibility scores into the final edge weights. In the intelligent engine optimization module, the anomaly detection engine adopts a variational autoencoder model, and the attack recognition engine adopts a convolutional neural network model; the reinforcement learning adopts a deep Q-network to select the optimal signature subset using detection accuracy and resource consumption as reward functions; the generative adversarial network uses the attack recognition engine as the generator and the anomaly detection engine as the discriminator. The cloud-edge collaborative resource scheduling module uses the particle swarm optimization algorithm to solve the Nash equilibrium point to obtain the optimal CPU resource ratio allocated to the dual engines, and deploys traffic collection, feature encoding and real-time detection tasks on the edge side according to the GPU acceleration ratio, while offloading knowledge graph update, global signature library synchronization and complex graph reasoning tasks to the cloud. The output module iterates through the edges in the knowledge with weights higher than a set threshold to reconstruct the attack chain, and classifies the attack chain into three alarm priorities: high, medium, and low, based on the causal strength and attack type label of the attack chain.

[0016] The beneficial effects of this application are as follows: This application provides a deep learning-based method for anomaly traffic detection and attack identification. By using cross-modal semantic alignment of TrafficBERT and Sig2Vec, two types of heterogeneous features are mapped to a unified 768-dimensional semantic space. In DDoS attack detection scenarios, the system can directly calculate the cosine similarity between the statistical feature vector of high-frequency SYN packets and the signature feature vector of SYN Flood, detecting the high semantic correlation between traffic anomalies and specific attacks. This significantly improves the accuracy and interpretability of feature association, laying a solid foundation for the subsequent construction of high-quality knowledge graphs.

[0017] This method introduces causal enhanced correlation reasoning and a dynamic graph update mechanism to analyze and quantify the causal direction and strength between features from a time-series perspective. In advanced persistent threat detection scenarios, the system can effectively distinguish whether port scanning leads to subsequent data leakage or whether the two are merely unrelated background noise, thereby constructing a credible causal relationship that conforms to the attack logic chain, greatly reducing the false alarm rate and improving the credibility of threat assessment.

[0018] This method employs a dual-engine collaborative optimization and cloud-edge game-theoretic scheduling strategy, based on reinforcement learning-driven dynamic signature scheduling. This allows the system to load only the most relevant subset of signatures and unload irrelevant ones when facing specific attacks. This reduces the computational resource consumption of edge devices by over 30% while maintaining the detection accuracy of critical attacks, achieving optimal scheduling with limited resources. Through the adversarial feedback loop of GANs and federated learning mechanisms, it builds self-evolution capabilities. When an unknown ransomware variant is detected for the first time at an edge node, GAN can generate its simulated signature and automatically update the local and global signature databases after confirmation. Simultaneously, federated learning allows multiple edge nodes to aggregate model parameter increments without sharing original data, jointly training a more powerful detection model. This protects data privacy and significantly shortens the global response time against new threats. Attached Figure Description

[0019] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of abnormal traffic detection and attack identification provided in the embodiments of this application; Figure 2 This is a flowchart of dual-engine collaboration and adversarial optimization provided in the embodiments of this application; Figure 3 This is an exemplary system architecture diagram in which this application can be applied. Detailed Implementation

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0024] Example 1: A deep learning-based method for anomaly traffic detection and attack identification includes steps 1-6. Wherein: Step 1: Data preprocessing and feature extraction: Collect raw traffic data from network devices, extract statistical features and signature features in parallel, and standardize the statistical features to obtain standardized statistical features.

[0025] Furthermore, statistical feature extraction employs a sliding window method, with the window size consistent with the sampling frequency, to extract data packets per second, average packet length, retransmission rate, connection duration, and connection establishment success rate; signature feature extraction utilizes regular expression matching and ngram methods to obtain attack signature codes, abnormal ports, and abnormal protocols from a publicly available attack rule base; standardization is performed based on Z-score normalization using the mean and standard deviation of the training set.

[0026] The workflow for data preprocessing and feature extraction is as follows: Step 1.1: Raw Traffic Data Acquisition. Using network packet capture tools such as tcpdump and Zeek, or the Flink stream processing framework, capture raw network transmission data from devices such as routers, switches, and network probes. This data should include formats such as NetFlow traffic session logs, pcap data packets, and device logs. Extract basic information, including source / destination IPs, protocols, packet sizes, and timestamps. Set the sampling frequency to 1 second to ensure real-time performance. The collected data will then be the raw traffic dataset.

[0027] Step 1.2: Dual-engine feature extraction: extract statistical features and signature features in parallel.

[0028] Step 1.2.1: Statistical Feature Extraction: Using the sliding window statistical method, with a window size of 1 second (consistent with the sampling frequency), quantitative features reflecting the macroscopic patterns of traffic are extracted from the original traffic dataset, including data packets per second, average packet length, connection establishment success rate, retransmission rate, and connection duration; the statistical feature matrix X is output. stat ∈R N×MN represents the number of samples, and M represents the statistical feature dimension.

[0029] Formula 1: X stat =[f pps ,f bytes ,f retrans ,f duration ,f success ].

[0030] In Formula 1: f pps Data packets per second (bpd) is the total number of data packets transmitted per unit of time, expressed in packets per second. bytes This represents the average packet length, the average number of bytes in all data packets per unit time, expressed in bytes. retrans The retransmission rate is the percentage of data packets retransmitted per unit time out of the total number of data packets sent, expressed as a percentage (%). duration Average connection duration: the average duration of all successful connections per unit time, in seconds. success Connection establishment success rate is the percentage of successfully established TCP / UDP connections per unit of time out of the total number of connection requests, expressed in percent (%).

[0031] Step 1.2.2: Signature Feature Extraction: Obtain attack fingerprints from a publicly available attack rule base, such as the UNION SELECT statement in SQL injection and the SYN flag in DDoS attacks; use regular expression matching and ngram extraction methods to extract discrete features reflecting the attack fingerprints from the original traffic dataset, including attack signature codes, abnormal ports, and abnormal protocols. Output signature feature set X. sig K represents the number of signatures.

[0032] Formula 2: X sig =[s code ,s port ,s proto ,...,s K In Formula 2: S i ∈{0,1}. s code This is an attack signature identifier. port This is an identifier for an abnormal port. proto This is an identifier for abnormal protocols. K represents the number of signatures.

[0033] Step 1.3: Data Standardization: Use Z-score standardization to standardize the statistical features, eliminating the influence of dimensions, and obtain the standardized statistical features X. norm .

[0034] Standardization is performed using formula 3: X norm =(X stat -μ train ) / σ train .

[0035] In Formula 3: X stat For statistical characteristics, μ train σ represents the mean of the features in the training set. train denoted as the standard deviation of the training set features.

[0036] Formula 4: X norm =[f norm,pps ,f norm,bytes ,f norm,retrans ,f norm,duration ,f norm,success In Formula 4: f norm,pps ,f norm,bytes ,f norm,retrans ,f norm,duration ,f norm,success f pps ,f bytes ,f retrans ,f duration ,f success The normalized component after standardization.

[0037] Step 2: Cross-modal semantic alignment and knowledge graph construction: Encode standardized statistical features and signature features into a set of vectors in a unified semantic space to construct an initial knowledge graph.

[0038] Furthermore, the statistical feature encoding uses the TrafficBERT model, outputting a 768-dimensional semantic vector, while the signature feature encoding uses the Sig2Vec model, mapping discrete signatures to continuous vectors that reside in the same space as the statistical feature semantic vectors. The initial knowledge graph is constructed using the k-nearest neighbor algorithm, and node semantic similarity is calculated using cosine similarity. The nodes with the highest similarity are selected to construct initial edges with the current node.

[0039] The workflow for cross-modal semantic alignment and knowledge graph construction is as follows: Step 2.1: Statistical Feature Encoding: Use the Traffic Bidirectional Encoder Representation Model (TraffBERT) to encode the statistical features X. norm Encode as a vector.

[0040] TraffBERT is an improved traffic statistics feature encoding model based on the bidirectional encoder representation model (BERT), which maps numerical statistical features to a 768-dimensional unified semantic space.

[0041] Formula 5: h stat =TraffBERT(X norm ;θ stat In Formula 5: X norm For standardized statistical characteristics. θ statThese are the parameters for the TrafficBERT model, obtained through pre-training with normal traffic. stat h is a statistical feature semantic vector. stat ∈R N×768 N is the number of samples, and 768 is the semantic vector dimension.

[0042] Step 2.2: Signature Feature Embedding: Use the Signature Feature Vector Embedding Model (Sig2Vec) to embed the signature features X sig Encoded as a continuous vector.

[0043] Formula 6: h sig =Sig2Vec(X sig ;θ sig In Formula 6: X sig θ represents the set of signature features. sig The parameters for the Sig2Vec model are obtained through training using the Snort rule base. sig h is the signature feature semantic vector. sig ∈R K×768 K is the number of signatures, and 768 is the semantic vector dimension.

[0044] h sig and h stat Encoding into a unified semantic space results in a set of vectors sharing the same vector space: h sig with h stat Both have 768 dimensions. TrafficBERT is pre-trained with normal traffic and learns the statistical semantics of normal and abnormal traffic in 768 dimensions. Sig2Vec is trained with the Snort rule base and learns the semantics of attack signatures in 768 dimensions. Although the training data of the two are different, the target semantics are consistent, both revolving around whether the traffic is abnormal and whether it contains attacks. In the end, they achieve the same semantics in the same dimension and eliminate the semantic gap.

[0045] Step 2.3: Initial Knowledge Graph (CMSKG) Construction: Using semantic vectors as graph nodes, initial edges are constructed through nearest neighbor relationships to form a cross-modal semantically aligned knowledge graph.

[0046] h stat and h sig As node V, initialize the borderless graph.

[0047] Using the k-nearest neighbor (kNN) graph construction algorithm, for any node V i ∈V, calculate its relationship with all other nodes V j Cosine similarity S ∈V semantic .

[0048] Formula 7: S semantic (i,j)=(h i ·hj ) / (||h i ||2·||h j ||2). In Formula 7: S semantic (i,j) represents the semantic similarity score between node i and node j; the larger the value, the more similar the two nodes are. i h j The node vectors all originate from h. stat and h sig The set of nodes formed by these nodes. · represents the vector dot product. ||·||2 represents the L2 norm.

[0049] For V i Select the k nodes with the highest similarity (k=5, determined by cross-validation), and then select V. i The connections to these k nodes are labeled as edges; after traversing all nodes, all edges are summed to form an edge set E={(i,j)}. Combining the node set V and the edge set E yields the initial graph G=(V,E).

[0050] Step 3: Causal Enhancement and Association Reasoning: A graph attention network is used to calculate the attention weights between nodes in the initial knowledge graph and to select candidate node pairs with high similarity. The algorithm distinguishes the causal and correlation relationships between candidate node pairs through temporal causality. The causal strength is quantified by combining transitive entropy. The final edge weights are obtained by fusing semantic similarity and causal credibility. At the same time, the knowledge graph is dynamically updated through incremental learning and federated learning.

[0051] Furthermore, the temporal causal discovery algorithm filters potential causal relationships through conditional independence tests and determines reliable causal pairs by combining statistical significance p-values. The transfer entropy, after normalization, is fused with statistical significance weights to calculate causal credibility. Finally, the edge weights are obtained through a weighted fusion of semantic similarity and causal credibility, and the weight coefficients are optimized by minimizing the inference error of the attack chain. Incremental learning uses an incremental graph neural network to update graph parameters, while federated learning is implemented by updating parameters locally at edge nodes and aggregating global parameters in the cloud, protecting the data privacy of edge nodes while adapting to new attack patterns.

[0052] The workflow for causal-enhanced associative reasoning is as follows: Step 3.1: Initial association reasoning: Calculate the attention weights between nodes using a graph attention network (GATv2).

[0053] Formula 8: e ij =LeakyReLU(a T [Wh i ||Wh j ]). In formula 8: e ij The attention score is given for nodes i and j. i h is the semantic vector of node i. jLet W be the semantic vector of node j. 256×768 is the learnable weight matrix. a is the attention vector. || is vector concatenation. LeakyReLU is the activation function.

[0054] Normalized attention weights, Formula 9: α ij =exp(e ij ) / ∑ k∈N(i) exp(e ik In Formula 9: α ij ∈[0,1] represents the attention weights. N(i) is the set of neighboring nodes of node i, and exp(·) is an exponential function to ensure that the weights are positive. ij Give attention scores for nodes i and j. ik The attention scores for nodes i and k.

[0055] The normalized attention weights are directly used as the semantic similarity weights S. semantic (i,j), i.e., S semantic (i,j)=α ij For all edges (i,j) of the initial graph G, calculate e in sequence. ij α ij Ultimately, S was obtained. semantic S semantic ∈[0,1], a larger value indicates that nodes i and j are semantically more similar. Update the edge set to E. sem ={(i,j,S semantic (i,j))}, retain S semantic Node pairs with a high similarity of ≥0.3 are used as candidate sets for subsequent causal analysis.

[0056] Step 3.2: Temporal Causal Discovery: Analyze the temporal dependencies of candidate node pairs, distinguish between causality and correlation, and screen for significant causal relationships.

[0057] Step 3.2.1: Conditional Independence Test: Use the Temporal Causal Discovery (PCMCI) algorithm to test the independence of X(t) and Y(t+W) after excluding the interference Z.

[0058] Formula 10: X(t)⊥Y(t+W1)|Z(t),W1. In Formula 10: X(t) is the causal variable, such as the number of port scans at time t. Y(t+W1) is the result variable at time t+W1, such as the outgoing data traffic. Z(t) is the set of condition variables, such as the number of normal HTTP requests at time t. W1 is the time window. ⊥ indicates conditional independence.

[0059] Test logic: Test the independence of X(t) and Y(t+W1) after excluding interference Z(t). If X(t) and Y(t+W1) are not independent, then X(t) is considered to be a potential cause of Y(t+W1).

[0060] Step 3.2.2: Calculation of statistical significance p-value: Calculate causal significance through permutation test.

[0061] Formula 11: p(X→Y)=P(stat(X′,Y′|Z)≥stat(X,Y|Z)). In Formula 11: p(X→Y) is the statistical significance p-value indicating a causal relationship between X and Y, ranging from [0,1]. stat(▪) is the mutual information, measuring nonlinear association. X′,Y′ are time-series permutations of X and Y, disrupting the causal order. P(·) is the probability function.

[0062] Judgment criterion: If p(X→Y) < 0.05 (statistical significance level), then X→Y is considered a reliable causal relationship. Node pairs with p(X→Y) < 0.05 are retained, resulting in the set of reliable causal pairs C = {(X→Y, p(X→Y))}.

[0063] Step 3.3: Transmission Entropy Calculation: Calculate the transmission entropy (TE) to measure the strength of the causal effect.

[0064] Formula 12: TE(X→Y) = H(Y(t+W1)|Y(t)) - H(Y(t+W1)|Y(t),X(t)). In Formula 12: TE(X→Y) is the transfer entropy of X to Y (unit: bits), which measures the amount of information transferred. The larger the value, the greater the amount of information transferred from X to Y, and the stronger the causal effect. H(·) is the conditional entropy, a measure of uncertainty. Y(t+W1) is the outcome variable at time t+W1. X(t) is the X variable at time t. Y(t) is the Y variable at time t.

[0065] Normalize the transfer entropy and scale the TE to the [0,1] interval.

[0066] Formula 13: TE norm (X→Y)=TE(X→Y) / TE max . In Equation 13, TE norm TE is the normalized transitive entropy. TE(X→Y) is the original transitive entropy. max The maximum TE value for all causal pairs.

[0067] Step 3.4: Causal credibility assessment: Combining semantic similarity and causal credibility, the final weights of the graph edges are obtained. Formula 14: S causal =(1-p(X→Y))·TE norm (X→Y). In Formula 14, S causal represents the causal confidence score, ranging from [0,1]. p(X→Y) is the statistical significance p-value. TE norm To normalize the transfer entropy.

[0068] Final edge weight fusion, Formula 15: S final =α·S semantic +β·S causal In Formula 15: α = 0.4 is the semantic similarity weight, β = 0.6 is the causal credibility, obtained by minimizing the inference error of the attack chain, α + β = 1. final This represents the final edge weight, ranging from [0,1]. The larger the value, the more reliable the association.

[0069] Let the edge weight of the reliable causal pair be S. final Update the graph edge set to E final ={(i,j,S final (i,j))}, to obtain the final cross-modal semantic alignment knowledge graph (CMSKG), where edge weights reflect the reliability of associations.

[0070] Step 3.5: Dynamic graph update: Adapt to new attacks through incremental learning and federated learning to avoid graph obsolescence, while protecting the data privacy of edge nodes.

[0071] Incremental graph neural network (IncGNN) is used for parameter updates.

[0072] Formula 16: θ GNN (t+1)=θ GNN (t)+η·▽ θGNN L(θ GNN (t),D new In Formula 16: θ GNN (t+1) represents the GNN parameters at time t+1. θ GNN (t) represents the GNN parameters at time t, such as W in step 3.1. new For the characteristic data of new traffic, such as the new X stat X sig L represents the cross-entropy loss during attack chain inference. ▽ θGNN L is the loss function L against θ GNN The gradient. η = 0.001 is the learning rate.

[0073] Parameter aggregation, Formula 17: θ GNN，global =(∑ k n k ·θ GNN,local,k ) / (∑ k n k In Formula 17: θ GNN，global These are the global GNN parameters aggregated in the cloud. θ GNN,local,k Let n be the local GNN parameters for the k-th edge node. k This represents the new data volume of the k-th node.

[0074] Edge nodes use D locallynew Update θ GNN,local Only the parameter gradients are uploaded to the cloud, and the cloud aggregates and then sends θ. GNN，global Edge nodes update their local CMSKG. The resulting dynamically evolved CMSKG can identify new attacks and traffic patterns, providing the latest correlation information for subsequent engine optimization.

[0075] Step 4: Intelligent Engine Optimization: Build and optimize the anomaly detection engine and attack identification engine, use reinforcement learning to dynamically schedule signature subsets, and achieve dual-engine co-evolution through generative adversarial networks.

[0076] Furthermore, the anomaly detection engine employs a variational autoencoder, while the attack identification engine uses a convolutional neural network. The reinforcement learning signature scheduling uses a deep Q-network with detection accuracy and resource consumption as reward functions. The deep Q-network models signature selection as a Markov decision process, with states including attack type, traffic intensity, and CPU utilization, and actions including selecting a subset of signatures with different attack types. The reward function prioritizes detection accuracy and outputs the optimal subset of signatures with the largest Q value.

[0077] Furthermore, in the generative adversarial network (GAN), the attack identification engine acts as the generator, and the anomaly detection engine acts as the discriminator. Unknown anomalies trigger the generation of new signatures. The GAN constructs a feedback loop of generating simulated attacks, detecting unknown anomalies, and updating signatures. The method is as follows: the attack identification engine acts as the generator, inputting random noise and attack statistical features to generate simulated attack vectors. The anomaly detection engine acts as the discriminator, distinguishing between real traffic and simulated attack vectors. Through adversarial training, the discriminator's ability to detect unknown anomalies and the authenticity of the simulated attacks generated by the generator are improved. After detecting unknown anomalies, a new signature is generated to update the signature database of the attack identification engine.

[0078] The intelligent engine optimization workflow is as follows: Step 4.1: Dual Engine Initialization: Initialize the anomaly detection engine (E1) and the attack identification engine (E2).

[0079] Step 4.1.1: Anomaly Detection Engine E1 Initialization: Construct an unsupervised anomaly detection model to identify unknown abnormal traffic.

[0080] Train on normal traffic using a variational autoencoder (VAE) to learn the normal distribution. Use a large amount of normal traffic h... stat Train the VAE so that the model learns to reconstruct normal traffic with a reconstruction error < 0.1. The processed output is the initial E1, with parameters θ. enc0 (Encoder weights), θ dec0 (Decoder weights) can be used to determine anomalies by "reconstruction error > threshold 0.5", and the threshold is determined by cross-validation.

[0081] Step 4.1.2: Initialization of the attack identification engine E2: Build a supervised attack identification model and match known attack signatures.

[0082] Train a convolutional neural network (CNN) on known attack data, using h of known attack samples. sig CNN trained with human labels achieves target accuracy ≥90%; The processed output is the initial E2, with parameter θ. cnn0 (CNN layer weights), with an initial signature library S init (X) sig (Corresponding signature).

[0083] Step 4.2: Reinforcement learning signature scheduling: Dynamically select the optimal subset of E2 signatures to improve E2 detection efficiency and reduce resource consumption.

[0084] The signature selection is modeled as a Markov decision process (MDP) using a DeepQ Network (DQN).

[0085] MDP quadruple definition: State S=[A type ,T intensity Res usage Among them: A type The current attack type is: 0 = Normal / 1 = DDoS / 2 = SQL Injection. intensity For flow intensity, normalized to [0,1]. Res usage CPU utilization is normalized to [0,1].

[0086] Action A: Select a subset of signatures, such as A1=DDoS subset, A2=SQL injection subset.

[0087] The reward R(S,A) = λ1·Acc(A) - λ2·Res(A). Where: R(S,A) is the reward value for taking action A in state S. Acc(A) is the detection accuracy of the subset of signatures corresponding to action A. Res(A) is the resource consumption rate of taking action A. λ1 and λ2 are weights, λ1=0.9, λ2=0.1, prioritizing accuracy.

[0088] The DQN model is trained using historical data, and the Q function is updated.

[0089] Formula 18: Q(S,A;θ) q )=E[R(S,A)+γ·max A' Q(S',A';θ q In Formula 18: Q(S,A;θ) q Let θ be the expected cumulative reward for taking action A in the current state S. qLet θ be the current Q-network parameters. E[·] is the expected value. R(S,A) is the immediate reward. q ' represents the target network parameters. γ=0.9 is the discount factor. max A' Q(S',A';θ q ') represents the maximum expected reward Q value among all possible actions A' in the next state S'.

[0090] DQN is trained using historical state-action-reward data and outputs the action A that maximizes the Q-value. opt The optimal subset of signatures, A, is output after processing. opt Example: A in a DDoS scenario opt =DDoS signature), E2 only loads A opt Detect and archive invalid signatures, such as SQL injection signatures, reducing resource consumption by 30%.

[0091] Step 4.3: GAN adversarial feedback loop: Construct a closed loop for generating simulated attacks → detecting unknown anomalies → updating signatures to improve the dual-engine capabilities and enhance detection capabilities through adversarial learning between E1 and E2.

[0092] Step 4.3.1: GAN adversarial training.

[0093] E2 acts as the generator G, taking random noise z (such as Gaussian noise) and attack statistics stat as input. attack (e.g., traffic statistics for DDoS attacks), output simulated attack vector h gen =G(z,stat attack ;θ g ), where z is random noise, stat attack For attack statistical characteristics, θ g The parameters are for the generator. E1 acts as the discriminator D, and the actual normal flow rate h is input. normal Real attack traffic h abnormal And the simulated attack traffic h generated by E2 gen Output the judgment probability D(h;θ) d The value ∈ [0,1] indicates the degree to which it considers the input traffic to be anomalous (attack), with a value close to 1 indicating authenticity and a value close to 0 indicating simulation.

[0094] Formula 19: L g =-E[logD(h gen In Formula 19: L g Let E[·] be the generator loss value, E[·] be the expected value, and D(·) be the output probability of the discriminator. gen This is the simulated attack vector output by the generator.

[0095] Formula 20: Ld =-E[logD(h abnormal )]-E[log(1-D(h normal ))]-E[log(1-D(h gen ))).

[0096] In Formula 20: L d Let E[·] be the discriminator loss value, E[·] be the expected value, and D(·) be the output probability of the discriminator. abnormal This represents the actual attack vector. normal This is a normal vector. gen This is a simulated attack vector.

[0097] The generator expects the discriminator output to be close to 0 (considering the generated data as fake), therefore the loss value L... g The smaller the value of θ, the more successfully the generator can fool the discriminator. θ is updated using gradient descent. g This makes the generated attack traffic more realistic. The discriminator aims to output 1 for real traffic and 0 for simulated traffic; the smaller the loss value, the stronger the discrimination ability. θ is updated through gradient descent. d This enhances D's ability to distinguish between reality and simulation. Through adversarial training, D's discriminatory ability continuously improves, ultimately enabling it to accurately identify the optimized h. gen ,D(h gen The value is approximately 0.85, which is close to a real attack.

[0098] Step 4.3.2: Unknown anomaly detection and signature update.

[0099] Through repeated adversarial training, E1's discriminative capabilities have been greatly enhanced, enabling it to detect some previously unseen and strange patterns. These patterns resemble neither known attacks, nor normal traffic, nor the simulated attacks that E2 can currently generate. These are the unknown anomalies. unknown .

[0100] Unknown abnormal traffic h unknown Two conditions must be met simultaneously, as determined by E1: High probability of anomaly: D(h) unknown () > 0.9; No matching signature: h unknown The signature matching module for E2 did not find any existing signatures.

[0101] When h is detected unknown Then, the process enters the feedback phase: h unknown Input E2. E2 analyzes h. unknown The feature pattern is used to create a new signature rule s through the signature generation module SigGen (which can be a feature extractor + rule generator). new .

[0102] Formula 21: s new =SigGen(h unknown ;θ sig,new In Formula 21: h unknown This represents the vector of unknown anomalies detected by E1. SigGen is the CNN feature extractor. θ sig,new These are the generated parameters after training.

[0103] New signature new By adding it to E2's signature library, this unknown attack becomes a known attack, enabling the model to identify similar unknown attacks.

[0104] Step 4.4: Engine Parameter Update: Integrate the optimization signals from DQN and GAN, and update the E1 and E2 parameters. The formula for updating the E1 parameter is as follows: Formula 22: θ enc1 =θ enc0 -η·▽θ enc L d In Formula 22: θ enc1 For the updated E1 encoder. θ enc0 These are the E1 encoder parameters before the update. η is the learning rate. ▽θ enc L d For loss function L d Relative to θ enc The gradient.

[0105] Formula 23: θ dec1 =θ dec0 -η·▽θ dec L d In Formula 23: θ dec1 These are the updated E1 decoder parameters. θ dec0 These are the E1 decoder parameters before the update.

[0106] E2 parameter updated to formula 24: θ cnn1 =θ cnn0 -η·▽θ cnn (L CNN +λ·L g In Formula 24: θ cnn1 These are the updated E2 parameters. θ cnn0 These are the E2 parameters before the update, including the weights of the generator and classifier. η is the learning rate. ▽θ cnn This represents the gradient with respect to the E2 parameter. CNN For classification loss, such as cross-entropy loss, it measures the accuracy of E2 in classifying known attacks. g The generator loss, such as negative log-likelihood loss, measures the realism of the simulated attack generated by E2. λ=0.5 is the balancing coefficient, which balances the classification loss L. CNNWith generation loss L g .

[0107] After processing, the optimized E1 and E2 are output, forming a detection → optimization → re-detection loop.

[0108] Step 5: Cloud-Edge Collaborative Resource Scheduling: The dual engines are treated as game participants to construct a game model. The particle swarm optimization algorithm is used to optimize the resource allocation of the dual engines. Based on the GPU acceleration ratio, the cloud-edge tasks that need to be executed collaboratively between the edge nodes and the cloud are divided into lightweight tasks and computationally intensive tasks, and cloud-edge collaborative offloading is performed. At the same time, the resource priority of the dual engines is adjusted according to the attack severity based on the deep Q network.

[0109] Furthermore, the game model is constrained by the sum of the CPU percentages of the two engines being ≤0.5, reserving 50% of CPU resources for other system tasks. The payoff function of the game model is the weighted sum of detection accuracy and resource consumption. The particle swarm optimization algorithm aims to maximize the total payoff of the two engines and converges to the Nash equilibrium point through iterative updates of particle velocity and position.

[0110] Furthermore, lightweight tasks are executed at the edge, while computationally intensive tasks are executed in the cloud. The method for offloading cloud-edge tasks is as follows: lightweight tasks such as traffic collection, feature encoding, and real-time detection are processed on the edge side, and only the input data of intensive tasks is uploaded to the cloud; intensive tasks such as knowledge graph updates and global parameter synchronization are processed on the cloud side using GPUs, and only the processing results are sent to the edge, reducing bandwidth consumption and latency.

[0111] The cloud-edge collaborative resource scheduling workflow is as follows: Step 5.1: Game modeling: Treat E1 and E2 as game participants and define the payoff function.

[0112] Participants: Anomaly detection engine E1: Policy space r1∈[0,0.5] (CPU percentage).

[0113] Attack identification engine E2: policy space r2∈[0,0.5] (CPU percentage).

[0114] The E1 revenue function is Equation 25: U1(r1,r2) = α1·Acc1(r1) - β1·r1. In Equation 25: U1(r1,r2) is the revenue value of E1. r1 is the CPU usage percentage of E1. Acc1(r1) is the detection accuracy of E1 with resource r1. α1 = 1.0 is the accuracy weight. β1 = 0.5 is the resource consumption weight.

[0115] The E2 revenue function is Equation 26: U2(r1,r2) = α2·Acc2(r2) - β2·r2. In Equation 26: U2(r1,r2) is the revenue value of E2. r2 is the CPU usage percentage of E2. Acc2(r2) is the detection accuracy of E2 with resource r2. α2 = 1.0 is the accuracy weight. β2 = 0.5 is the resource consumption weight.

[0116] Constraints: r1 + r2 ≤ 0.5, 50% CPU is reserved for other system tasks. After processing, the game model {U1, U2, r1, r2} is obtained, and the goal is to find the Nash equilibrium point where "changing r of any engine cannot improve its own reward".

[0117] Step 5.2: Nash Equilibrium Solution: Solve for the optimal resource allocation using the Particle Swarm Optimization (PSO-Game) algorithm. Particle definition: Each particle's position x i =[r1,r2] (resource allocation ratio), speed v i (Strategy adjustment direction).

[0118] Speed ​​updated to Formula 27: v i (t+1)=ω·v i (t)+c1·r1·(p i -x i (t))+c2·r2·(p g -x i (t)).

[0119] In Formula 27: v i (t+1) represents the velocity of particle i at time t+1. ω=0.7 is the inertial weight. c1=c2=1.49 are the learning factors. p i Let p be the optimal position for particle i. g This is the globally optimal position. i (t) represents the position of particle i at time t.

[0120] The position is updated to formula 28: x i (t+1)=x i (t)+v i (t+1). In Formula 28: x i (t+1) represents the position of particle i at time t+1.

[0121] The fitness function is Equation 29: fit(x i ) = U1(r1,r2) + U2(r1,r2). In formula 29: fit(x i x represents the fitness value of particle i. i Let be the position vector of particle i. Maximizing the total payoff corresponds to the Nash equilibrium point.

[0122] The particles move towards the high-yield region, and the solution is iterated until it converges to the Nash equilibrium point, yielding the optimal resource allocation strategy (r1). * r2 * ).

[0123] Step 5.3: Cloud-edge task unloading: Allocate computing resources according to task type.

[0124] Step 5.3.1: Task partitioning: By experimentally testing the GPU speedup ratio of different tasks, lightweight tasks and computationally intensive tasks are partitioned.

[0125] GPU acceleration formula 28: Speedup = T GPU / T CPU In Formula 28: T CPU T represents CPU processing time. GPU This refers to GPU processing time. Tasks with a Speedup < 2 are classified as lightweight tasks, while those with a Speedup ≥ 10 are classified as computationally intensive tasks. The specific task classifications are as follows: Lightweight task: Processing nodes are at the edge; processing includes: traffic collection, TrafficBERT / Sig2Vec encoding, and real-time E1 / E2 detection; low resource requirements; Computationally intensive tasks: Processing nodes are located in the cloud, and the tasks include: CMSKG updates, S... updated Global synchronization, GAT inference; high resource requirements.

[0126] Step 5.3.2: Perform cloud-edge unloading according to the division criteria.

[0127] Once the task division is determined, the core of cloud-edge offloading is to perform fast and lightweight real-time processing at the edge, while performing heavy and slow intensive computing in the cloud. The offloading process must rely on the speedup ratio to ensure effectiveness.

[0128] Edge-side offloading action: Only the input data of intensive tasks (such as 1,000 newly collected traffic feature vectors at the edge, about 768KB) are uploaded to the cloud, without transmitting the original traffic, to avoid bandwidth waste; Cloud-side offloading: The GPU is used to quickly process intensive tasks, such as updating IncGNN parameters in 0.3 seconds, and only the processing results (such as the updated GNN parameters θ) are loaded. GNN,global The data (approximately 10MB) is sent to the edge without transmitting intermediate data, further reducing latency.

[0129] If the acceleration ratio of cloud GPUs is insufficient, the processing time for intensive tasks will increase from 0.3s to 1.5s, resulting in excessively long wait times at the edge and disrupting real-time target detection. Therefore, the acceleration ratio is crucial for the system to maintain low latency after uninstallation.

[0130] After completing their respective tasks at the edge and in the cloud, the task execution results are output, which are divided into two categories: edge-side results and cloud-side results.

[0131] Edge-side task execution results: After the lightweight task is processed at the edge, the output is the real-time traffic detection results, including: Traffic tags: the detection conclusion for each traffic, such as normal, known attack - DDoS, unknown anomaly; Basic information: the source IP, destination IP, port number, and collection timestamp of the traffic; Preliminary association: the CMSKG node and initial edge weight corresponding to the traffic, such as traffic A being associated with the DDoS signature node, with Ssemantic=0.72.

[0132] End-side task execution results: After processing intensive tasks in the cloud, the output is global data for system optimization, including: Synchronized data: Globally updated signature library S updated For example, expanding the signature library from 1000 to 1005 signatures; optimization parameters: the global GNN parameter θ updated by IncGNN. GNN,global .

[0133] Step 5.4: DQN Dynamic Priority Adjustment: Adjust the resource priorities of E1 and E2 according to the severity of the attack.

[0134] Define states and actions: State S´´=[A type A severity A severity Attack severity: 0 = low, 1 = medium, 2 = high.

[0135] Action A´´={A E1 A E2}, A E1 =E1 takes precedence, A E2 =E2 takes precedence.

[0136] Reward R´´=Acc total (A´´)-Res delay (A´´), Acc total For the overall accuracy of the dual engines, Res delay This is due to resource delays and losses.

[0137] The DQN processing method is the same as in step 4.2. The DQN model is trained using historical data. DQN selects the action with the highest reward and adjusts the resource allocation accordingly. Example: During a DDoS attack, the CPU usage of E2 increases from 30% to 35%, and the detection latency decreases from 0.8 seconds to 0.5 seconds.

[0138] Step 6: Output Results: Trace nodes along the high-weight edges of the knowledge graph to construct a complete attack chain, prioritize alarms according to edge weights and attack severity and output alarm information, and output a resource allocation report based on resource utilization and task execution time.

[0139] Based on the high-weight edge, trace the attack chain and output hierarchical alarm information and resource allocation report.

[0140] The workflow for outputting the results is as follows: Step 6.1: Attack Chain Inference Result: Trace nodes along high-weight edges to construct the complete attack chain. Processing Result: Attack Chain Report, Example: Port Scan, S Node, S final =0.82 → Data transmission, S node, S final =0.78 → Ransomware, T node, S final =0.91, marking the causal relationship and time sequence window for each link.

[0141] Step 6.2: Alarm Information Output: Based on S final and A severity Prioritize alarms.

[0142] Hierarchy: High Priority: S final ≥0.9 and A severity =2, such as a ransomware attack, triggering SMS and email alerts; Medium priority: 0.7≤S final <0.9 and A severity =1, if SQL injection occurs, trigger email alert; low priority: S final <0.7 and A severity =0, for suspicious port connections, only log entries will be recorded. Processing result: Tiered alert information, including attack type, time of occurrence, affected IPs, and recommended remedial measures.

[0143] Step 6.3: Resource Allocation Report: Statistics on resource utilization rate and task execution time.

[0144] Processing results: Resource report, example: E1 CPU utilization 25%, E2 CPU utilization 30%, cloud GAT inference time 0.3s, edge real-time detection time 0.5s, resource utilization 85.3%, providing a basis for subsequent resource optimization.

[0145] Using the traffic of a core router in a campus network as the monitoring object, the target detection included DDoS attacks and SQL injection attacks. DDoS attacks are characterized by SYNFlood, while SQL injection attacks are marked by the "UNIONSELECT" signature. The time window was 1 second, and a total of 10 seconds of raw traffic data were collected for explanation.

[0146] Traffic was collected from the core router of the campus network using the tcpdump tool at a sampling frequency of 1 second, resulting in raw traffic data for 10 time windows, including data packets, session records, and device logs. For each 1-second window, three statistical features were extracted: number of packets per second, average packet length, and retransmission rate. For example, in the 5th second window, the data was: pkg / s = 1200, average packet length = 512 bytes, and retransmission rate = 3%. Attack signatures were extracted from the traffic using regular expression matching: DDoS: The number of SYN packets in this window was 950, out of a total of 1000 packets, resulting in a SYN packet percentage of 95%, exceeding the 90% threshold, matching a DDoS signature; SQL Injection: The traffic contained the "UNIONSELECT" signature, matching an SQL injection signature. The mean μ and standard deviation σ of the training set statistical features were assumed to be: μ = 800, σ = 200; average packet length: μ = 400, σ = 100; retransmission rate: μ = 1, σ = 0.5. Standardized calculation of statistical characteristics for the 5th second window: Standardized pkg / s = (1200-800) / 200 = 2, Standardized average packet length = (512-400) / 100 = 1.12, Standardized retransmission rate = (3-1) / 0.5 = 4.

[0147] The standardized statistical feature vector [2, 1, 12, 4] is input into the TrafficBERT model pre-trained with normal traffic to obtain the statistical feature semantic vector h. stat =[0.8,0.2,-0.5,…]. The DDoS signature, after Sig2Vec encoding, yields a semantic vector: h sig,ddos =[0.7,0.3,-0.4,…]. After SQL injection signature encoding, we get: h sig,sql =[0.6,0.4,-0.3,…].

[0148] Calculate h stat with h sig,ddos The cosine similarity is 0.989, which is greater than the threshold of 0.3. Therefore, in the knowledge graph, it is h. stat with h sig,ddos Construct the edges. Let the weight matrix W be a 3×3 matrix [[0.1,0,0],[0,0.1,0],[0,0,0.1]], and the attention vector a=[1,1,1].

[0149] For h stat and h sig,ddos The first 3 dimensions are calculated as follows: Wh stat =[0.8×0.1,0.2×0.1,-0.5×0.1]=[0.08,0.02,-0.05]; Wh sig,ddos=[0.7×0.1,0.3×0.1,-0.4×0.1]=[0.07,0.03,-0.04]; The concatenated vector is [0.08, 0.02, -0.05, 0.07, 0.03, -0.04]; e ij =LeakyReLU(a T ·{concatenated vector})=0.09.

[0150] If h stat Only with h sig,ddos If they are neighbors, then α ij =1, that is, semantic similarity weight S semantic =1.

[0151] Taking the SYN packet percentage (X) → total traffic surge (Y) as an example: Let Z be the number of normal HTTP requests. Through mutual information calculation, we find that the actual data stat(X,Y|Z) = 0.8. In the replaced data, the number of times stat(X',Y'|Z) ≥ 0.8 is 20 times, and the total number of replacements is 1000. Therefore, the p-value = 20 / 1000 = 0.02 < 0.05, which is statistically significant. We determine that X is the cause of Y.

[0152] Transitive entropy calculation: TE(X→Y)=H(Y(t+1)|Y(t))-H(Y(t+1)|Y(t),X(t))=0.6-0.2=0.4.

[0153] Normalized transitive entropy, assuming TE{text{max}}=0.5: TE norm =0.4 / 0.5=0.8.

[0154] Causal reliability S causal =(1-p(X→Y))·TE norm =(1-0.02)×0.8=0.784.

[0155] Final edge weight fusion, α=0.4, β=0.6: S final =0.4×1+0.6×0.784=0.8704.

[0156] DQN dynamically selects a subset of signatures: State S: Attack type, traffic intensity, CPU utilization; Action A: Select a subset of DDoS signatures; Reward function, λ1=0.9, λ2=0.1: R=0.9×Acc-0.1×Res. Assuming that after selecting this subset, the detection accuracy Acc=0.95 and the resource consumption Res=0.4, then: R=0.9×0.95-0.1×0.4=0.815.

[0157] GAN Adversarial Training: Generator: Input random noise z and DDoS statistical features to generate a simulated attack vector h. gen Discriminator: Distinguishes between real DDoS vectors h abnormal with h gen .

[0158] Assume D(h) gen If )=0.3, then the generator loss L g ≈1.204; Assume D(h) abnormal )=0.9, D(h normal If )=0.1, then the discriminator loss L d ≈0.567.

[0159] Model parameter update, learning rate η=0.001: Encoder parameter update for E1: θ enc1 =θ enc0 -η·▽θ enc L d ≈θ enc0 -0.005. Game Theory Model and Nash Equilibrium: Let the payoff functions of E1 and E2 be: U1(r1,r2)=1.0×Acc1(r1)-0.5×r1, U2(r1,r2)=1.0×Acc2(r2)-0.5×r2, where r1 is the CPU percentage of E1 and r2 is the CPU percentage of E2, with the constraint r1+r2≤0.5. Initial Resource Allocation: r1=0.2, Acc1=0.9; r2=0.3, Acc2=0.92, then: U1=1.0×0.9-0.5×0.2=0.8, U2=1.0×0.92-0.5×0.3=0.77. Particle Swarm Optimization Iteration: Through particle velocity and position updates, it eventually converges to the Nash equilibrium point r1. * =0.25, r2 * =0.25, at this point: U1=1.0×0.92-0.5×0.25=0.795, U2=1.0×0.93-0.5×0.25=0.805. The total profit is maximized, which is the optimal resource allocation strategy.

[0160] Attack Chain and Alerts: A DDoS attack chain is constructed along the edge with a weight of 0.8704 in the knowledge graph, combined with causal relationships. Because the edge weight 0.8704 ≥ 0.7 and the DDoS attack severity is high, a high-priority alert is triggered.

[0161] Resource Reports: E1: CPU usage 25%, resource utilization 80%. E2: CPU usage 25%, resource utilization 85%; Cloud graph inference time: 0.5 seconds / attempt; Edge real-time detection time: 0.1 seconds / attempt. Conclusion: Through the above steps, this method successfully detected DDoS attacks in the campus network, completed feature semantic alignment, causal enhanced inference, intelligent engine optimization and resource scheduling, and output high-priority alarms and resource reports, verifying the effectiveness of the method.

[0162] Based on the same inventive concept as the deep learning-based abnormal traffic detection and attack identification method provided in the embodiments of this application, the embodiments of this application also provide a deep learning-based abnormal traffic detection and attack identification system. If there is anything unclear about the content of the system embodiment, please refer to the corresponding content in the method embodiment.

[0163] Example 2: A deep learning-based system for anomaly traffic detection and attack identification includes: The data acquisition module collects raw traffic data from network devices in real time; The preprocessing module cleans, normalizes, and formats the raw traffic data to generate a structured dataset. The feature extraction module extracts statistical features and signature features in parallel from the structured dataset. The statistical features are used to calculate traffic distribution indicators through time-series windows, and the signature features are used to generate behavioral fingerprints based on the attack rule base. At the same time, the statistical features are standardized to eliminate the influence of units, and the standardized statistical features and signature features are output. The cross-modal semantic alignment module deploys the Traffic Bidirectional Encoder Representation Model (TraffBERT) and the Signature Feature Vector Embedding Model (Sig2Vec). The former encodes standardized statistical features into statistical feature semantic vectors, and the latter encodes signature features into signature feature semantic vectors, so that the two types of vectors are in the same 768-dimensional semantic space to eliminate the semantic gap. The knowledge graph construction module uses two types of semantic vectors as graph nodes, calculates the semantic similarity between nodes using the k-nearest neighbor algorithm, and selects high-similarity nodes to construct the initial knowledge graph. The causal reinforcement reasoning module uses graph attention networks to calculate the semantic association weights between nodes, and uses temporal causal discovery algorithms and transitive entropy to analyze the causal relationships and strengths between node pairs. It integrates semantic and causal information to update the edge weights of the knowledge graph, and realizes the dynamic evolution of the graph through incremental learning and federated learning mechanisms. The intelligent engine optimization module includes an anomaly detection engine based on unsupervised learning and an attack recognition engine based on supervised learning. It improves efficiency by dynamically scheduling a subset of attack signatures through reinforcement learning, and constructs an adversarial feedback loop through generative adversarial networks to generate simulated attacks, detect unknown anomalies, and update the signature database, thereby achieving the collaborative evolution of the two engines. The cloud-edge collaborative resource scheduling module models the system resource allocation as a Nash equilibrium problem through a game theory model, and uses optimization algorithms to solve the optimal resource allocation strategy. At the same time, it performs intelligent task offloading and result synchronization between edge nodes and the cloud based on the computational intensity of the task. The results output module performs attack chain reasoning based on the final knowledge graph and generates a visualization report. It outputs graded alarm information according to the association weight and attack severity level, and generates a system resource utilization report.

[0164] Furthermore, the feature extraction module uses the sliding window statistical method to extract the number of data packets per second, average packet length, retransmission rate, average connection duration, and connection establishment success rate as statistical features; and extracts attack signature codes, abnormal ports, and abnormal protocols as signature features from the public attack rule base through regular expression matching.

[0165] The cross-modal semantic alignment module uses the TrafficBERT model to encode statistical features and the Sig2Vec model to embed signature features. The parameters of the TrafficBERT model are obtained through pre-training on normal traffic, and the parameters of the Sig2Vec model are obtained through training on the Snort rule base.

[0166] The knowledge graph construction module uses cosine similarity to calculate the semantic similarity between nodes. During initialization, it first constructs a borderless graph, and then selects the node with the highest similarity for each node to construct an edge, thus forming the initial knowledge graph.

[0167] The causal enhanced association reasoning module uses the PCMCI algorithm to perform conditional independence tests and significance calculations to discover reliable causal relationships, and uses transitive entropy to quantify causal strength. Finally, the semantic similarity weights and causal credibility scores are weighted and fused into the final edge weights.

[0168] In the intelligent engine optimization module, the anomaly detection engine adopts a variational autoencoder model, and the attack recognition engine adopts a convolutional neural network model; reinforcement learning uses a deep Q-network to select the optimal signature subset with detection accuracy and resource consumption as reward functions; the generative adversarial network uses the attack recognition engine as the generator and the anomaly detection engine as the discriminator.

[0169] The cloud-edge collaborative resource scheduling module uses the particle swarm optimization algorithm to solve for the Nash equilibrium point to obtain the optimal CPU resource ratio allocated to the dual engines. Based on the GPU acceleration ratio, traffic collection, feature encoding and real-time detection tasks are deployed on the edge side, while knowledge graph updates, global signature library synchronization and complex graph reasoning tasks are offloaded to the cloud.

[0170] The output module iterates through the edges in the knowledge with weights higher than a set threshold to reconstruct the attack chain, and classifies the attack chain into three alarm priorities: high, medium, and low, based on the causal strength and attack type label.

[0171] The system's abnormal traffic detection and attack identification process begins with data acquisition. Each module achieves full-process detection and identification through an orderly data flow, as follows: The data acquisition module collects raw traffic data in real time from network devices such as routers and switches, and transmits it to the preprocessing module. The preprocessing module receives the raw traffic data, cleans, normalizes, and formats it to generate a structured dataset, which is then transmitted to the feature extraction module. The feature extraction module receives the structured dataset and uses a sliding window statistical method to extract statistical features such as the number of packets per second, average packet length, and retransmission rate in parallel. Combined with a publicly available attack rule base, it extracts signature features such as attack signature codes, abnormal ports, and abnormal protocols through regular expression matching. After feature standardization, the standardized statistical features and signature features are transmitted to the cross-modal semantic alignment module. This module receives both types of features, encodes the standardized statistical features into statistical feature semantic vectors using a TrafficBERT model pre-trained with normal traffic, and encodes the signature features into signature feature semantic vectors using a Sig2Vec model trained with the Snort rule base (both types of vectors reside in the same 768-dimensional semantic space to eliminate semantic gaps). These two semantic vectors are then transmitted to the knowledge graph construction module. The knowledge graph construction module receives the semantic vectors, uses them as nodes, calculates the semantic similarity between nodes using cosine similarity, selects high-similarity nodes to construct the initial knowledge graph, and then... The initial knowledge graph is transmitted to the causal reinforcement reasoning module. This module receives the initial knowledge graph, calculates the semantic association weights of nodes using a graph attention network, performs conditional independence checks and significance calculations using the PCMCI algorithm to discover reliable causal relationships, quantifies causal strength using transitive entropy, and updates graph edge weights by fusing semantic similarity and causal credibility. It then achieves dynamic graph evolution through incremental learning and federated learning, transmitting the final knowledge graph to the intelligent engine optimization module and the result output module, respectively. The intelligent engine optimization module receives the final knowledge graph and uses it to optimize the anomaly detection engine (composed of a variational autoencoder) and the attack recognition engine (composed of a convolutional neural network). Simultaneously, it dynamically schedules attack signatures using a deep Q-network. The subset, with the help of generative adversarial networks, constructs a feedback loop of "generating simulated attacks - detecting unknown anomalies - updating the signature library", and transmits the resource requirements data of the dual engines to the cloud-edge collaborative resource scheduling module. The cloud-edge collaborative resource scheduling module receives the resource requirements of the intelligent engine optimization module and the task computing intensity data of each module, solves the Nash equilibrium through the particle swarm optimization algorithm to determine the optimal CPU resource allocation strategy, and issues resource allocation instructions to each module. At the same time, based on the GPU acceleration ratio, lightweight tasks such as traffic collection, feature encoding, and real-time detection are deployed on edge nodes, while computationally intensive tasks such as knowledge graph updates, global signature library synchronization, and complex graph reasoning are offloaded to the cloud. The cloud processing results are transmitted back to the edge nodes to achieve data synchronization.The output module receives the final knowledge graph from the causal enhanced association reasoning module, reconstructs the attack chain by traversing edges with weights exceeding a set threshold, and classifies the attack chain into high, medium, and low alert levels based on attack type tags. Simultaneously, it receives system resource utilization data from the cloud-edge collaborative resource scheduling module, generating a visualized attack chain report, tiered alert information, and a resource utilization report, thus completing the entire abnormal traffic detection and attack identification process.

[0172] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A deep learning-based method for anomaly traffic detection and attack identification, characterized in that, Includes the following steps: Raw traffic data is collected from network devices, and statistical features and signature features are extracted in parallel. The statistical features are then standardized to obtain standardized statistical features. Standardized statistical features and signature features are encoded into a set of vectors in a unified semantic space to construct an initial knowledge graph; A graph attention network is used to calculate the attention weights between nodes in the initial knowledge graph and to select high-similarity candidate node pairs. The temporal causal relationship is distinguished between the causal and correlation relationships of the candidate node pairs through the temporal causal relationship algorithm. The causal strength is quantified by combining the propagation entropy. The final edge weights are obtained by fusing semantic similarity and causal credibility. The knowledge graph is dynamically updated through incremental learning and federated learning. We construct and optimize an anomaly detection engine and an attack identification engine, utilize reinforcement learning to dynamically schedule signature subsets, and achieve the collaborative evolution of the two engines through generative adversarial networks. A game model is built based on dual engines. The particle swarm game optimization algorithm is used to optimize the resource allocation of dual engines. According to the GPU acceleration ratio, the cloud-edge tasks that need to be executed collaboratively between edge nodes and the cloud are divided into lightweight tasks and computationally intensive tasks. Cloud-edge collaborative offloading is performed. Based on the deep Q network, the resource priority of dual engines is adjusted according to the attack severity. Based on the high-weight edge, trace the attack chain and output hierarchical alarm information and resource allocation report.

2. The method according to claim 1, characterized in that, The statistical feature extraction adopts the sliding window method, with the window size consistent with the sampling frequency, to extract the number of data packets per second, average packet length, retransmission rate, connection duration, and connection establishment success rate. The signature feature extraction uses regular expression matching and ngram methods to obtain attack signature codes, abnormal ports and abnormal protocols from a public attack rule base. The standardization process is based on Z-score normalization using the mean and standard deviation of the training set.

3. The method according to claim 1, characterized in that, The statistical feature encoding uses the TrafficBERT model to output a 768-dimensional semantic vector, and the signature feature encoding uses the Sig2Vec model to map discrete signatures into continuous vectors that are in the same space as the statistical feature semantic vectors. The initial knowledge graph is constructed using the k-nearest neighbor algorithm, and the semantic similarity of nodes is calculated using cosine similarity. The nodes with the highest similarity are selected to construct the initial edge with the current node.

4. The method according to claim 1, characterized in that, The temporal causal discovery algorithm filters potential causal relationships through conditional independence tests and determines reliable causal pairs by combining statistical significance p-values. The transit entropy is normalized and then fused with statistical significance weights to calculate causal credibility. Finally, the edge weights are obtained by weighted fusion of semantic similarity and causal credibility, and the weight coefficients are optimized by minimizing the inference error of the attack chain. The incremental learning uses an incremental graph neural network to update graph parameters, while federated learning is achieved by updating parameters locally at edge nodes and aggregating global parameters in the cloud, thus protecting the data privacy of edge nodes while adapting to new attack patterns.

5. The method according to claim 1, characterized in that, The anomaly detection engine uses a variational autoencoder, and the attack identification engine uses a convolutional neural network. The reinforcement learning signature scheduling uses a deep Q-network with detection accuracy and resource consumption as reward functions. The deep Q-network models signature selection as a Markov decision process, with states including attack type, traffic intensity, and CPU utilization, and actions including selecting a subset of signatures with different attack types. The reward function prioritizes detection accuracy and outputs the optimal subset of signatures with the largest Q value.

6. The method according to claim 5, characterized in that, In the generative adversarial network, the attack identification engine acts as the generator, the anomaly detection engine acts as the discriminator, and unknown anomalies trigger the generation of new signatures. The generative adversarial network constructs a feedback loop of generating simulated attacks, detecting unknown anomalies, and updating signatures. The method is as follows: the attack identification engine acts as a generator, inputting random noise and attack statistical features to generate simulated attack vectors; the anomaly detection engine acts as a discriminator to distinguish between real traffic and simulated attack vectors; adversarial training improves the discriminator's ability to detect unknown anomalies and the authenticity of the simulated attacks generated by the generator; after detecting unknown anomalies, a new signature is generated to update the signature library of the attack identification engine.

7. The method according to claim 1, characterized in that, The constraints of the game model are that the sum of the CPU ratios of the two engines is ≤0.5, and 50% of the CPU resources are reserved for other system tasks. The payoff function of the game model is the weighted sum of detection accuracy and resource consumption. The particle swarm optimization algorithm aims to maximize the total profit of the dual engines and converges to the Nash equilibrium point through iterative updates of particle velocity and position.

8. The method according to claim 7, characterized in that, The lightweight tasks are executed at the edge, while the computationally intensive tasks are executed in the cloud. The method for offloading cloud-edge tasks is as follows: lightweight tasks, including traffic collection, feature encoding, and real-time detection, are processed at the edge, and only the input data of the computationally intensive tasks is uploaded to the cloud; the computationally intensive tasks, such as knowledge graph updates and global parameter synchronization, are processed by GPUs at the cloud, and only the processing results are sent to the edge.

9. A deep learning-based abnormal traffic detection and attack identification system, characterized in that, include: The data acquisition module collects raw traffic data from network devices in real time; The preprocessing module cleans, normalizes, and formats the raw traffic data to generate a structured dataset. The feature extraction module extracts statistical features and signature features in parallel from the structured dataset. The statistical features are used to calculate traffic distribution indicators through time-series windows, and the signature features are used to generate behavioral fingerprints based on the attack rule base. At the same time, the statistical features are standardized to eliminate the influence of units, and the standardized statistical features and signature features are output. The cross-modal semantic alignment module deploys a bidirectional traffic encoder representation model and a signature feature vector embedding model. The former encodes standardized statistical features into statistical feature semantic vectors, and the latter encodes signature features into signature feature semantic vectors, so that the two types of vectors are in the same semantic space to eliminate the semantic gap. The knowledge graph construction module uses two types of semantic vectors as graph nodes, calculates the semantic similarity between nodes using the k-nearest neighbor algorithm, and selects high-similarity nodes to construct the initial knowledge graph. The causal reinforcement reasoning module uses graph attention networks to calculate the semantic association weights between nodes, and uses temporal causal discovery algorithms and transitive entropy to analyze the causal relationships and strengths between node pairs. It integrates semantic and causal information to update the edge weights of the knowledge graph, and realizes the dynamic evolution of the graph through incremental learning and federated learning mechanisms. The intelligent engine optimization module includes an anomaly detection engine based on unsupervised learning and an attack recognition engine based on supervised learning. It improves efficiency by dynamically scheduling a subset of attack signatures through reinforcement learning, and constructs an adversarial feedback loop through generative adversarial networks to generate simulated attacks, detect unknown anomalies, and update the signature database, thereby achieving the collaborative evolution of the two engines. The cloud-edge collaborative resource scheduling module models the system resource allocation as a Nash equilibrium problem through a game theory model, and uses optimization algorithms to solve the optimal resource allocation strategy. At the same time, it performs intelligent task offloading and result synchronization between edge nodes and the cloud based on the computational intensity of the task. The results output module performs attack chain reasoning based on the final knowledge graph and generates a visualization report. It outputs graded alarm information according to the association weight and attack severity level, and generates a system resource utilization report.

10. The system according to claim 9, characterized in that, The feature extraction module uses the sliding window statistical method to extract the number of data packets per second, average packet length, retransmission rate, average connection duration, and connection establishment success rate as statistical features. Furthermore, attack signature codes, abnormal ports, and abnormal protocols are extracted from a publicly available attack rule base using regular expression matching as signature features. The cross-modal semantic alignment module uses the TrafficBERT model to encode statistical features and the Sig2Vec model to embed signature features. The parameters of the TrafficBERT model are obtained through pre-training on normal traffic, and the parameters of the Sig2Vec model are obtained through training on the Snort rule base. The knowledge graph construction module uses cosine similarity to calculate the semantic similarity between nodes. During initialization, it first constructs a borderless graph, and then selects the node with the highest similarity for each node to construct an edge, thus forming an initial knowledge graph. The causal enhanced association reasoning module uses the PCMCI algorithm to perform conditional independence tests and significance calculations to discover reliable causal relationships, and uses transitive entropy to quantify causal strength. Finally, it weights and fuses semantic similarity weights with causal credibility scores into the final edge weights. In the intelligent engine optimization module, the anomaly detection engine adopts a variational autoencoder model, and the attack identification engine adopts a convolutional neural network model. The reinforcement learning uses a deep Q-network with detection accuracy and resource consumption as reward functions to select the optimal signature subset; the generative adversarial network uses an attack identification engine as a generator and an anomaly detection engine as a discriminator. The cloud-edge collaborative resource scheduling module uses the particle swarm optimization algorithm to solve the Nash equilibrium point to obtain the optimal CPU resource ratio allocated to the dual engines, and deploys traffic collection, feature encoding and real-time detection tasks on the edge side according to the GPU acceleration ratio, while offloading knowledge graph update, global signature library synchronization and complex graph reasoning tasks to the cloud. The output module iterates through the edges in the knowledge with weights higher than a set threshold to reconstruct the attack chain, and classifies the attack chain into three alarm priorities: high, medium, and low, based on the causal strength and attack type label of the attack chain.

Images