Deep learning-based abnormal traffic detection and attack identification method and system

Abstract

The application belongs to the technical field of network security, and provides an abnormal traffic detection and attack identification method and system based on deep learning, which comprises data preprocessing and feature extraction, cross-modal semantic alignment and knowledge graph construction, causal enhancement correlation reasoning, intelligent engine optimization, cloud-edge collaborative resource scheduling and result output; the method maps statistical features and signature features to a unified semantic space through a cross-modal semantic alignment and knowledge graph construction module, breaks the semantic barriers between heterogeneous features, introduces time series causal discovery and transfer entropy calculation, can distinguish simple correlation from reliable causality, improves the accuracy and reliability of attack chain reasoning, and reduces the false alarm rate; through the intelligent engine optimization and cloud-edge collaborative resource scheduling module, online self-evolution of the detection model and dynamic optimal allocation of system resources are realized, and the overall adaptability, robustness and practicality of the system are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and particularly relates to an abnormal traffic detection and attack identification method and system based on deep learning. BACKGROUND

[0002] Abnormal traffic detection and attack identification are core tasks of network security protection. Abnormal traffic detection refers to identifying abnormal data deviating from normal traffic patterns by analyzing traffic data transmitted in a network. Attack identification further matches attack features on the basis of abnormal detection to determine attack types.

[0003] Existing abnormal traffic detection and attack identification technologies mainly fall into three categories: signature-based detection, statistical-based abnormal detection technology, and traditional deep learning-based detection.

[0004] However, the existing technology has some defects, which are as follows: The existing method usually processes numerical statistical features and symbolic signature features in parallel or independently. These two feature modalities are quite different: statistical features are continuous numerical vectors, while signature features are discrete symbolic identifiers. Due to the lack of cross-modal semantic alignment mechanism, these two features are mapped into completely different semantic spaces, forming a semantic gap. This fundamental heterogeneity prevents the system from effectively associating traffic anomalies with specific attack behaviors, and only allows isolated judgment, making it difficult to build a complete attack chain view and causing false positives and false negatives. The existing methods based on knowledge graphs or association rules have static preset or simple co-occurrence statistics for the association relationships they build. They lack the ability to analyze causal relationships and temporal evolution, and cannot distinguish between simple correlation and reliable causality between features, for example, they cannot determine whether port scanning caused data exfiltration or whether the two occurred simultaneously by chance. At the same time, their knowledge system cannot be updated online, and lacks incremental learning and federated learning mechanisms to integrate new attack patterns emerging on edge nodes. This causes the knowledge graph to quickly become outdated and the system to have serious lack of adaptive ability when the attacker fine-tunes attack methods or launches new zero-day attacks. SUMMARY

[0005] The purpose of the embodiments of the present application is to propose an abnormal traffic detection and attack identification method and system based on deep learning to solve at least one technical problem proposed in the background art.

[0006] To solve the above technical problems, the embodiments of the present application provide an abnormal traffic detection and attack identification method based on deep learning, which adopts the technical solutions as follows:

[0007] The abnormal traffic detection and attack identification method based on deep learning comprises the following steps:

[0008] Raw traffic data is collected from network devices, statistical features and signature features are extracted in parallel, and the statistical features are standardized to obtain standardized statistical features; the standardized statistical features and the signature features are encoded into a vector set in a unified semantic space to construct an initial knowledge graph; a graph attention network is used to calculate the attention weights between nodes of the initial knowledge graph and to screen high-similarity candidate node pairs, a time-series causality algorithm is used to distinguish the causality and correlation of the candidate node pairs, the causality strength is quantified by using a transfer entropy, the final edge weight is obtained by fusing the semantic similarity and the causality reliability, and the knowledge graph is dynamically updated by using incremental learning and federated learning; an anomaly detection engine and an attack identification engine are constructed and optimized, a signature subset is dynamically scheduled by using reinforcement learning, and a generative adversarial network is used to realize the co-evolution of the two engines; a game model is constructed based on the two engines, a particle swarm game optimization algorithm is used to optimize the resource allocation of the two engines, the GPU acceleration ratio is used to divide light tasks and computationally intensive tasks, cloud-edge collaborative offloading is performed, and the resource priority of the two engines is adjusted based on a deep Q network according to the severity of the attack; the attack link is traced back according to the high-weight edge, and hierarchical alarm information and resource allocation reports are output.

[0009] Preferably, the statistical feature extraction uses a sliding window method, the window size is consistent with the sampling frequency, the number of data packets per second, the average packet length, the retransmission rate, the connection duration and the connection establishment success rate are extracted; the signature feature extraction obtains attack feature codes, abnormal ports and abnormal protocols from a public attack rule library through regular matching and ngram methods; the standardization processing is based on the mean and standard deviation of the training set for Z-score normalization.

[0010] Preferably, the statistical feature encoding uses a TraffBERT model to output 768-dimensional semantic vectors, the signature feature encoding uses a Sig2Vec model to map discrete signatures to continuous vectors and to the same space as the statistical feature semantic vectors; the initial knowledge graph is constructed by using a k-nearest neighbor algorithm, the cosine similarity is used to calculate the node semantic similarity, and the initial edges are constructed by selecting a number of nodes with the highest similarity and the current node.

[0011] Preferably, the time-series causality discovery algorithm screens potential causal relationships by conditional independence test, and determines reliable causal pairs by combining statistical significance p values; the transfer entropy is fused with the statistical significance weight after normalization to calculate the causality reliability, and the final edge weight is obtained by weighted fusion of the semantic similarity and the causality reliability, and the weight coefficient is optimized by minimizing the attack chain reasoning error; the incremental learning uses an incremental graph neural network to update the graph parameters, and the federated learning realizes the local parameter update of the edge node and the aggregation of the global parameters in the cloud to protect the data privacy of the edge node while adapting to new attack modes.

[0012] Preferably, the anomaly detection engine adopts a variational autoencoder, and the attack identification engine adopts a convolutional neural network; the reinforcement learning signature scheduling uses a deep Q network to detect the accuracy rate and resource consumption rate as a reward function, the deep Q network models the signature selection as a Markov decision process, the state includes attack type, traffic intensity, CPU usage, the action includes selecting a signature subset of different attack types, the reward function prioritizes the detection accuracy rate, and outputs the optimal signature subset with the maximum Q value.

[0013] Preferably, in the generative adversarial network, the attack identification engine acts as a generator, and the anomaly detection engine acts as a discriminator, and unknown anomalies trigger new signature generation; the generative adversarial network constructs a feedback loop of generating simulated attacks, detecting unknown anomalies, and updating signatures, and the method is as follows: the attack identification engine inputs random noise and attack statistical features to generate a simulated attack vector as a generator, the anomaly detection engine distinguishes real traffic from simulated attack vectors as a discriminator, and through adversarial training, the detection ability of the discriminator for unknown anomalies and the authenticity of the simulated attack generated by the generator are improved, and a new signature is generated to update the signature library of the attack identification engine after detecting unknown anomalies.

[0014] Preferably, the constraint condition of the game model is that the sum of the CPU occupancy ratios of the two engines is less than or equal to 0.5, and 50% of the CPU resources are reserved for other system tasks, and the reward function of the game model is the weighted sum of the detection accuracy rate and the resource consumption; the particle swarm game optimization algorithm takes the maximization of the total revenue of the two engines as the goal, and converges to the Nash equilibrium point through the iterative update of particle speed and position.

[0015] Preferably, the lightweight task is executed on the edge, and the compute-intensive task is executed in the cloud; the method of cloud-edge task offloading is as follows: the edge side processes lightweight tasks, the lightweight tasks include traffic collection, feature encoding, and real-time detection, and only the input data of intensive tasks are uploaded to the cloud; the cloud side uses GPU to process intensive tasks such as knowledge graph update and global parameter synchronization, and only the processing results are sent to the edge.

[0016] To solve the above technical problems, the embodiment of the application also provides a deep learning-based abnormal traffic detection and attack identification system, which comprises: a data acquisition module, which acquires original traffic data from network equipment in real time; a preprocessing module, which performs cleaning, normalization and formatting processing on the original traffic data to generate a structured data set; a feature extraction module, which extracts statistical features and signature features from the structured data set in parallel, the statistical features are calculated by a time window to calculate traffic distribution indicators, the signature features are generated based on an attack rule library to generate behavior fingerprints, and the statistical features are standardized to eliminate dimensional influence; a cross-modal semantic alignment module, which deploys a traffic bidirectional encoder representation model and a signature feature vector embedding model, the former encodes the standardized statistical features into statistical feature semantic vectors, and the latter encodes the signature features into signature feature semantic vectors, so that the two types of vectors are in the same semantic space to eliminate semantic gaps; a knowledge graph construction module, which takes the two types of semantic vectors as graph nodes, calculates the semantic similarity between nodes by a k-nearest neighbor algorithm, and filters high-similarity nodes to construct an initial knowledge graph; a causal enhancement correlation reasoning module, which calculates the semantic correlation weight between nodes by using a graph attention network, and uses a time series causal discovery algorithm and a transfer entropy to analyze the causal relationship and strength between nodes, fuses semantic and causal information to update the edge weight of the knowledge graph, and realizes dynamic evolution of the graph by using an incremental learning and a federated learning mechanism; an intelligent engine optimization module, which comprises an unsupervised learning-based anomaly detection engine and a supervised learning-based attack identification engine, dynamically schedules attack signature subsets by reinforcement learning to improve efficiency, and constructs an adversarial feedback closed loop by a generative adversarial network to generate simulated attacks, detect unknown anomalies and update the signature library, realizes the co-evolution of the double engines; a cloud-edge collaborative resource scheduling module, which models system resources as a Nash equilibrium problem by using a game theory model, and solves the optimal resource allocation strategy by using an optimization algorithm, and intelligently offloads tasks and synchronizes results between edge nodes and the cloud according to the calculation-intensive degree of the tasks; and a result output module, which performs attack chain reasoning based on the final knowledge graph and generates a visual report, outputs hierarchical alarm information according to the correlation weight and attack severity level, and generates a system resource utilization report.

[0017] Preferably, the feature extraction module uses a sliding window statistical method to extract the number of packets per second, the average packet length, the retransmission rate, the average connection duration and the connection establishment success rate as statistical features; and extracts attack feature codes, abnormal ports and abnormal protocols as signature features from a public attack rule library by using a regular matching method.

[0018] The cross-modal semantic alignment module encodes the statistical features using a TraffBERT model, and embeds the signature features using a Sig2Vec model, parameters of the TraffBERT model are obtained by normal traffic pre-training, and parameters of the Sig2Vec model are obtained by Snort rule library training;

[0019] The knowledge graph construction module calculates the semantic similarity between nodes using cosine similarity, initializes by first constructing an edgeless graph, and then constructs edges for each node by selecting the node with the highest similarity to form an initial knowledge graph.

[0020] The causal reinforcement correlation reasoning module uses the PCMCI algorithm for conditional independence test and significance calculation to discover reliable causal relationships, and uses transfer entropy to quantify causal strength, and finally weights and fuses the semantic similarity weight and the causal reliability score into the final edge weight.

[0021] In the intelligent engine optimization module, the anomaly detection engine adopts a variational autoencoder model, and the attack identification engine adopts a convolutional neural network model; the reinforcement learning adopts a deep Q network to detect accuracy and resource consumption as a reward function to select the optimal signature subset; the generative adversarial network uses the attack identification engine as the generator and the anomaly detection engine as the discriminator.

[0022] The cloud-edge collaborative resource scheduling module uses a particle swarm optimization algorithm to solve the Nash equilibrium point to obtain the optimal CPU resource ratio allocated to the double-engine, and according to the GPU acceleration ratio, the traffic collection, feature encoding and real-time detection tasks are deployed on the edge side, and the knowledge graph update, global signature library synchronization and complex graph reasoning tasks are offloaded to the cloud.

[0023] The result output module traverses the edges in the knowledge graph with a weight higher than a set threshold to reconstruct the attack chain, and divides the attack chain into high, medium and low alert priorities according to the causal strength and attack type label.

[0024] The beneficial effects of the present application are as follows:

[0025] The present application provides an abnormal traffic detection and attack identification method based on deep learning, which maps two types of heterogeneous features into a unified 768-dimensional semantic space through cross-modal semantic alignment of TraffBERT and Sig2Vec, so that in the DDoS attack detection scenario, the system can directly calculate the cosine similarity between the high-frequency SYN packet statistical feature vector and the SYN Flood signature feature vector, detect the high correlation between traffic anomalies and specific attacks in semantics, and significantly improve the accuracy and explainability of feature correlation, laying a solid foundation for subsequent construction of high-quality knowledge graphs.

[0026] The method introduces causal reinforcement correlation reasoning and dynamic graph updating mechanism, analyzes and quantifies the causal direction and strength between features from time series. In the advanced persistent threat detection scene, the system can effectively distinguish whether the port scanning leads to subsequent data export or both are just irrelevant background noise, thereby constructing a credible causal relationship consistent with the attack logic chain, greatly reducing the false positive rate and improving the credibility of threat research and judgment.

[0027] The method designs a double-engine cooperative optimization and cloud-edge game scheduling strategy, and a dynamic signature scheduling based on reinforcement learning, so that the system only loads the most relevant signature subset when facing a specific attack, and unloads irrelevant signatures. This reduces the computing resource consumption of edge devices by more than 30%, while ensuring the detection accuracy of key attacks, achieving optimal scheduling under limited resources. Through the GAN's feedback closed loop and federated learning mechanism, the self-evolution ability is constructed. When an unknown ransomware variant is first detected at a certain edge node, the GAN can generate its simulated signature, and after confirmation, automatically update the local and global signature library. At the same time, federated learning allows multiple edge nodes to aggregate model parameter increments without sharing raw data, and jointly train a more powerful detection model, protecting data privacy and greatly shortening the global response time to new threats. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the schemes in the present application, the following will briefly introduce the drawings needed in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0029] Figure 1 is the abnormal flow detection and attack identification flowchart provided by the embodiments of the present application;

[0030] Figure 2 is the double-engine cooperative and antagonistic optimization flowchart provided by the embodiments of the present application;

[0031] Figure 3 is an exemplary system architecture to which the present application can be applied. DETAILED DESCRIPTION

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting upon the application; the terms "comprising," "including," and "having," and variations thereof, as used in enrolling and claims herein, are intended to be open-ended and to mean including, but not limited to; the terms "first," "second," and the like, as used in the description herein, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. Unless otherwise indicated, the terms "one" and "the" and singular forms thereof, as used in the specification and claims herein, are intended to mean one or more. Unless otherwise indicated, the terms "plurality" and "a plurality" as used herein mean two or more.

[0033] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase that in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of other embodiments. It is expressly understood that the embodiments described herein are merely examples from a whole class of comparable embodiments which those skilled in the art will readily appreciate. It is further understood that the descriptions of various embodiments do not imply that the compositions are necessarily the only way to implement the methods and / or apparatuses.

[0034] In order to make the technical personnel in the art better understand the scheme of the application, the technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings.

[0035] Embodiment 1:

[0036] The deep learning-based abnormal traffic detection and attack identification method comprises steps 1-6. Among them:

[0037] Step 1: data preprocessing and feature extraction: collecting raw traffic data from network devices, extracting statistical features and signature features in parallel, and standardizing the statistical features to obtain standardized statistical features.

[0038] Further, the statistical feature extraction adopts a sliding window method, the window size is consistent with the sampling frequency, and the number of data packets per second, the average packet length, the retransmission rate, the connection duration and the connection establishment success rate are extracted; the signature feature extraction obtains attack feature codes, abnormal ports and abnormal protocols from the public attack rule library through regular matching and ngram method; the standardization processing is based on the mean and standard deviation of the training set for Z-score normalization.

[0039] The workflow of data preprocessing and feature extraction is as follows:

[0040] Step 1.1: Raw traffic data collection. Use network packet capture tools such as tcpdump, Zeek, or Flink stream processing framework to capture network raw transmission data from routers, switches, network probes, etc. devices, covering traffic session records (NetFlow), packet files (pcap), device logs, etc. formats, extracting basic information including source / destination IP, protocol, packet size, timestamp. The sampling frequency is set to 1 second / time, ensuring real-time performance. After collection, the raw traffic dataset is obtained.

[0041] Step 1.2: Dual-engine feature extraction: parallel extraction of statistical features and signature features.

[0042] Step 1.2.1: Statistical feature extraction: use sliding window statistics method, window size 1 second, consistent with sampling frequency, extract quantitative features reflecting traffic macro rules from raw traffic dataset, including packets per second, average packet length, connection establishment success rate, retransmission rate, connection duration; output statistical feature matrix X stat ∈R N×M , N is the number of samples, M is the dimension of statistical features.

[0043] Formula 1: X stat =[f pps ,f bytes ,f retrans ,f duration ,f success ]。

[0044] In formula 1: f pps is the number of packets per second, the total number of packets transmitted in a unit of time, unit: packets / second. f bytes is the average packet length, the average value of the number of bytes of all packets in a unit of time, unit: bytes. f retrans is the retransmission rate, the proportion of retransmitted packets to the total number of sent packets in a unit of time, unit: %. f duration is the average connection duration, the average value of the duration of all successful connections in a unit of time, unit: seconds. f success is the connection establishment success rate, the proportion of successfully established TCP / UDP connections to the total number of connection requests in a unit of time, unit: %.

[0045] Step 1.2.2: Signature feature extraction: obtain attack fingerprints from public attack rule library, such as SQL injection UNION SELECT, DDoS SYN flag; use regular matching and ngram extraction method to extract discrete features reflecting attack fingerprints from raw traffic dataset, including attack feature code, abnormal port, abnormal protocol. Output signature feature set X sig , K is the number of signatures.

[0046] Formula 2: X sig = [s code ,s port ,s proto ,...,s K ] Formula 2: S i ∈{0,1} s code is the attack signature identifier. s port is the abnormal port identifier. s proto is the abnormal protocol identifier. K is the number of signatures.

[0047] Step 1.3: Data standardization: using Z-score standardization to standardize statistical features, eliminate dimension influence, get standardized statistical features X norm .

[0048] Standardization uses formula 3: X norm = (X stat - μ train ) / σ train .

[0049] Formula 3: X stat is the statistical feature, μ train is the mean of the training set features. σ train is the standard deviation of the training set features.

[0050] Formula 4: X norm = [f norm,pps ,f norm,bytes ,f norm,retrans ,f norm,duration ,f norm,success ] Formula 4: f norm,pps ,f norm,bytes ,f norm,retrans ,f norm,duration ,f norm,success are the normalized components corresponding to f pps ,f bytes ,f retrans ,f duration ,f success after standardization.

[0051] Step 2: Cross-modal semantic alignment and knowledge graph construction: encode standardized statistical features and signature features into a vector set in a unified semantic space, and construct an initial knowledge graph.

[0052] Further, the statistical feature encoding uses the TraffBERT model to output a 768-dimensional semantic vector, and the signature feature encoding uses the Sig2Vec model to map discrete signatures to continuous vectors and to the same space as the statistical feature semantic vector. The initial knowledge graph is constructed by a k-nearest neighbor algorithm, and the cosine similarity is used to calculate the node semantic similarity. The nodes with the highest similarity are selected to construct the initial edge with the current node.

[0053] The cross-modal semantic alignment and knowledge graph construction workflow is as follows:

[0054] Step 2.1: Statistical feature encoding: use the traffic bidirectional encoder representation model (TraffBERT) to encode the statistical feature X norm into a vector.

[0055] TraffBERT is a traffic statistical feature encoding model based on the bidirectional encoder representation model (BERT). It maps numerical statistical features to a 768-dimensional unified semantic space.

[0056] Formula 5: h stat = TraffBERT (X norm ; θ stat ). In formula 5: X norm is the standardized statistical feature. θ stat is the TraffBERT model parameter, obtained by normal traffic pre-training. h stat is the statistical feature semantic vector, h stat ∈ R N×768 , N is the sampling number, and 768 is the semantic vector dimension.

[0057] Step 2.2: Signature feature embedding: use the signature feature vector embedding model (Sig2Vec) to encode the signature feature X sig into a continuous vector.

[0058] Formula 6: h sig = Sig2Vec (X sig ; θ sig ). In formula 6: X sig is the signature feature set. θ sig is the Sig2Vec model parameter, obtained by training the Snort rule library. h sig is the signature feature semantic vector, h sig ∈ R K×768 , K is the number of signatures, and 768 is the semantic vector dimension.

[0059] h sig and h stat are encoded into a unified semantic space, resulting in a set of vectors that share the same vector space: h sigWith h stat Both dimensions are 768, TraffBERT is pre-trained by normal traffic, and 768-dimensional learning of normal and abnormal traffic statistical semantics, Sig2Vec is trained by Snort rule library, and 768-dimensional learning of attack signature semantics, although the training data is different, but the target semantics is consistent, both around whether the traffic is abnormal, whether it contains attacks, finally realize the same dimension expression of the same semantics, eliminate the semantic gap.

[0060] Step 2.3: Initial knowledge graph (CMSKG) construction: take semantic vector as graph node, build initial edge through near neighbor relationship, form cross-modal semantic alignment knowledge graph.

[0061] Take h stat and h sig as nodes V, initialize the edgeless graph.

[0062] Using k-neighbor graphing algorithm (kNN), calculate the cosine similarity S i of any node V j ∈V with all other nodes V semantic ∈V.

[0063] Formula 7: S semantic (i,j)=(h i ·h j ) / (||h i ||2·||h j ||2). In formula 7: S semantic (i,j) is the semantic similarity score of node i and node j, the greater the value, the more similar the two nodes. h i , h j are node vectors, both from the node set composed of h stat and h sig . · is the dot product. ||·||2 is L2 norm.

[0064] For V i , select the top k nodes with the highest similarity (k=5, determined by cross-validation), mark the connection between V i and these k nodes as an edge; after traversing all nodes, all edges are summarized to form an edge set E={(i,j)}. Combine the node set V and the edge set E, and the initial graph G=(V,E) is obtained.

[0065] Step 3: Causal Enhancement and Association Reasoning: A graph attention network is used to calculate the attention weights between nodes in the initial knowledge graph and to select candidate node pairs with high similarity. The algorithm distinguishes the causal and correlation relationships between candidate node pairs through temporal causality. The causal strength is quantified by combining transitive entropy. The final edge weights are obtained by fusing semantic similarity and causal credibility. At the same time, the knowledge graph is dynamically updated through incremental learning and federated learning.

[0066] Furthermore, the temporal causal discovery algorithm filters potential causal relationships through conditional independence tests and determines reliable causal pairs by combining statistical significance p-values. The transfer entropy, after normalization, is fused with statistical significance weights to calculate causal credibility. Finally, the edge weights are obtained through a weighted fusion of semantic similarity and causal credibility, and the weight coefficients are optimized by minimizing the inference error of the attack chain. Incremental learning uses an incremental graph neural network to update graph parameters, while federated learning is implemented by updating parameters locally at edge nodes and aggregating global parameters in the cloud, protecting the data privacy of edge nodes while adapting to new attack patterns.

[0067] The workflow for causal-enhanced associative reasoning is as follows:

[0068] Step 3.1: Initial association reasoning: Calculate the attention weights between nodes using a graph attention network (GATv2).

[0069] Formula 8:e ij =LeakyReLU(a T [Wh i ||Wh j ]). In formula 8: e ij The attention score is given for nodes i and j. i h is the semantic vector of node i. j Let W be the semantic vector of node j. 256×768 is the learnable weight matrix. a is the attention vector. || is vector concatenation. LeakyReLU is the activation function.

[0070] Normalized attention weights, Formula 9: α ij =exp(e ij ) / ∑ k∈N(i) exp(e ik In Formula 9: α ij ∈[0,1] represents the attention weights. N(i) is the set of neighboring nodes of node i, and exp(·) is an exponential function to ensure that the weights are positive. ij Give attention scores for nodes i and j. ik The attention scores for nodes i and k.

[0071] The normalized attention weights are directly used as the semantic similarity weights S. semantic (i,j), i.e., Ssemantic (i,j)=a ij . For all edges (i,j) in the initial graph G, compute e ij , a ij , and finally get S semantic , S semantic ∈[0,1], the larger the value, the more similar the semantics of nodes i / j. Update the edge set to E sem ={(i,j,S semantic (i,j))}, keep the high similarity node pairs with S semantic ≥0.3 as the candidate set for subsequent causal analysis.

[0072] Step 3.2: Temporal causal discovery: analyze the temporal dependency relationship of candidate node pairs, distinguish cause and effect, and screen significant causal relationships.

[0073] Step 3.2.1: Conditional independence test: use the temporal causal discovery algorithm (PCMCI) to test the independence of X(t) and Y(t+W) after excluding interference Z.

[0074] Formula 10: X(t) Y(t+W1)|Z(t), W1. In formula 10: X(t) is the cause variable, such as the number of port scans at time t. Y(t+W1) is the result variable at time t+W1, such as data transmission traffic. Z(t) is the set of conditional variables, such as the number of normal HTTP requests at time t. W1 is the time window. is the conditional independence.

[0075] Test logic: Test the independence of X(t) and Y(t+W1) after excluding interference Z(t). If X(t) and Y(t+W1) are not independent, then X(t) is considered to be a potential cause of Y(t+W1).

[0076] Step 3.2.2: Statistical significance p-value calculation: calculate the causal significance by permutation test.

[0077] Formula 11: p(X→Y)=P(stat(X',Y'|Z)≥stat(X,Y|Z)). In formula 11: p(X→Y) is the statistical significance p-value of the causal relationship between X and Y, ranging from 0 to 1. stat(▪) is the mutual information, which measures the nonlinear correlation. X', Y' are the time series permutation samples of X, Y, which disrupt the cause and effect by disturbing the order. P(·) is the probability function.

[0078] Decision criteria: If p(X→Y) < 0.05 (statistical significance level), then X→Y is considered to be a reliable causal relationship. Keep the node pairs with p(X→Y) < 0.05 to get the reliable causal pair set C={(X→Y,p(X→Y))}.

[0079] Step 3.3: Transfer entropy calculation: Calculate the transfer entropy (TE) to measure the strength of causal effect.

[0080] Formula 12: TE(X→Y) = H(Y(t+W1)|Y(t)) - H(Y(t+W1)|Y(t),X(t)). In Formula 12: TE(X→Y) is the transfer entropy of X to Y (unit: bits), which measures the amount of information transfer, and the larger the value, the greater the information transfer of X to Y and the stronger the causal effect. H(·) is the conditional entropy, a measure of uncertainty. Y(t+W1) is the result variable at time t+W1. X(t) is the X variable at time t. Y(t) is the Y variable at time t.

[0081] Normalized transfer entropy, scaling TE to the interval [0,1].

[0082] Formula 13: TE norm (X→Y) = TE(X→Y) / TE max . In Formula 13, TE norm is the normalized transfer entropy. TE(X→Y) is the original transfer entropy. TE max is the maximum TE value of all causal pairs.

[0083] Step 3.4: Causal credibility evaluation: Combine semantic similarity and causal credibility to obtain the final weight of the graph edge.

[0084] Formula 14: S causal = (1-p(X→Y))·TE norm (X→Y). In Formula 14, S causal is the causal credibility score, ranging from 0 to 1. p(X→Y) is the statistical significance p-value. TE norm is the normalized transfer entropy.

[0085] Final edge weight fusion, Formula 15: S final =α·S semantic +β·S causal . In Formula 15: α=0.4 is the semantic similarity weight, β=0.6 is the causal credibility, which is optimized by minimizing the attack chain reasoning error, and α+β=1. S final is the final edge weight, ranging from 0 to 1, and the larger the value, the more reliable the association.

[0086] Set the edge weight of the reliable causal pair to S final , update the graph edge set to E final ={(i,j,S final (i,j))}, and obtain the final cross-modal semantic alignment knowledge graph (CMSKG), with the edge weight reflecting the reliability of the association.

[0087] Step 3.5: Dynamic graph update: Adapt to new attacks through incremental learning and federated learning to avoid graph obsolescence, while protecting the data privacy of edge nodes.

[0088] Incremental graph neural network (IncGNN) is used for parameter updates.

[0089] Formula 16: θ GNN (t+1)=θ GNN (t)+η·▽ θGNN L(θ GNN (t),D new In Formula 16: θ GNN (t+1) represents the GNN parameters at time t+1. θ GNN (t) represents the GNN parameters at time t, such as W in step 3.1. new For the characteristic data of new traffic, such as the new X stat X sig L represents the cross-entropy loss during attack chain inference. ▽ θGNN L is the loss function L against θ GNN The gradient. η = 0.001 is the learning rate.

[0090] Parameter aggregation, Formula 17: θ GNN，global =(∑ k n k ·θ GNN,local,k ) / (∑ k n k In Formula 17: θ GNN，global These are the global GNN parameters aggregated in the cloud. θ GNN,local,k Let n be the local GNN parameters for the k-th edge node. k This represents the new data volume of the k-th node.

[0091] Edge nodes use D locally new Update θ GNN,local Only the parameter gradients are uploaded to the cloud, and the cloud aggregates and then distributes θ. GNN，global Edge nodes update their local CMSKG. The resulting dynamically evolved CMSKG can identify new attacks and traffic patterns, providing the latest correlation information for subsequent engine optimization.

[0092] Step 4: Intelligent Engine Optimization: Build and optimize the anomaly detection engine and attack identification engine, use reinforcement learning to dynamically schedule signature subsets, and achieve dual-engine co-evolution through generative adversarial networks.

[0093] Further, the anomaly detection engine adopts a variational autoencoder, and the attack recognition engine adopts a convolutional neural network; the reinforcement learning signature scheduling uses a deep Q network, taking the detection accuracy and resource consumption rate as the reward function, the deep Q network models the signature selection as a Markov decision process, the state includes the attack type, traffic intensity, and CPU usage rate, the action includes selecting a signature subset of different attack types, the reward function prioritizes the detection accuracy, and the output is the optimal signature subset with the maximum Q value.

[0094] Further, the attack recognition engine in the generative adversarial network acts as a generator, and the anomaly detection engine acts as a discriminator, and unknown anomalies trigger new signature generation; the generative adversarial network constructs a feedback loop for generating simulated attacks, detecting unknown anomalies, and updating signatures, and the method is as follows: the attack recognition engine as a generator inputs random noise and attack statistical features to generate a simulated attack vector, the anomaly detection engine as a discriminator distinguishes between real traffic and simulated attack vectors, and through adversarial training, the detection ability of the discriminator on unknown anomalies and the authenticity of the simulated attack generated by the generator are improved, and a new signature is generated to update the signature library of the attack recognition engine after detecting unknown anomalies.

[0095] The intelligent engine optimizes the workflow as follows:

[0096] Step 4.1: Dual-engine initialization: initialize the anomaly detection engine (E1) and the attack recognition engine (E2).

[0097] Step 4.1.1: Anomaly detection engine E1 initialization: build an unsupervised anomaly detection model to identify unknown abnormal traffic.

[0098] Train the variational autoencoder (VAE) on normal traffic to learn the normal distribution. Use the h stat Train the VAE to make the model learn to reconstruct normal traffic, with a reconstruction error <0.1. The output of the initial E1 after processing is the parameter θ enc0 (encoder weights), θ dec0 (decoder weights), which can be determined by the "reconstruction error > threshold 0.5", and the threshold is determined by cross-validation.

[0099] Step 4.1.2: Attack recognition engine E2 initialization: build a supervised attack recognition model to match known attack signatures.

[0100] Use a convolutional neural network (CNN) to train on known attack data, and use the h sig and artificial labels to train the CNN, with an accuracy of ≥90%;

[0101] The output of the initial E2 after processing is the parameter θ cnn0 (CNN layer weights), and the supporting initial signature library S init (Xsig corresponding signature).

[0102] Step 4.2: Reinforcement learning signature scheduling: dynamically select the optimal signature subset of E2, improve the detection efficiency of E2, and reduce resource consumption.

[0103] The signature selection is modeled as a Markov Decision Process (MDP) using a Deep Q Network (DQN).

[0104] MDP four-tuple definition:

[0105] State S = [A type , T intensity , Res usage ]. Where: A type is the current attack type: 0 = normal / 1 = DDoS / 2 = SQL injection. T intensity is the traffic intensity, normalized [0, 1]. Res usage is the CPU usage rate, normalized [0, 1].

[0106] Action A: Select signature subset, such as A1 = DDoS subset, A2 = SQL injection subset.

[0107] Reward R(S, A) = λ1·Acc(A) - λ2·Res(A). Where: R(S, A) is the reward value of action A under state S. Acc(A) is the detection accuracy of the signature subset corresponding to action A. Res(A) is the resource consumption rate of action A. λ1, λ2 are weights, λ1 = 0.9, λ2 = 0.1, priority is given to accuracy.

[0108] Train the DQN model using historical data, update the Q function.

[0109] Equation 18: Q(S, A; θ q ) = E[R(S, A) + γ·max A' Q(S', A'; θ q ')]. In equation 18: Q(S, A; θ q ) is the expected cumulative reward of action A under current state S. θ q is the current Q network parameter. E[·] is the mathematical expectation. R(S, A) is the immediate reward. θ q ' is the target network parameter. γ = 0.9 is the discount factor. max A' Q(S', A'; θ q ') is the maximum expected reward Q value of all possible actions A' under the next state S'.

[0110] DQN is trained by historical state-action-reward data, and outputs the action A with the maximum Q valueopt . Processed output optimal signature subset A opt , Example: A under DDoS scenario opt =DDoS signature subset), E2 only loads A opt Detection, archive invalid signatures such as SQL injection signatures, resource consumption reduced by 30%.

[0111] Step 4.3: GAN feedback loop: build a closed loop of generating simulated attacks → detecting unknown anomalies → updating signatures, improve dual-engine capabilities, and improve detection capabilities through adversarial learning of E1 and E2.

[0112] Step 4.3.1: GAN adversarial training.

[0113] E2 as generator G, input random noise z (such as Gaussian noise) and attack statistical features stat attack (such as DDoS attack traffic statistics), output simulated attack vector h gen =G(z,stat attack ;θ g ), where: z is random noise, stat attack is attack statistical features, θ g is the generator parameter. E1 as discriminator D, input real normal traffic h normal , real attack traffic h abnormal and E2 generated simulated attack traffic h gen , output judgment probability D(h; θ d ) ∈ [0,1], indicating its confidence that the input traffic is abnormal (attack), close to 1 indicates real, close to 0 indicates simulation.

[0114] Formula 19: L g =-E[logD(h gen )]. In formula 19: L g is the generator loss value. E[·] is the mathematical expectation. D(·) is the output probability of the discriminator. h gen is the simulated attack vector output by the generator.

[0115] Formula 20:

[0116] L d =-E[logD(h abnormal )]-E[log(1-D(h normal ))]-E[log(1-D(h gen ))].

[0117] In formula 20: L d is the discriminator loss value. E[·] is the mathematical expectation. D(·) is the output probability of the discriminator. h abnormalReal attack vector.h normal Normal vector.h gen Simulated attack vector.

[0118] The generator wants the discriminator to output close to 0 (think the generated is fake), so the loss value L g The smaller, the more successful the generator tricked the discriminator. Update θ g , so that the generated attack traffic is more realistic. The discriminator wants to output 1 for real traffic and 0 for simulated traffic, and the smaller the loss value, the stronger the distinguishing ability. Update θ d , to improve the ability to distinguish between real and simulated. Through adversarial training, the distinguishing ability of D is constantly improved, and finally it can accurately identify the optimized h gen , D(h gen )≈0.85, close to the real attack.

[0119] Step 4.3.2: Unknown anomaly detection and signature update.

[0120] In repeated adversarial training, the distinguishing ability of E1 is greatly enhanced, and it can find some never-before-seen, strange patterns that are neither like known attacks nor like normal traffic, nor like the simulated attacks E2 can currently generate. These are unknown anomalies h unknown .

[0121] Unknown anomaly traffic h unknown must meet two conditions, judged by E1:

[0122] High anomaly probability: D(h unknown )>0.9;

[0123] No matching signature: input h unknown into E2's signature matching module, no existing signature is matched.

[0124] When h unknown is detected, the process enters the feedback loop: input h unknown into E2. E2 analyzes the characteristic pattern of h unknown , and creates a new signature rule s new through the signature generation module SigGen (which can be a feature extractor + rule generator).

[0125] Equation 21: s new =SigGen(h unknown ;θ sig,new ). In equation 21: h unknown is the unknown anomaly vector detected by E1. SigGen is a CNN feature extractor. θ sig,new is the trained generation parameter.

[0126] The new signature s new is added to the signature library of E2. From now on, the unknown attack becomes known attack, making the model able to identify similar unknown attacks.

[0127] Step 4.4: Engine parameter update: integrate the optimization signals of DQN and GAN, update E1, E2 parameters. The E1 parameter update formula is as follows:

[0128] Formula 22: θ enc1 = θ enc0 - η ·▽θ enc L d . In formula 22: θ enc1 is the updated E1 encoder. θ enc0 is the E1 encoder parameter before updating. η is the learning rate.▽θ enc L d is the gradient of the loss function L d relative to θ enc .

[0129] Formula 23: θ dec1 = θ dec0 - η ·▽θ dec L d . In formula 23: θ dec1 is the updated E1 decoder parameter. θ dec0 is the E1 decoder parameter before updating.

[0130] The E2 parameter update is formula 24: θ cnn1 = θ cnn0 - η ·▽θ cnn (L CNN + λ · L g ). In formula 24: θ cnn1 is the updated E2 parameter. θ cnn0 is the E2 parameter before updating, including the weights of the generator and the classifier. η is the learning rate.▽θ cnn is the gradient of the E2 parameter. L CNN is the classification loss, such as cross-entropy loss, which measures the classification accuracy of E2 on known attacks. L g is the generator loss, such as negative log-likelihood loss, which measures the authenticity of the simulated attack generated by E2. λ = 0.5 is the balance coefficient, balancing the classification loss L CNN and the generation loss L g .

[0131] After processing, the optimized E1 and E2 are output, forming a detection → optimization → re-detection cycle.

[0132] Step 5: Cloud-edge collaborative resource scheduling: The dual-engine is regarded as a game player to build a game model, and the particle swarm game optimization algorithm is used to optimize the dual-engine resource allocation. According to the GPU acceleration ratio, the cloud-edge tasks that need to be executed collaboratively between the edge node and the cloud are divided into lightweight tasks and computationally intensive tasks. The cloud-edge collaborative offloading is performed, and the deep Q network is used to adjust the dual-engine resource priority based on the attack severity.

[0133] Further, the constraint condition of the game model is that the sum of the dual-engine CPU occupancy is less than or equal to 0.5, and 50% of the CPU resources are reserved for other system tasks. The payoff function of the game model is the weighted sum of the detection accuracy and resource consumption. The particle swarm game optimization algorithm aims to maximize the total revenue of the dual-engine, and converges to the Nash equilibrium point through iterative updates of particle velocity and position.

[0134] Further, the lightweight tasks are executed on the edge, and the computationally intensive tasks are executed on the cloud. The cloud-edge task offloading method is as follows: the edge side handles lightweight tasks such as traffic collection, feature encoding, and real-time detection, and only uploads the input data of intensive tasks to the cloud. The cloud side uses GPU to handle intensive tasks such as knowledge graph updating and global parameter synchronization, and only issues the processing results to the edge, reducing bandwidth consumption and delay.

[0135] The workflow of cloud-edge collaborative resource scheduling is as follows:

[0136] Step 5.1: Game modeling: E1 and E2 are regarded as game players, and the payoff function is defined.

[0137] Players: anomaly detection engine E1: strategy space r1 ∈ [0, 0.5] (CPU occupancy).

[0138] Attack identification engine E2: strategy space r2 ∈ [0, 0.5] (CPU occupancy).

[0139] The E1 payoff function is formula 25: U1(r1, r2) = α1·Acc1(r1) - β1·r1. In formula 25: U1(r1, r2) is the revenue value of E1. r1 is the CPU occupancy of E1. Acc1(r1) is the detection accuracy of E1 under resource r1. α1 = 1.0 is the accuracy weight. β1 = 0.5 is the resource consumption weight.

[0140] The E2 payoff function is formula 26: U2(r1, r2) = α2·Acc2(r2) - β2·r2. In formula 26: U2(r1, r2) is the revenue value of E2. r2 is the CPU occupancy of E2. Acc2(r2) is the detection accuracy of E2 under resource r2. α2 = 1.0 is the accuracy weight. β2 = 0.5 is the resource consumption weight.

[0141] Constraint: r1+r2≤0.5, reserve 50% CPU for other system tasks. After processing, the game model {U1, U2, r1, r2} is obtained, and the goal is to find the Nash equilibrium point where "any engine cannot improve its own income by changing r".

[0142] Step 5.2: Nash equilibrium solution: use particle swarm game optimization algorithm (PSO-Game) to solve the optimal resource allocation. Particle definition: each particle position x i =[r1, r2] (resource allocation ratio), velocity v i (direction of strategy adjustment).

[0143] The velocity is updated as formula 27:

[0144] v i (t+1)=ω·v i (t)+c1·r1·(p i -x i (t))+c2·r2·(p g -x i (t))。

[0145] In formula 27: v i (t+1) is the velocity of particle i at time t+1. ω=0.7 is the inertia weight. c1=c2=1.49 is the learning factor. p i is the individual optimal position of particle i. p g is the global optimal position. x i (t) is the position of particle i at time t.

[0146] The position is updated as formula 28: x i (t+1)=x i (t)+v i (t+1). In formula 28: x i (t+1) is the position of particle i at time t+1.

[0147] The fitness function is formula 29: fit(x i )=U1(r1, r2)+U2(r1, r2). In formula 29: fit(x i ) is the fitness value of particle i. x i is the position vector of particle i. The total income is maximized, corresponding to the Nash equilibrium point.

[0148] Particles move to high-income areas, iteratively solve, and converge to the Nash equilibrium point to get the optimal resource allocation strategy (r1 * , r2 * ).

[0149] Step 5.3: Cloud-edge task offloading: allocate computing resources according to task type.

[0150] Step 5.3.1: Task division: divide light tasks and compute-intensive tasks by testing the GPU speedup ratio of different tasks through experiments.

[0151] GPU speedup formula 28: Speedup = T GPU / T CPU . In formula 28: T CPU is the CPU processing time. T GPU is the GPU processing time. The division with Speedup < 2 is a light task, and the division with Speedup ≥ 10 is a compute-intensive task. The specific task division is as follows:

[0152] Light task: the processing node is the edge; the processing content includes: traffic collection, TraffBERT / Sig2Vec encoding, E1 / E2 real-time detection; the resource demand is low;

[0153] Compute-intensive task: the processing node is the cloud, and the processing content includes: CMSKG update, S updated global synchronization, GAT inference; the resource demand is high.

[0154] Step 5.3.2: Perform cloud-edge offloading according to the division standard.

[0155] After task division, the core of cloud-edge offloading is that the edge does fast and light real-time processing, and the cloud does heavy and slow intensive computing, and the offloading process needs to rely on the speedup ratio to ensure efficiency.

[0156] Edge side offloading action: only upload the input data of intensive tasks (such as 1000 traffic feature vectors newly collected by the edge, about 768KB) to the cloud, without transmitting the original traffic, to avoid bandwidth waste;

[0157] Cloud side offloading action: use GPU to quickly process intensive tasks, such as completing IncGNN parameter update in 0.3s, and only transmit the processing result (such as the updated GNN parameter θ GNN,global , about 10MB) to the edge, without transmitting intermediate data, to further reduce the delay.

[0158] If the GPU speedup ratio of the cloud is insufficient, the intensive task processing time will increase from 0.3s to 1.5s, causing the edge to wait for a long time, which destroys the real-time detection goal, so the speedup ratio is the key to keep the system low delay after offloading.

[0159] The edge and the cloud output the task execution results after completing the corresponding tasks, which are divided into two categories: edge side results and cloud side results.

[0160] Edge-side task execution result: After the edge processing of light tasks, the output is the real-time traffic detection result, including: traffic label: the detection conclusion of each traffic, such as normal, known attack-DDoS, unknown anomaly; basic information: source IP, destination IP, port number, and collection timestamp of the traffic; preliminary association: the CMSKG node and initial edge weight corresponding to the traffic, such as traffic A associated with DDoS signature node, Ssemantic=0.72.

[0161] End-side task execution result: After the cloud processing of intensive tasks, the output is the global data for system optimization, including: synchronization data: the globally updated signature library S updated , such as the new signature library expanded from 1000 to 1005; optimization parameters: the global GNN parameters θ GNN,global updated by IncGNN.

[0162] Step 5.4: DQN dynamic priority adjustment: adjusting the resource priority of E1 and E2 according to the attack severity.

[0163] Define state and action:

[0164] State S´´=[A type ,A severity ],A severity is the attack severity: 0=low, 1=medium, 2=high.

[0165] Action A´´={A E1 ,A E2},A E1 =E1 priority, A E2 =E2 priority.

[0166] Reward R´´=Acc total (A´´)-Res delay (A´´),Acc total is the total accuracy of the dual engine, Res delay is the resource delay loss.

[0167] DQN processing method synchronizes step 4.2. Use historical data to train the DQN model, and DQN selects the action with the maximum reward to adjust the resource proportion. Example: during DDoS attack, the CPU proportion of E2 is increased from 30% to 35%, and the detection delay is reduced from 0.8 seconds to 0.5 seconds.

[0168] Step 6: result output: trace the nodes along the high-weight edges of the knowledge graph to construct the complete attack chain, divide the alarm priority according to the edge weight and attack severity, and output the alarm information, and output the resource allocation report by counting the resource utilization and task execution time.

[0169] According to the high-weight edge attack chain, output hierarchical alarm information and resource allocation report.

[0170] The output workflow is as follows:

[0171] Step 6.1: Attack chain reasoning result: trace the nodes along the high-weight edge to build a complete attack link. Processing result: attack chain report, example: port scanning, S node, S final = 0.82 → data export, S node, S final = 0.78 → ransomware, T node, S final = 0.91, label the cause and effect of each link and the time window.

[0172] Step 6.2: Alarm information output: according to S final and A severity , alarm priority is divided.

[0173] Classification criteria: high priority: S final ≥ 0.9 and A severity = 2, such as ransomware attack, trigger SMS + email alarm; medium priority: 0.7 ≤ S final <0.9 and A severity = 1, such as SQL injection, trigger email alarm; low priority: S final <0.7 and A severity = 0, such as suspicious port connection, only log record. Processing result: hierarchical alarm information, including attack type, occurrence time, affected IP, and recommended disposal measures.

[0174] Step 6.3: Resource allocation report: statistics resource utilization rate, task execution time.

[0175] Processing result: resource report, example: E1 CPU utilization rate 25%, E2 CPU utilization rate 30%, cloud GAT reasoning time 0.3s, edge real-time detection time 0.5s, resource utilization rate 85.3%, providing basis for subsequent resource optimization.

[0176] Taking the traffic of a campus network core router as the monitoring object, the target detection DDoS attack and SQL injection attack are detected. The DDoS attack is characterized by SYNFlood, and the SQL injection attack is marked by "UNION SELECT" characteristic code. The time window is 1 second, and a total of 10 seconds of original traffic data is collected for illustration.

[0177] The tcpdump tool is used to collect traffic from the campus network core router with a sampling frequency of 1 second per time, and 10 time window original traffic data are obtained, including packet, session record, device log, etc. For each 1 second window, 3 types of statistical features are extracted: packet number per second, average packet length, retransmission rate. Take the 5th second window as an example, the data is: pkg / s=1200, average packet length=512 byte, retransmission rate=3%. Attack signatures are extracted from the traffic by regular matching: DDoS: the number of SYN packets in this window is 950, the total number of packets is 1000, the SYN packet ratio is 95%, which exceeds the 90% threshold, matching the DDoS signature; SQL injection: the traffic contains the “UNIONSELECT” feature code, matching the SQL injection signature. Assuming that the mean μ and standard deviation σ of the statistical features in the training set are: μ=800, σ=200; average packet length: μ=400, σ=100; retransmission rate: μ=1, σ=0.5. The statistical feature standardization calculation of the 5th second window: standardized pkg / s=(1200-800) / 200=2, standardized average packet length=(512-400) / 100=1.12, standardized retransmission rate=(3-1) / 0.5=4.

[0178] The standardized statistical feature vector [2, 1.12, 4] is input into the TraffBERT model pre-trained by normal traffic, and the statistical feature semantic vector h stat =[0.8, 0.2, -0.5, …] is obtained. The DDoS signature is encoded by Sig2Vec, and the semantic vector h sig,ddos =[0.7, 0.3, -0.4, …] is obtained. The SQL injection signature after encoding gets: h sig,sql =[0.6, 0.4, -0.3, …].

[0179] The cosine similarity between h stat and h sig,ddos is 0.989, which is greater than the threshold 0.3, so h stat and h sig,ddos are constructed as edges in the knowledge graph. Let the weight matrix W be a 3x3 matrix [[0.1, 0, 0], [0, 0.1, 0], [0, 0, 0.1]], and the attention vector a=[1, 1, 1].

[0180] The first 3 dimensions of h stat and h sig,ddos are calculated:

[0181] Wh stat =[0.8×0.1, 0.2×0.1, -0.5×0.1]=[0.08, 0.02, -0.05];

[0182] Wh sig,ddos= [0.7 x 0.1, 0.3 x 0.1, -0.4 x 0.1] = [0.07, 0.03, -0.04];

[0183] Concatenated vector = [0.08, 0.02, -0.05, 0.07, 0.03, -0.04];

[0184] e ij = LeakyReLU(a T · {Concatenated vector}) = 0.09.

[0185] If h stat is only adjacent to h sig,ddos , then a ij = 1, that is, semantic similarity weight S semantic = 1.

[0186] Take the proportion of SYN packets (X) to total traffic surge (Y) as an example: let Z be the number of normal HTTP requests, and calculate the mutual information: stat (X, Y | Z) of the real data = 0.8; In the replaced data, the number of times stat (X', Y' | Z) ≥ 0.8 is 20, and the total number of replacements is 1000, so the p value = 20 / 1000 = 0.02 < 0.05, which is statistically significant, and it is determined that X is the cause of Y.

[0187] Transfer entropy calculation:

[0188] TE (X→Y) = H (Y (t+1) | Y (t) ) - H (Y (t+1) | Y (t), X (t) ) = 0.6 - 0.2 = 0.4.

[0189] Normalized transfer entropy, assuming TE {text{max}} = 0.5: TE norm = 0.4 / 0.5 = 0.8.

[0190] Causal credibility S causal = (1 - p (X→Y) ) · TE norm = (1 - 0.02) x 0.8 = 0.784.

[0191] Final edge weight fusion, a = 0.4, b = 0.6: S final = 0.4 x 1 + 0.6 x 0.784 = 0.8704.

[0192] DQN dynamic selection of signature subset: state S: attack type, traffic intensity, CPU usage; action A: select DDoS signature subset; reward function, λ1=0.9, λ2=0.1: R=0.9xAcc-0.1xRes. Assuming that after selecting the subset, the detection accuracy Acc=0.95, the resource consumption rate Res=0.4, then: R=0.9x0.95-0.1x0.4=0.815.

[0193] GAN adversarial training: generator: input random noise z and DDoS statistical features, generate simulated attack vector h gen . Discriminator: distinguish real DDoS vector h abnormal from h gen .

[0194] Assuming D(h gen )=0.3, then the generator loss L g ≈1.204; assuming D(h abnormal )=0.9, D(h normal )=0.1, then the discriminator loss L d ≈0.567.

[0195] Model parameter update, learning rate η=0.001: E1 encoder parameter update: θ enc1 =θ enc0 -η·▽θ enc L d ≈θ enc0 -0.005. Game model and Nash equilibrium: let the payoff functions of E1 and E2 be: U1(r1, r2)=1.0xAcc1(r1)-0.5xr1, U2(r1, r2)=1.0xAcc2(r2)-0.5xr2, where r1 is the CPU occupancy of E1, r2 is the CPU occupancy of E2, and the constraint is r1+r2≤0.5. Initial resource allocation: r1=0.2, Acc1=0.9; r2=0.3, Acc2=0.92, then: U1=1.0x0.9-0.5x0.2=0.8, U2=1.0x0.92-0.5x0.3=0.77. Particle swarm optimization iteration: through particle velocity and position update, finally converges to Nash equilibrium point r1 * =0.25, r2 * =0.25, at this time: U1=1.0x0.92-0.5x0.25=0.795, U2=1.0x0.93-0.5x0.25=0.805. The total revenue is the optimal resource allocation strategy.

[0196] Attack chain and alarm: along the edge with an edge weight of 0.8704 in the knowledge graph, combined with the causal relationship, the DDoS attack chain is constructed. Because the edge weight is 0.8704≥0.7 and the DDoS attack severity is high, a high-priority alarm is triggered.

[0197] Resource report: E1: CPU usage 25%, resource utilization 80%. E2: CPU usage 25%, resource utilization 85%; cloud graph reasoning time: 0.5 seconds / time; edge real-time detection time: 0.1 seconds / time. Conclusion Through the above steps, the method successfully detects the DDoS attack in the campus network, completes feature semantic alignment, causal enhanced reasoning, intelligent engine optimization and resource scheduling, and outputs high-priority alarm and resource report, verifying the effectiveness of the method.

[0198] Based on the same inventive concept as the deep learning-based abnormal traffic detection and attack identification method provided in the embodiments of the present application, the embodiments of the present application also provide a deep learning-based abnormal traffic detection and attack identification system. For the content in the system embodiments, if there is any ambiguity, please refer to the corresponding content in the method embodiments.

[0199] Embodiment 2:

[0200] The deep learning-based abnormal traffic detection and attack identification system comprises:

[0201] The data acquisition module acquires raw traffic data from network devices in real time;

[0202] The preprocessing module cleans, normalizes and formats the raw traffic data to generate a structured data set;

[0203] The feature extraction module extracts statistical features and signature features from the structured data set in parallel. The statistical features calculate traffic distribution indicators through a time window, and the signature features generate behavior fingerprints based on an attack rule library. At the same time, the statistical features are standardized to eliminate dimensional effects. The standardized statistical features and signature features are outputted;

[0204] The cross-modal semantic alignment module deploys a traffic bidirectional encoder representation model (TraffBERT) and a signature feature vector embedding model (Sig2Vec). The former encodes the standardized statistical features into statistical feature semantic vectors, and the latter encodes the signature features into signature feature semantic vectors, so that the two types of vectors are in the same 768-dimensional semantic space to eliminate the semantic gap;

[0205] The knowledge graph construction module takes the two types of semantic vectors as graph nodes, calculates the semantic similarity between the nodes through a k-nearest neighbor algorithm, and selects high-similarity nodes to construct an initial knowledge graph;

[0206] The causal enhanced correlation reasoning module calculates the semantic correlation weight between nodes by using a graph attention network, and uses a time series causal discovery algorithm and a transfer entropy analysis to analyze the causal relationship and strength between nodes, fuses semantic and causal information to update the edge weight of the knowledge graph, and realizes the dynamic evolution of the graph through incremental learning and federated learning mechanism.

[0207] The intelligent engine optimization module includes an unsupervised learning-based anomaly detection engine and a supervised learning-based attack identification engine, which dynamically schedules attack signature subsets through reinforcement learning to improve efficiency, and constructs an adversarial feedback loop through a generative adversarial network to generate simulated attacks, detect unknown anomalies and update the signature library, realizing the co-evolution of the double-engine.

[0208] The cloud-edge collaborative resource scheduling module models the system resource allocation as a Nash equilibrium problem through a game theory model, and uses an optimization algorithm to solve the optimal resource allocation strategy, while intelligently offloading tasks and synchronizing results between edge nodes and the cloud according to the computational intensity of the tasks.

[0209] The result output module performs attack chain reasoning based on the final knowledge graph and generates a visual report, outputs hierarchical alarm information according to the correlation weight and attack severity level, and generates a system resource utilization report.

[0210] Further, the feature extraction module uses a sliding window statistical method to extract the number of packets per second, average packet length, retransmission rate, average connection duration and connection establishment success rate as statistical features; and extracts attack feature codes, abnormal ports and abnormal protocols as signature features from the public attack rule library through regular matching.

[0211] The cross-modal semantic alignment module uses the TraffBERT model to encode the statistical features, and uses the Sig2Vec model to embed the signature features. The parameters of the TraffBERT model are obtained by pre-training on normal traffic, and the parameters of the Sig2Vec model are obtained by training on the Snort rule library.

[0212] The knowledge graph construction module calculates the semantic similarity between nodes using cosine similarity, initializes by first constructing an edgeless graph, then constructs edges for each node by selecting the node with the highest similarity, and forms an initial knowledge graph.

[0213] The causal enhanced correlation reasoning module uses the PCMCI algorithm for conditional independence test and significance calculation to discover reliable causal relationships, and uses transfer entropy to quantify causal strength, and finally weights and fuses the semantic similarity weight and causal reliability score into the final edge weight.

[0214] In the intelligent engine optimization module, the anomaly detection engine adopts a variational autoencoder model, and the attack identification engine adopts a convolutional neural network model; reinforcement learning adopts a deep Q network to select the optimal signature subset by taking detection accuracy and resource consumption as the reward function; and a generative adversarial network takes the attack identification engine as the generator and the anomaly detection engine as the discriminator.

[0215] The cloud-edge collaborative resource scheduling module uses a particle swarm optimization algorithm to solve the Nash equilibrium point to obtain the optimal CPU resource ratio allocated to the double engine, and according to the GPU acceleration ratio, the traffic collection, feature encoding and real-time detection tasks are deployed on the edge side, and the knowledge graph update, global signature library synchronization and complex graph reasoning tasks are unloaded to the cloud.

[0216] The result output module traverses the edges in the knowledge whose weight is higher than the set threshold to reconstruct the attack chain, and divides the attack chain into high, medium and low alert priorities according to the causal strength and attack type label of the attack chain.

[0217] The system carries out the flow of abnormal flow detection and attack identification, which starts from data collection, and each module realizes the whole process detection and identification through the orderly data flow, as follows: the data collection module collects the original flow data from the router, switch and other network devices in real time, and transmits it to the preprocessing module; the preprocessing module receives the original flow data, and generates a structured data set after cleaning, normalization and formatting processing, and then transmits the data set to the feature extraction module; the feature extraction module receives the structured data set, extracts the statistical features such as the number of data packets per second, the average packet length and the retransmission rate in parallel by using the sliding window statistical method, extracts the signature features such as attack feature code, abnormal port and abnormal protocol by regular matching combined with the public attack rule library, and transmits the standardized statistical features and signature features to the cross-modal semantic alignment module after standardizing the statistical features; the cross-modal semantic alignment module receives two types of features, encodes the standardized statistical features into statistical feature semantic vectors by the TraffBERT model pre-trained by normal flow, encodes the signature features into signature feature semantic vectors by the Sig2Vec model trained by the Snort rule library (the two types of vectors are in the same 768-dimensional semantic space to eliminate the semantic gap), and transmits the two types of semantic vectors to the knowledge graph construction module; the knowledge graph construction module receives the semantic vectors, takes them as nodes, calculates the semantic similarity between the nodes by the cosine similarity, filters the high similarity nodes to construct an initial knowledge graph, and transmits the initial graph to the causal enhancement correlation reasoning module; the causal enhancement correlation reasoning module receives the initial knowledge graph, calculates the semantic correlation weight of the nodes by using the graph attention network, discovers the reliable causal relationship by the conditional independence test and significance calculation of the PCMCI algorithm, quantifies the causal strength by the transfer entropy, updates the edge weight of the graph by fusing the semantic similarity and causal credibility, realizes the dynamic evolution of the graph by incremental learning and federated learning, and transmits the final knowledge graph to the intelligent engine optimization module and the result output module respectively; the intelligent engine optimization module receives the final knowledge graph, optimizes the anomaly detection engine composed of the variational autoencoder and the attack identification engine composed of the convolutional neural network, dynamically schedules the attack signature subset by the deep Q network, constructs the feedback closed loop of “generating simulated attack-detecting unknown anomaly-updating signature library” by the generative adversarial network, and transmits the resource demand data of the double-engine to the cloud-edge collaborative resource scheduling module; the cloud-edge collaborative resource scheduling module receives the resource demand of the intelligent engine optimization module and the task calculation intensity data of each module, solves the Nash equilibrium by the particle swarm optimization algorithm to determine the optimal CPU resource allocation strategy, issues the resource allocation instructions to each module, deploys the light tasks such as flow collection, feature encoding and real-time detection on the edge node according to the GPU acceleration ratio, unloads the calculation-intensive tasks such as knowledge graph updating, global signature library synchronization and complex graph reasoning to the cloud, and realizes data synchronization by returning the cloud processing result to the edge node;The result output module receives the final knowledge graph of the causal enhancement association reasoning module, traverses the attack chain of edges with a weight higher than a set threshold, and divides the attack type label into high, medium and low levels of alarm. Meanwhile, the result output module receives the system resource utilization data of the cloud-edge collaborative resource scheduling module, generates a visual attack chain report, a hierarchical alarm information and a resource utilization report, and completes the entire abnormal traffic detection and attack identification process.

[0218] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments or equivalently replace some of the technical features. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A method for detecting abnormal traffic and identifying attacks based on deep learning, characterized in that, The method comprises the following steps: Collecting raw traffic data from network devices, extracting statistical features and signature features in parallel, and standardizing the statistical features to obtain standardized statistical features; Encoding the standardized statistical features and signature features into a vector set in a unified semantic space to construct an initial knowledge graph; Using a graph attention network to calculate the attention weights between nodes in the initial knowledge graph and to filter to obtain high-similarity candidate node pairs, using a time-series causality algorithm to distinguish the causal and correlation relationships between the candidate node pairs, combining the semantic similarity and causal reliability to obtain the final edge weight, and realizing dynamic updating of the knowledge graph through incremental learning and federated learning; Building and optimizing an anomaly detection engine and an attack identification engine, dynamically scheduling a signature subset using reinforcement learning, and realizing co-evolution of the two engines through a generative adversarial network; Building a game model based on the two engines, optimizing resource allocation of the two engines using a particle swarm game optimization algorithm, dividing cloud-edge tasks that need to be cooperatively executed between edge nodes and the cloud into lightweight tasks and computationally intensive tasks according to a GPU acceleration ratio, performing cloud-edge cooperative offloading, and adjusting the resource priority of the two engines according to the severity of the attack based on a deep Q network; Tracing attack links according to high-weight edges, and outputting hierarchical alarm information and resource allocation reports.

2. The method of claim 1, wherein, The statistical feature extraction uses a sliding window method, and the window size is consistent with the sampling frequency. The number of packets per second, the average packet length, the retransmission rate, the connection duration, and the connection establishment success rate are extracted. The signature feature extraction obtains attack feature codes, abnormal ports, and abnormal protocols from a public attack rule library through regular matching and ngram methods. The standardization processing is based on the mean and standard deviation of the training set for Z-score normalization.

3. The method of claim 1, wherein, The statistical feature encoding uses a TraffBERT model to output 768-dimensional semantic vectors, and the signature feature encoding uses a Sig2Vec model to map discrete signatures to continuous vectors and to the same space as the statistical feature semantic vectors. The initial knowledge graph is constructed by a k-nearest neighbor algorithm, the node semantic similarity is calculated using cosine similarity, and the initial edges are constructed by selecting a number of nodes with the highest similarity and the current node.

4. The method of claim 1, wherein, The time-series causality discovery algorithm screens potential causal relationships through conditional independence testing, and determines reliable causal pairs in combination with statistical significance p values. The causal reliability is calculated by fusing the normalized transfer entropy and statistical significance weight. The final edge weight is obtained by weighted fusion of semantic similarity and causal reliability, and the weight coefficient is optimized by minimizing attack chain reasoning error. The incremental learning uses an incremental graph neural network to update graph parameters, and the federated learning realizes local parameter updating by edge nodes and global parameter aggregation by the cloud in a manner that protects the data privacy of edge nodes while adapting to new attack modes.

5. The method of claim 1, wherein, The anomaly detection engine uses a variational autoencoder, and the attack identification engine uses a convolutional neural network. The reinforcement learning signature scheduling uses a deep Q network to detect the accuracy rate and resource consumption rate as a reward function, the deep Q network models the signature selection as a Markov decision process, the state includes attack type, traffic intensity, CPU usage, the action includes selecting a subset of signatures of different attack types, the reward function prioritizes the detection accuracy, and outputs the optimal signature subset with the maximum Q value.

6. The method of claim 5, wherein, In the generative adversarial network, the attack identification engine acts as a generator, and the anomaly detection engine acts as a discriminator, and unknown anomalies trigger new signature generation; The generative adversarial network constructs a feedback loop of generating simulated attacks, detecting unknown anomalies, and updating signatures, and the method is: the attack identification engine inputs random noise and attack statistical features to generate a simulated attack vector as a generator, the anomaly detection engine distinguishes between real traffic and simulated attack vectors as a discriminator, and through adversarial training, the detection ability of the discriminator for unknown anomalies and the authenticity of the simulated attack generated by the generator are improved, and new signatures are generated to update the signature library of the attack identification engine after detecting unknown anomalies.

7. The method of claim 1, wherein, The constraint condition of the game model is that the sum of the CPU occupancy ratios of the two engines is less than or equal to 0.5, and 50% of the CPU resources are reserved for other system tasks, and the reward function of the game model is the weighted sum of the detection accuracy and resource consumption; The particle swarm game optimization algorithm aims to maximize the total revenue of the two engines, and converges to the Nash equilibrium point through iterative updates of particle speed and position.

8. The method of claim 7, wherein, The lightweight tasks are executed on the edge, and the compute-intensive tasks are executed in the cloud; the method of cloud-edge task offloading is: processing lightweight tasks on the edge side, including traffic collection, feature encoding, and real-time detection, and only uploading the input data of intensive tasks to the cloud; the cloud side uses GPU to process intensive tasks, including knowledge graph updating and global parameter synchronization, and only issues the processing result to the edge.

9. A deep learning based abnormal traffic detection and attack identification system characterized in that, It comprises: A data acquisition module acquires real-time raw traffic data from network devices; A preprocessing module cleans, normalizes, and formats the raw traffic data to generate a structured data set; A feature extraction module extracts statistical features and signature features from the structured data set in parallel, the statistical features calculate traffic distribution indicators through a time window, and the signature features generate behavior fingerprints based on an attack rule library, and the statistical features are standardized to eliminate dimensional effects, and the standardized statistical features and signature features are output; A cross-modal semantic alignment module deploys a traffic bidirectional encoder representation model and a signature feature vector embedding model, the former encodes the standardized statistical features into statistical feature semantic vectors, and the latter encodes the signature features into signature feature semantic vectors, so that the two types of vectors are in the same semantic space to eliminate the semantic gap; A knowledge graph construction module takes the two types of semantic vectors as graph nodes, calculates the semantic similarity between nodes through a k-nearest neighbor algorithm, and selects high-similarity nodes to construct an initial knowledge graph. The causal reinforcement correlation reasoning module calculates the semantic correlation weight between nodes by using a graph attention network, and uses a time series causal discovery algorithm and a transfer entropy analysis to analyze the causal relationship and strength between nodes, fuses semantic and causal information to update the edge weight of the knowledge graph, and realizes the dynamic evolution of the graph by using an incremental learning and a federated learning mechanism; The intelligent engine optimization module includes an anomaly detection engine based on unsupervised learning and an attack recognition engine based on supervised learning, dynamically schedules a subset of attack signatures through reinforcement learning to improve efficiency, and constructs an adversarial feedback loop through a generative adversarial network to generate simulated attacks, detect unknown anomalies and update the signature library, realizing the co-evolution of the double engines; The cloud-edge collaborative resource scheduling module models the system resource allocation as a Nash equilibrium problem by using a game theory model, and solves the optimal resource allocation strategy by using an optimization algorithm, and intelligently offloads tasks and synchronizes results between edge nodes and the cloud according to the calculation-intensive degree of the tasks; The result output module performs attack chain reasoning based on the final knowledge graph and generates a visual report, outputs graded alarm information according to the correlation weight and attack severity level, and generates a system resource utilization report.

10. The system of claim 9, wherein, The feature extraction module uses a sliding window statistical method to extract the number of packets per second, the average packet length, the retransmission rate, the average connection duration and the connection establishment success rate as statistical features; And from the public attack rule library, the attack feature code, the abnormal port and the abnormal protocol are extracted as signature features by regular matching; The cross-modal semantic alignment module uses a TraffBERT model to encode the statistical features, and uses a Sig2Vec model to embed the signature features, the parameters of the TraffBERT model are obtained by pre-training on normal traffic, and the parameters of the Sig2Vec model are obtained by training on the Snort rule library; The knowledge graph construction module calculates the semantic similarity between nodes using cosine similarity, initializes by first constructing an edgeless graph, then constructs edges for each node by selecting the node with the highest similarity, and forms an initial knowledge graph; The causal reinforcement correlation reasoning module uses the PCMCI algorithm for conditional independence test and significance calculation to discover reliable causal relationships, and uses transfer entropy to quantify causal strength, and finally weights and fuses the semantic similarity weight and the causal reliability score into the final edge weight; In the intelligent engine optimization module, the anomaly detection engine uses a variational autoencoder model, and the attack recognition engine uses a convolutional neural network model; The reinforcement learning uses a deep Q network to select the optimal signature subset by taking detection accuracy and resource consumption as the reward function; the generative adversarial network takes the attack recognition engine as the generator and the anomaly detection engine as the discriminator; The cloud-edge collaborative resource scheduling module uses a particle swarm optimization algorithm to solve the Nash equilibrium point to obtain the optimal CPU resource ratio allocated to the double engines, and according to the GPU acceleration ratio, the traffic collection, feature encoding and real-time detection tasks are deployed on the edge side, and the knowledge graph update, global signature library synchronization and complex graph reasoning tasks are offloaded to the cloud. The result output module traverses edges in the knowledge whose weights are higher than a set threshold to reconstruct an attack chain, and divides three alarm priorities of high, medium and low according to a causal strength of the attack chain and an attack type label.

Images