A method and system for adaptive anomaly detection in dynamic networks

Abstract

This invention claims protection for an adaptive anomaly detection method and system in dynamic networks, comprising the following steps: S1: Constructing a dynamic adjacency tensor sequence based on network traffic logs and performing exponential smoothing denoising; S2: Aggregating local features using a graph attention network and calculating node embedding vectors by combining high-order modularity constraints; S3: Constructing a path semantic representation vector by fusing the node embedding sequence and dynamic edge weight features; S4: Parallel calculation of a multi-dimensional comprehensive anomaly score including reconstruction error and distribution deviation; S5: Calculating an adaptive judgment threshold based on the historical statistical distribution of the comprehensive score, and detecting abnormal paths in the network based on this threshold. This invention solves the problems of low accuracy and lack of adaptability in existing technologies for identifying anomalies in dynamic networks, effectively improving the accuracy of detecting complex anomalies such as DDoS attacks.

Description

Technical Field

[0001] This invention belongs to the field of network data analysis and intelligent security monitoring technology, specifically relating to an adaptive anomaly detection method and system in dynamic networks. Background Technology

[0002] With the rapid development of communication networks, IoT systems, and online platforms, dynamic networks have gradually become an important data structure for characterizing entity interactions, and their node relationships and edge weights change continuously over time. In applications such as intrusion detection, abnormal traffic monitoring, and device behavior analysis, accurately detecting abnormal patterns in dynamic networks has become a key technology for ensuring system security. However, the time-varying characteristics and complex topology of dynamic networks pose significant challenges to anomaly detection, especially when facing highly concealed and large-scale structural attacks, where existing detection methods often prove ineffective.

[0003] Currently, anomaly detection in dynamic networks primarily relies on capturing statistical characteristics of network traffic or changes in network topology. Early solutions typically used sketches of random structures or statistical features of flow tables, focusing on low-order metrics such as node degree and edge weight distribution. However, these methods are essentially shallow statistics on local network information, lacking the ability to perceive global topological semantics. When attackers use multi-hop nodes as stepping stones to construct covert attack links spanning multiple communities, these local-perspective-based detection methods often produce false negatives because they cannot penetrate the deep network community structure.

[0004] To improve detection accuracy, deep learning-based methods have become a research hotspot in recent years, particularly graph neural networks (GNNs), multi-view contrastive learning, and Transformer temporal modeling. While these methods have made significant progress in extracting node features and capturing temporal dependencies, they still have limitations in practical applications. Most existing deep learning models still focus on "node-level" anomaly detection, tending to discover anomalies by comparing node representations in different views or calculating reconstruction errors of time slices, while neglecting joint modeling of "path-level" interaction semantics. Furthermore, dynamic networks exhibit significant traffic fluctuations. Existing detection models often rely on fixed thresholds or simple statistical distributions in anomaly judgment, making it difficult to dynamically adjust the detection level according to real-time network load fluctuations. This leads to a large number of false alarms during peak traffic periods and the potential to miss minor attacks during off-peak periods. In summary, existing anomaly detection technologies generally suffer from a lack of structural awareness and environmental adaptability when dealing with complex dynamic network environments. Therefore, how to design an anomaly detection method that can integrate high-order topology and path-level semantic features and has environmental adaptability has become a key problem that urgently needs to be solved in the current cybersecurity field. Summary of the Invention

[0005] The purpose of this invention is to provide an adaptive anomaly detection method and system in dynamic networks, solving the problem of accurate identification of abnormal paths and abnormal nodes in dynamic networks using existing technologies. Specifically, it includes the following steps:

[0006] S1: Construct a dynamic adjacency tensor sequence reflecting network temporal changes based on network traffic logs, and use an exponential smoothing mechanism to denoise the edge weight time series to obtain smooth dynamic network data that retains long-term trends.

[0007] S2: Based on the smooth dynamic network data obtained in S1, the local features of nodes are aggregated using a graph attention network and the global topology is constrained by a high-order modularity loss function to calculate the node embedding vector containing deep semantic information.

[0008] S3: Based on the node embedding vector obtained in S2, sample the multi-hop paths in the network, and use the feature aggregation mechanism to concatenate and fuse the node embedding sequence on the path with the dynamic edge weight features to construct the path semantic representation vector;

[0009] S4: Based on the path semantic representation vector obtained in S3, the reconstruction error score, semantic deviation score and contrast distribution score based on the autoencoder are calculated in parallel, and a multidimensional comprehensive anomaly score is calculated using a weighted fusion strategy.

[0010] S5: Calculate the adaptive judgment threshold based on the historical statistical distribution of the comprehensive anomaly score obtained in S4, and detect abnormal paths in the network based on the threshold to determine whether the path is subject to DDoS attacks or other abnormal behaviors.

[0011] Further, in step S1, the construction and denoising of the dynamic adjacency tensor sequence specifically includes: stacking network traffic snapshots within consecutive time windows to form a three-dimensional tensor to capture the time-varying characteristics of the network topology; and for the time series of edge weights, using the formula... Perform exponential smoothing updates, where This is a smoothing coefficient, used to suppress sudden high-frequency noise caused by network jitter, ensuring that subsequent models focus on identifying long-term structural change trends.

[0012] Further, in step S2, the calculation of the node embedding vector specifically includes: using a graph attention network (GAT) to perform weighted aggregation of the first-order neighborhood of the node to capture the local microstructure; and simultaneously constructing... The higher-order modularity matrix is ​​calculated, and by minimizing the higher-order modularity loss function, the distribution of node embeddings in the feature space is forced to reflect their position in the feature space. The community affiliation characteristics within the neighborhood of the model enable it to identify long links or hidden ring structures across communities, providing a global macroscopic perspective.

[0013] Furthermore, in step S3, the construction of the path semantic representation vector specifically includes: extracting the static backbone features of the path (i.e., the embedding sequence of all nodes on the path) and the dynamic edge weight features (i.e., the dynamic pattern extracted by one-dimensional convolution of the edge weight sequence); using a multilayer perceptron (MLP) to concatenate and fuse the static backbone features and the dynamic edge weight features to generate a unified path-level high-dimensional vector, thereby realizing the joint modeling of "node identity" and "communication behavior".

[0014] Furthermore, in step S4, the multidimensional comprehensive anomaly score is specifically composed of the following three weighted parts:

[0015] 1. Reconstruction error scoring: The trained autoencoder is used to compress and restore the path vector, and the Euclidean distance before and after restoration is calculated to detect anomalies that disrupt the regularity of the network structure.

[0016] 2. Semantic Deviation Score: Calculates the distance between the current path vector and the average historical normal behavior of that path, used to detect abrupt changes in behavior patterns;

[0017] 3. Comparison of distribution scores: Calculate the distance between the current path vector and the center of the normal sample cluster in the feature space to detect outlier patterns outside the distribution.

[0018] Furthermore, in step S5, the calculation of the adaptive judgment threshold specifically includes: maintaining a sliding window that records historical comprehensive scores; dynamically determining the current anomaly judgment threshold using a quantile modeling method; and judging an anomaly when the real-time score exceeds the threshold. This mechanism enables the threshold to automatically rise and fall with the tidal fluctuations of network traffic (such as peak and off-peak periods), thereby maintaining stable detection sensitivity under different load environments.

[0019] The beneficial effects of this invention are as follows:

[0020] (1) Compared with existing technologies that rely solely on first-order adjacency relationships, this invention introduces higher-order modularity constraints, enabling the model to see through deep network community structures. This gives the system stronger identification capabilities and detection coverage when facing scenarios that utilize complex topologies, such as multi-hop jumpers and ring propagation links, to carry out covert attacks.

[0021] (2) This invention innovatively integrates the "topological identity of a node" and the "dynamic change of edge weights" into a unified path vector. Compared with the traditional method of analyzing traffic statistics in isolation, this scheme can accurately depict the complete propagation path of attack traffic in the network, effectively solving the problem that simple traffic threshold detection is difficult to distinguish between normal burst traffic and attack traffic.

[0022] (3) By combining three scoring mechanisms in parallel—reconstruction error (structure lookup), semantic deviation (behavior lookup), and comparative distribution (pattern lookup)—this invention avoids the limitations of a single indicator. Combined with exponential smoothing denoising, it effectively reduces false alarms caused by data noise or slight local jitter, significantly improving the detection accuracy of the system. Attached Figure Description

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0024] Figure 1 This is a flowchart of the method in a specific embodiment of the present invention.

[0025] Figure 2 A flowchart illustrating feature modeling in a specific embodiment of the present invention.

[0026] Figure 3 This is a flowchart of anomaly detection in a specific embodiment of the present invention. Detailed Implementation

[0027] The following specific examples illustrate the embodiments of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. The technical solution of the present invention will be further described below in conjunction with specific embodiments and accompanying drawings.

[0028] This invention proposes an adaptive anomaly detection method and system in dynamic networks, the specific method flow of which is described below:

[0029] In this embodiment, in step S1, the collected network traffic logs are serialized using the following method:

[0030] S11: Given a time series For the sampling time of the network snapshot, any two nodes and Changes in edge weights (such as communication frequency and traffic volume) between them are recorded as follows It should be noted that, in order to capture the evolution of the network over time, we stacked the adjacency matrices at these moments to construct a three-dimensional tensor. This is equivalent to taking a series of "CT scans" of the network, with each slice corresponding to the network state at a given moment.

[0031] S12: To address the common network jitter in the original data, this embodiment uses an exponential smoothing function to process the edge weights. The calculation formula is as follows: ,in This is the smoothing coefficient. Furthermore, existing techniques typically use raw time-slice data directly, which is susceptible to high-frequency noise interference such as sudden packet loss and retransmissions, leading to false alarms. This step, through smoothing, effectively suppresses high-frequency disturbances, preserves the long-term trend of edge weight changes, and allows subsequent models to focus on identifying genuine structural changes.

[0032] In this embodiment, in step S2, the following method is used to generate an embedding vector that can represent the deep topological role of a node:

[0033] S21: Aggregate the neighborhood features of nodes using a Graph Attention Network (GAT). The calculation formula is as follows: ,in Represents a node The neighborhood group, This is the weight matrix. The attention coefficient is calculated using the following formula: This coefficient dynamically reflects the importance of neighboring nodes to the central node. Compared to the average aggregation of traditional graph convolutional networks (GCNs), the attention mechanism can more accurately characterize local structural features and distinguish the influence of different neighbors.

[0034] S22: Construction Module degree matrix And introduce a higher-order modularity loss function: Furthermore, this loss function mandates that the generated nodes be embedded... and Maintaining proximity in high-order topologies allows the model to have a global perspective, enabling it to identify covert attack chains spanning multiple communities and overcome the limitations of local perspectives.

[0035] S23: Based on the current embedding Clustering to obtain pseudo-labels Construct InfoNCE contrastive loss This further enhances the distinguishability of the embedded information.

[0036] In this embodiment, in step S3, semantic modeling is performed on the multi-hop paths in the network, elevating the features of "points" to the features of "lines".

[0037] S31: For the node pair to be detected According to the preset strategy, the sampling length is path set Each path is represented as a sequence of nodes. .

[0038] S32: Calculate the characteristics of nodes and edge weights, and the fusion of the two characteristics.

[0039] First, embed sequences into the nodes along the path. Attention aggregation is performed to obtain path backbone features. .

[0040] Secondly, the time series of edge weights on the path One-dimensional convolution (Conv1D) is used to extract dynamic patterns and obtain edge weight features. .

[0041] Finally, calculate the fusion of the two: It should be noted that traditional methods often analyze node attributes (such as IP reputation) or edge weight statistics (such as traffic thresholds) in a fragmented manner. This invention organically integrates these two aspects through the aforementioned formula, generating a path representation that contains complete semantics. This allows the system to not only perceive the "volume" of traffic but also the "path structure" through which the traffic passes, thereby effectively identifying complex behaviors that use normal nodes for jump attacks.

[0042] In this embodiment, step S4 involves calculating a multidimensional anomaly score. To comprehensively capture anomalies, this embodiment innovatively introduces a three-way parallel scoring mechanism:

[0043] (1) Reconstruction error score : Utilizing the reconstruction capability of autoencoders to detect anomalies that disrupt the regularity of network structure;

[0044] (2) Path semantic deviation score : Calculate the distance between the current path and the historical average to detect anomalies that deviate from historical behavior patterns;

[0045] (3) Comparison of distribution anomaly scores Used to detect outliers that are far from normal clusters in the feature space.

[0046] Furthermore, the comprehensive score calculation formula is as follows: Compared to existing technologies that rely on a single indicator, this multi-dimensional scoring mechanism significantly improves the robustness of detection and reduces misjudgments caused by fluctuations in a single feature.

[0047] In this embodiment, step S5 employs quantile modeling to dynamically adjust the judgment threshold based on the statistical distribution of scores within the sliding window. And based on this adaptive threshold, it determines the abnormal path and outputs the detection result. When When the path is deemed abnormal, this mechanism ensures that the system maintains stable detection performance during both peak network traffic periods (high noise) and off-peak periods (low baseline), effectively balancing the false alarm rate and the false negative rate.

[0048] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An adaptive anomaly detection method in dynamic networks, characterized in that, The method specifically includes the following steps: S1: Construct a dynamic adjacency tensor sequence reflecting network temporal changes based on network traffic logs, and use an exponential smoothing mechanism to denoise the edge weight time series to obtain smooth dynamic network data that retains long-term trends. S2: Based on the smooth dynamic network data obtained in S1, the local features of nodes are aggregated using a graph attention network and the global topology is constrained by a high-order modularity loss function to calculate the node embedding vector containing deep semantic information. S3: Based on the node embedding vector obtained in S2, sample the multi-hop paths in the network, and use the feature aggregation mechanism to concatenate and fuse the node embedding sequence on the path with the dynamic edge weight features to construct the path semantic representation vector; S4: Based on the path semantic representation vector obtained in S3, the reconstruction error score, semantic deviation score and contrast distribution score based on the autoencoder are calculated in parallel, and a multidimensional comprehensive anomaly score is calculated using a weighted fusion strategy. S5: Calculate the adaptive judgment threshold based on the historical statistical distribution of the comprehensive anomaly score obtained in S4, and detect abnormal paths in the network based on the threshold to determine whether the path is subject to DDoS attacks or other abnormal behaviors.

2. The adaptive anomaly detection method in a dynamic network according to claim 1, characterized in that, The construction of the dynamic adjacency tensor sequence in step S1 specifically includes: stacking network traffic snapshots within a continuous time window to form a three-dimensional tensor; the exponential smoothing mechanism specifically includes: updating the time series of edge weights using a weighted moving average formula, wherein the smoothing coefficient is used to adjust the degree of dependence on current traffic fluctuations and historical trends in order to suppress high-frequency noise interference.

3. The adaptive anomaly detection method in a dynamic network according to claim 1, characterized in that, Step S2 specifically includes: capturing the first-order neighborhood features of nodes using a graph attention network (GAT); constructing... The higher-order modularity matrix is ​​used to characterize the community affiliation of nodes in a multi-hop range, and the node embedding vector is constrained by minimizing the higher-order modularity loss function to maintain the consistency of the higher-order topology in the feature space.

4. The adaptive anomaly detection method in a dynamic network according to claim 1, characterized in that, The feature aggregation mechanism described in step S3 specifically includes: extracting the embedding vector sequence of all nodes on the path as static backbone features; using a one-dimensional convolutional neural network (Conv1D) to extract the dynamic pattern of the edge weight time series as dynamic edge weight features; and using a multilayer perceptron (MLP) to concatenate and fuse the static backbone features and dynamic edge weight features to generate a path semantic representation vector.

5. The adaptive anomaly detection method in a dynamic network according to claim 1, characterized in that, The multidimensional comprehensive anomaly score mentioned in step S4 is specifically composed of the following three weighted parts: the reconstruction error score obtained by compressing and restoring the path vector based on the autoencoder, the semantic deviation score obtained based on the distance between the path vector and the historical normal mean, and the comparative distribution score obtained based on the distance between the path vector and the center of the normal sample cluster.

6. The adaptive anomaly detection method in a dynamic network according to claim 1, characterized in that, The calculation of the adaptive judgment threshold in step S5 specifically includes: maintaining a sliding window that records historical comprehensive anomaly scores, dynamically calculating the current anomaly judgment water level using the quantile modeling method, and judging an abnormal path when the real-time score exceeds the water level.

7. An anomaly detection system for implementing the adaptive anomaly detection method in a dynamic network as described in claim 1, characterized in that, The system includes: The dynamic tensor construction module is used to execute step S1, which is responsible for constructing dynamic adjacency tensors and performing exponential smoothing denoising. The node embedding generation module is used to execute step S2 and is responsible for generating node vectors by combining the graph attention mechanism and the higher-order modularity constraint. The path semantic representation module is used to execute step S3 and is responsible for fusing node sequences and dynamic edge weights to generate path representations. The multidimensional anomaly scoring module is used to execute step S4, and is responsible for parallel calculation of multiple anomaly scores and weighted fusion. The adaptive threshold detection module is used to execute step S5, which is responsible for calculating the dynamic threshold and determining abnormal results.

8. The adaptive anomaly detection system in a dynamic network according to claim 7, characterized in that, The dynamic tensor construction module includes a serialization unit for acquiring traffic snapshots and a noise filtering unit for performing exponential smoothing calculations.

9. The adaptive anomaly detection system in a dynamic network according to claim 7, characterized in that, The node embedding generation module has a built-in graph attention network computing unit and a high-order modularity optimization unit, which are used to extract and fuse the local and global topological features of the network.

10. An adaptive anomaly detection system in a dynamic network according to claim 7, characterized in that, The multidimensional anomaly scoring module includes a parallel reconstruction error calculation unit, a semantic deviation calculation unit, and a contrast distribution calculation unit, as well as a weighted fusion unit for generating a comprehensive score.

Images