A network anomaly rapid detection method based on a knowledge distillation framework

Abstract

This invention belongs to the field of network anomaly detection technology, and its purpose is to provide a fast network anomaly detection method based on a knowledge distillation framework. This invention improves detection speed and solves the problem of a single knowledge source in knowledge distillation, resulting in high detection accuracy. Specifically, this invention uses knowledge distillation to transfer knowledge from a teacher model with a large number of parameters and layers to a student model with a small number of parameters and layers. The trained student model is then used for network anomaly detection. Compared to traditional anomaly detection schemes, this invention avoids the problem of using deep neural networks with a large number of parameters for network anomaly detection, resulting in faster detection speed. Furthermore, this invention introduces a multi-student model structure based on response-based knowledge distillation, which helps alleviate the student model's dependence on the teacher model, solves the problem of a single knowledge source for the student model in knowledge distillation, and improves the accuracy of network anomaly detection using the trained student model.

Description

Technical Field

[0001] This invention belongs to the field of network anomaly detection technology, specifically relating to a fast network anomaly detection method based on a knowledge distillation framework. Background Technology

[0002] Network anomaly detection plays a crucial role in modern cybersecurity. As cyberattacks become increasingly sophisticated, timely detection and response to potential anomalies are essential for ensuring the security of information systems. Anomaly detection can help identify issues including malicious attacks, data breaches, system failures, and abnormal user behavior, preventing data loss, service interruptions, and other serious security incidents. Through effective network anomaly detection, organizations can identify potential threats at an early stage, take necessary preventative measures, reduce security risks, and improve system stability and reliability.

[0003] In 2014, Sakurada et al. proposed a machine learning-based method for network anomaly detection in their paper "Anomaly detection using autoencoders with nonlinear dimensionality reduction" (Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis, pp. 4-11). This method aimed to model the complex correlations of multivariate time series in a simple linear manner. With the rapid development of deep learning and its excellent nonlinear fitting capabilities, in 2019, Su et al. proposed a recurrent neural network-based method for anomaly detection in multivariate time series in their paper "Robust anomaly detection for multivariate time series through stochastic recurrent neural network". However, the above approach has not yet clearly modeled the complex spatial dependencies between different network devices.

[0004] In 2019, authors Ngo et al., in their paper "Fence GAN: Towards better anomaly detection" (2019, IEEE 31st International Conference on tools with artificial intelligence, pp. 141-148; Ngo PC, Winarto AA, Kou CKL, et al. Fence GAN: Towards better anomaly detection[C]. 2019 IEEE 31st International Conference on tools with artificial intelligence (ICTAI). IEEE, 2019: 141-148.), discovered that the loss of traditional GANs (Generative Adversarial Networks) does not directly correspond to the anomaly detection target, and made simple modifications to the GAN loss. However, the training process of GANs is relatively complex, requiring careful adjustment of many hyperparameters, such as the learning rate and the number of network layers; in addition, GANs tend to be unstable, and may experience pattern collapse or pattern degeneration in the early stages of training. Furthermore, the GAN network requires the complete operation of the generator and possible additional computations, which cannot meet the demands of fast detection speeds.

[0005] In 2021, Henry Clausen et al. proposed CBAM, a contextual network anomaly detection model based on deep bidirectional LSTM networks, in the paper "CBAM: A Contextual Model for Network Anomaly Detection" (Computers, Vol. 10, p. 79; Clausen H, Grov G, Aspinall D. Cbam: A contextual model for network anomaly detection[J].Computers, 2021, 10(6): 79.). CBAM learns short-term sequence patterns in network traffic, mapping traffic events to a discrete set of states based on protocol, port, direction, and size. It is trained in a self-supervised manner without labeling attack traffic to reduce the false positive rate and maintain robustness to concept drift. However, the spatial attention mechanism used in CBAM mainly captures local spatial information through two-dimensional convolution, which limits the extraction of spatial features. Furthermore, the attention mechanism requires the introduction of additional attention parameters, increasing computational overhead.

[0006] In 2022, Chen Aiguo et al. proposed an efficient network anomaly detection algorithm using Deep Belief Network (DBN) and Long Short-Term Memory (LSTM) in their paper "An efficient network behavior anomaly detection using a hybrid DBN-LSTM network" (Computers & Security, Vol. 114, p. 102600). This algorithm automatically extracts features and reduces data dimensionality using DBN, then uses LSTM to capture time-series information to classify network behavior data. However, this method involves using DBN for dimensionality reduction, which results in the loss of some data information, ultimately leading to poor model performance in real-world applications. Furthermore, DBN is sensitive to the distribution and scale of the input data; changes in the distribution or lack of diversity in the input data can affect its performance and cause the learned features to become invalid.

[0007] In 2023, Gabriele Gemmi et al. proposed four machine learning methods for network anomaly detection based on different principles in their paper "Anomaly detection for fault detection in wireless community networks using machine learning" (Computer Communications, Vol. 202, pp. 191-203; Cerdà-Alabern L, Iuhasz G, Gemmi G. Anomalydetection for fault detection in wireless community networks using machine learning[J]. Computer Communications, 2023, 202: 191-203.): Principal Component Analysis (PCA), Isolation Forest, Cluster-Based Local Anomaly Factor (CBLOF), and Variational Autoencoder (VAE). VAE performed best, and for specific networks, using specific feature selection methods can improve detection accuracy. However, VAE has a high computational cost, and manually applying different feature selection methods to different networks significantly increases the overhead of network anomaly detection, hindering its widespread application and deployment.

[0008] In 2024, authors Zhou Xiuzhuang et al. proposed a self-supervised anomaly detection network, RealNet, in their paper "RealNet: A Feature Selection Network with Realistic Synthetic Anomaly for Anomaly Detection" (Proceedings of the IEEE / CVF conference on computer vision and pattern recognition CVPR, pp. 16699-16708). This network improves anomaly detection performance by combining three innovative components: SDAS (Synthetic Reality Anomaly Synthesis), AFS (Active Feature Selection), and RRS (Residual Retention Selection). However, RealNet is a self-supervised model, and the effectiveness of self-supervised learning is highly dependent on the designed pre-task. Furthermore, RealNet's AFS feature selection faces the challenge of excessive computational overhead for high-dimensional data.

[0009] In recent years, deep neural networks have achieved remarkable success in the field of network anomaly detection. This success is mainly attributed to the ability of deep neural networks to handle large-scale datasets and cope with unknown network anomalies. However, deploying these parameter-intensive deep neural networks on resource-constrained devices (such as embedded systems) faces significant challenges, primarily stemming from the computational overhead of deep neural networks.

[0010] To address the aforementioned issues, various model compression and acceleration techniques have been developed. Among these techniques, knowledge distillation has attracted considerable attention due to its effectiveness in training smaller student models based on larger teacher models. It helps to solve the problems of high computational overhead and slow detection speed caused by current deep neural network-based network anomaly detection.

[0011] In knowledge distillation, the type of knowledge transferred plays a crucial role in the performance of the student model. Typically, this knowledge falls into three categories: 1) response-based knowledge; 2) feature-based knowledge; and 3) relation-based knowledge. Response-based knowledge distillation focuses on the final output layer of the teacher model, aiming to train the student model to replicate the teacher model's predictions. Compared to feature-based knowledge distillation, response-based knowledge distillation is more capable of capturing global structural information, and its computational process is simpler than that of relation-based knowledge distillation. However, response-based knowledge distillation suffers from the problem of relying on a single knowledge source and is highly dependent on the teacher model.

[0012] Therefore, it is necessary to propose a solution for rapid network anomaly detection based on a knowledge distillation framework to address the shortcomings of traditional anomaly detection schemes, such as high computational overhead and slow detection speed, while also solving the problem of a single knowledge source in existing knowledge distillation methods. Summary of the Invention

[0013] The present invention aims to solve the above-mentioned technical problems to at least some extent, and provides a method for rapid network anomaly detection based on a knowledge distillation framework.

[0014] To achieve the above objectives, the present invention adopts the following technical solution:

[0015] In a first aspect, the present invention provides a method for rapid network anomaly detection based on a knowledge distillation framework, comprising:

[0016] Obtain the teacher model and a collection of student models, including multiple student models;

[0017] Obtain a network anomaly detection dataset; wherein, the network anomaly detection dataset includes multiple training samples;

[0018] Training samples are randomly selected from the network anomaly detection dataset and input into the teacher model and each student model of the student model set, respectively, so as to obtain the prediction values ​​of the teacher model and each student model of the student model set for the training samples.

[0019] Based on the predicted values ​​of the teacher model and each student model in the student model set for the training samples, the loss function value of each student model in the student model set is calculated.

[0020] The stochastic gradient value of each student model is calculated based on the loss function value of each student model, and backpropagation is performed on each student model in the student model set based on the stochastic gradient value of each student model to obtain an updated student model set including multiple updated student models.

[0021] The student model set is updated to the updated student model set, and training samples are randomly drawn again from the network anomaly detection dataset until the obtained loss function value is less than the preset threshold or the preset maximum number of iterations is reached, and then the final updated student model set is output.

[0022] The final updated set of student models will be used as the set of trained student models.

[0023] Based on the trained student model set, network anomaly detection processing is performed on the target network.

[0024] In one possible design, obtaining the teacher model includes:

[0025] Network data generated by multiple network devices is acquired, and a dynamic graph construction method is used in conjunction with a depth graph convolutional network model to learn the spatial dependencies between network devices from the network data, thereby constructing a spatial modeling module that matches the multiple network devices; wherein, the network data includes time series data generated by each network device;

[0026] Based on the time series data generated by each network device, the time dependency relationship within a single time series data is extracted, the time feature representation of each network device is generated, and then a time modeling module matching multiple network devices is constructed.

[0027] Based on the spatial modeling module and the temporal modeling module, a comprehensive spatiotemporal feature representation is generated, thereby constructing a spatiotemporal fusion module that matches multiple network devices;

[0028] The spatiotemporal fusion module is used to collaboratively train the spatial modeling module and the temporal modeling module to obtain the teacher model.

[0029] In one possible design, a dynamic graph construction method is employed, combined with a depth graph convolutional network model, to learn the spatial dependencies between network devices from the network data, thereby constructing a spatial modeling module that matches multiple network devices, including:

[0030] The network data is input into the dynamic graph construction module, so that the dynamic graph construction module updates the graph structure layer by layer based on the deep graph convolutional network model, thereby obtaining the graph structure between network devices and the node features of the graph structure.

[0031] The graph structure is as follows:

[0032] ;

[0033] In the formula, K This represents the total number of layers in the depth map convolutional network model.K For natural numbers greater than 1, X K-1 The first term of the depth map convolutional network model represents the... K -1 Node features generated by depth map convolutional network layers DGC ( ) represents the function in the dynamic graph construction module used to determine whether there is a connection relationship between nodes. R n×n express n OK n A matrix of columns n This refers to the total number of network devices.

[0034] The node characteristics of the graph structure are: X K ;in, X K The first term of the depth map convolutional network model represents the... K Node features generated by depth map convolutional network layers;

[0035] Obtain the node features of each depth graph convolutional network layer of the depth graph convolutional network model and combine them into a node feature set;

[0036] Obtain the projection vector, and based on the node feature set and the projection vector, obtain the implicit high-order dependencies between network devices;

[0037] Based on the higher-order dependencies and the node feature set, the spatial correlation characteristics of deep coupling between network devices are obtained;

[0038] Based on the graph structure, the node features of the graph structure, and the spatial correlation features, a spatial modeling module matching multiple network devices is constructed.

[0039] In one possible design, the depth map convolutional network model is composed of the first... k The expression for the node features generated by the depth map convolutional network layer is:

[0040] ;

[0041] In the formula, k ∈{2,3,……, K+ 1}, This represents a normalized symmetric graph structure. , A k-1 The dynamic graph construction module calculates the first... k- 1. Diagram structure D Indicates the first k- 1 diagram structure A k-1 The degree matrix,I Represents the identity matrix;

[0042] The expression for the node feature set is:

[0043] ;

[0044] in, X 0 This represents the preset initial node features of the depth map convolutional network model. d Indicates the preset node feature dimension;

[0045] The higher-order dependency relationship is as follows:

[0046] ;

[0047] In the formula, This indicates the higher-order dependency. Reshape ( ) represents a function used to adjust the dimensions of a matrix. This indicates the importance of each convolutional network layer in the depth map. , c Represents the projection vector. c ∈ R d×1 ;

[0048] The spatial correlation feature is as follows:

[0049] ;

[0050] In the formula, This indicates the spatial correlation feature. Represents the matrix multiplication symbol.

[0051] In one possible design, based on the time-series data generated by each network device, the temporal dependencies within a single time-series data point are extracted to generate temporal feature representations for each network device. This leads to the construction of a time modeling module that matches multiple network devices, including:

[0052] The time series data generated by each network device is divided into multiple patches, and the multiple patches of each network device are projected into a high-dimensional space to obtain the high-dimensional time series data of each network device.

[0053] Based on multiple patches for each network device, multiple new patches for each network device are obtained using the self-attention module, and the new patches for each network device are combined and processed to obtain new network data for each network device.

[0054] The sum of new network data and high-dimensional time series data of each network device is batch normalized to obtain the initial time feature representation of each network device;

[0055] A feedforward neural network is used to enhance the initial time feature representation of each network device to obtain the enhanced time feature representation of each network device.

[0056] The enhanced temporal feature representations of each network device are reprojected onto the initial dimension of the network data to obtain the temporal feature representations of each network device.

[0057] Based on the time characteristics of each network device, a time modeling module matching multiple network devices is constructed.

[0058] In a possible design, the high-dimensional time series data of any network device is represented as follows:

[0059] ;

[0060] In the formula, Z ∈ R N×d , Z= ( z 1, z 2,…, z i ,…, z N ), z i This refers to the first of a plurality of patches for any of the network devices. i One patch, z i ∈ R d , N This indicates the total number of patches. d This represents the preset node feature dimension. Z 'Indicates multiple patches for any of the network devices. Z '∈ R N×P , P Indicates the length of the patch. W Z This represents the preset weight matrix. W Z ∈ R P×d ;

[0061] Based on multiple patches for any network device, a new patch for any network device is obtained using a self-attention module, and the new patches for any network device are combined to obtain new network data for that network device, including:

[0062] Initialize relative position encoding parameters w pos = {w 1, w 2,…, w j-i ,…, w N-1};in, w j-i This indicates a patch for any of the network devices. i and patches j index, w j-i ∈ R d ;

[0063] The attention weights and normalized attention weights among the patches of any network device are calculated based on the relative position encoding parameters; wherein, the patches of any network device i and patches j The attention weights between them are:

[0064] ;

[0065] In the formula, For patching any of the network devices i and patches j Attention weights between them z j This refers to the first of a plurality of patches for any of the network devices. j One patch, z j ∈ R d , β Q and β K This represents the preset weight matrix parameters. β Q ∈ R d×d , β K ∈ R d×d , a ij This indicates a patch for any of the network devices. i and patches j The level of attention to embedding a ij =w j-i ;

[0066] The patch for any network device i and patches j The normalized attention weights between them are:

[0067] ;

[0068] In the formula, For any of the network devices, a patch i and patches j Normalized attention weights between them μ ie This indicates a patch for any of the network devices. i and patches e Attention weights between them;

[0069] Based on multiple patches for any network device and the normalized attention weights between each patch, multiple new patches for any network device are calculated; wherein, the patch corresponding to the first patch among the multiple patches of any network device is... i The new patch corresponding to each patch is:

[0070] ;

[0071] In the formula, For any of the multiple patches for the network device, the first i The new patch corresponding to each patch. β V This represents the preset weight matrix parameters. β V ∈ R d×d ;

[0072] Multiple new patches for any one of the network devices are combined to obtain new network data for that network device; wherein, the new network data for any one network device is represented as follows:

[0073] Z attn =( z '1, z '2,…, z ' i ,…, z ' N );

[0074] In the formula, Z attn ∈ R N×d ;

[0075] The initial time characteristics of any network device are represented as follows:

[0076] Y = BatchNorm ( Z + Z attn );

[0077] In the formula, Y This represents the initial time characteristics of any of the network devices. BatchNorm ( ) is the batch normalization function.

[0078] In a possible design, the enhanced time characteristics of any network device are represented as follows:

[0079] ;

[0080] In the formula, For any network device, the enhanced time characteristic representation is... BatchNorm ( ) is the batch normalization function. ; Y This represents the initial time characteristics of any of the network devices; , For any network device, the enhanced time characteristic representation is... Relu ( ) represents the activation function. W 1 and W 2 represents the preset weight matrix. b 1 and b 2 represent the weight matrix respectively. W 1 and W The corresponding bias is 2;

[0081] The time characteristics of any network device are represented as follows:

[0082] ;

[0083] In the formula, This represents the time characteristics of any of the network devices. Y 'This is the feature representation obtained by flattening the enhanced temporal feature representation of any of the network devices.' Y '∈ R 1×(N×d) , W Linear This represents the preset linear layer weight matrix. W Linear ∈ R (N×d)×d ; .

[0084] In one possible design, the spatiotemporal characteristics are represented as:

[0085] ;

[0086] In the formula, This represents the spatiotemporal features. , softmax ( ) represents the activation function. Y outThis refers to the time modeling module. H out This represents the spatial correlation characteristics in the spatial modeling module. W tp and W sp This represents the preset weight parameters. W tp R d×d , W sp R d×d , T Indicates the transpose symbol. d This indicates the preset node feature dimension.

[0087] In one possible design, the loss function value of any student model is calculated using the cross-entropy loss function and the KL divergence.

[0088] In one possible design, the first student model in the set... i The loss function value for the student model is:

[0089] ;

[0090] In the formula, For the student model set, the first... i The loss function value of the student model. a , b and c 'Indicates the preset training weights, K 'Indicates the total number of student models in the student model set. Indicates the first i Hard cross-entropy loss of a student model Indicates the first i The soft cross-entropy loss of the student model, Indicates the first i The predicted values ​​of the student model. In the student model set, the first... k The predicted values ​​of the student model. Indicates the first i KL divergence of a student model.

[0091] Secondly, this invention provides a network anomaly rapid detection system based on a knowledge distillation framework, comprising:

[0092] The initial model acquisition module is used to acquire the teacher model and a set of student models, including multiple student models.

[0093] The model training module is communicatively connected to the initial model acquisition module and is used to acquire the network anomaly detection dataset; wherein, the network anomaly detection dataset includes multiple training samples;

[0094] The model training module is further configured to randomly extract training samples from the network anomaly detection dataset and input the training samples into the teacher model and each student model of the student model set, so as to obtain the prediction values ​​of the teacher model and each student model of the student model set for the training samples.

[0095] The model training module is further configured to calculate the loss function value of each student model in the student model set based on the prediction values ​​of the teacher model and each student model in the student model set for the training samples.

[0096] The model training module is also used to calculate the stochastic gradient value of each student model based on the loss function value of each student model, and to perform backpropagation update on each student model in the student model set based on the stochastic gradient value of each student model, so as to obtain an updated student model set including multiple updated student models.

[0097] The model training module is also used to update the student model set to the updated student model set, and randomly extract training samples from the network anomaly detection dataset again until the obtained loss function value is less than a preset threshold or the preset maximum number of iterations is reached, and then output the final updated student model set.

[0098] The model training module is also used to use the final updated student model set as the trained student model set.

[0099] An anomaly detection module is communicatively connected to the model training module and is used to perform network anomaly detection processing on the target network based on the trained student model set.

[0100] Thirdly, the present invention provides an electronic device, comprising:

[0101] Memory, used to store computer program instructions; and,

[0102] A processor is configured to execute the computer program instructions to perform the operation of a network anomaly rapid detection method based on a knowledge distillation framework as described above.

[0103] Fourthly, the present invention provides a computer program product, including a computer program or instructions, wherein when the computer program or instructions are executed by a computer, they implement a network anomaly rapid detection method based on a knowledge distillation framework as described in any one of the above.

[0104] The beneficial effects of this invention are as follows:

[0105] This invention discloses a rapid network anomaly detection method based on a knowledge distillation framework, which improves detection speed and solves the problem of a single knowledge source in knowledge distillation, resulting in high detection accuracy. Specifically, in implementation, this invention uses knowledge distillation to transfer knowledge from a teacher model with a large number of parameters and layers to a student model with a small number of parameters and layers. The trained student model is then used for network anomaly detection. Compared to traditional anomaly detection schemes, this invention avoids the problem of using deep neural networks with a large number of parameters for network anomaly detection, resulting in faster detection speed. Furthermore, this invention introduces a multi-student model structure based on response-based knowledge distillation, which helps alleviate the dependence of student models on teacher models, solves the problem of a single knowledge source for student models in knowledge distillation, and improves the accuracy of network anomaly detection using the trained student model.

[0106] Other beneficial effects of the present invention will be further explained in the specific embodiments. Attached Figure Description

[0107] Figure 1 This is a flowchart of the network anomaly rapid detection method based on the knowledge distillation framework in Example 1;

[0108] Figure 2 This is a flowchart illustrating the graph structure update process in Example 1;

[0109] Figure 3 This is a schematic diagram illustrating the process of training a set of student models, including two student models, as exemplified in Example 1.

[0110] Figure 4 This is a block diagram of the network anomaly rapid detection model based on the knowledge distillation framework in Example 2. Detailed Implementation

[0111] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0112] Example 1:

[0113] This embodiment discloses a method for rapid network anomaly detection based on a knowledge distillation framework, which can be executed, but is not limited to, by a computer device or virtual machine with certain computing resources, such as a personal computer, smartphone, personal digital assistant or wearable device, or by a virtual machine.

[0114] like Figure 1 As shown, a fast network anomaly detection method based on a knowledge distillation framework may include, but is not limited to, the following steps:

[0115] S1. Obtain the teacher model and a set of student models including multiple student models; It should be understood that in this embodiment, both the teacher model and the student model are used for network anomaly detection, and they can also be referred to as the network anomaly detection teacher model and the network anomaly detection student model, respectively. For the sake of simplicity, they are simply referred to as the teacher model and the student model.

[0116] It should be noted that in this embodiment, the teacher model is a model with a large number of parameters and many layers, while the student models in the student model set are models with a small number of parameters and few layers, such as shallow convolutional neural networks (CNNs) and lightweight Transformer models. In addition, in this embodiment, the structures of multiple student models can be the same or different, depending on the design goals and requirements. For example, multiple student models with the same structure can be used to improve the robustness of student models or to improve accuracy through ensemble prediction. If the goal is to take into account diversity (different student models are good at handling different types of inputs or specific features), multiple student models can be designed with different architectures. In this embodiment, multiple student models use different types of student models or student models with different pre-training levels.

[0117] In this embodiment, before training the student model set, multiple student models in the student model set are pre-initialized to different initial conditions, so that multiple student models use different initial parameters, in order to avoid multiple student models getting trapped in the same local optimum, and at the same time, it can help increase the prediction diversity of multiple student models and the prediction performance in the application stage.

[0118] In step S1, the teacher model employs Dynamic Deep Graph Convolution with Enhanced Transformer Networks (DDGCT). Correspondingly, obtaining the teacher model includes:

[0119] S101. Obtain network data generated by multiple network devices, and use a dynamic graph construction method combined with a depth graph convolutional network model to learn the spatial dependencies between network devices from the network data, thereby constructing a spatial modeling module that matches the multiple network devices; wherein, the network data includes time-series data generated by each network device. It should be understood that network devices include servers, routers, switches, and personal computers, etc., and are not limited here.

[0120] It should be noted that network devices in a network have complex and non-linear topological connections, lacking effective representation. Therefore, this embodiment proposes a novel dynamic graph construction method to learn the dependencies between network devices through a data-driven approach. Specifically, in step S101, a dynamic graph construction method is used, combined with a depth graph convolutional network model, to learn the spatial dependencies between network devices from the network data, thereby constructing a spatial modeling module matching multiple network devices, including:

[0121] S1011. Input the network data into the Dynamic Graph Construction (DGC) module so that the DGC module updates the graph structure layer by layer based on the deep graph convolutional network model, thereby obtaining the graph structure between network devices and the node features of the graph structure.

[0122] Specifically, in this embodiment, the network data is represented as ,in l and n These represent the length of the input time series and the number of network devices, respectively. After acquiring the network data, it is standardized through a data preprocessing step and a length of [length missing] is used. Using a sliding window to generate fixed-length network data X '= .

[0123] Figure 2 This is a flowchart illustrating the process of updating the graph structure. Figure 2 The Dynamic Graph Construction (DGC) box in the figure represents the dynamic graph construction module, which includes an input layer and multiple processing layers (DGCLayer 1, ..., DGC LayerK-1 and DGC Layer K). Figure 2 The Deep Graph Convolution Network (DGCN) box in the figure represents the Deep Graph Convolution Network model, which includes K Deep Graph Convolution Network layers (DGCN Layer 1, ..., DGCN Layer K).

[0124] like Figure 2 As shown, in this embodiment, the dynamic graph construction module updates the graph structure layer by layer based on the depth graph convolutional network model, including: the dynamic graph construction module updates the graph structure according to the network data. X 0 Obtain the first graph structure between network devices A 1 and the first graph structure A 1 The first depth map convolutional network layer (i.e., ...) is input to the depth map convolutional network model. Figure 2 In the DGCN Layer 1), so that the first depth map convolutional network layer generates the first node features. X 1 The dynamic graph construction module is based on the characteristics of the first node. X 1 Obtain the second graph structure between network devices A 2 and the second diagram structure A 2 The second depth map convolutional network layer of the depth map convolutional network model is input so that the second depth map convolutional network layer generates the second node features. X 2 The dynamic graph construction module is based on the features of the (K-2)th node. X K-2 Obtain the K-1th graph structure between network devices A K-1 and the structure of the (K-1)th graph A K-1 Input the (K-1)th depth map convolutional network layer of the depth map convolutional network model so that the (K-1)th depth map convolutional network layer generates the (K-1)th node features. X K-1 The dynamic graph construction module is based on the features of the (K-1)th node. X K-1 Obtain the Kth graph structure between network devices A K and the Kth graph structure A K The Kth depth map convolutional network layer (i.e., ...) is input to the depth map convolutional network model. Figure 2 In the DGCN Layer K), so that the Kth depth map convolutional network layer generates the features of the Kth node. X K It should be understood that the structure of the Kth graph... A K That is, the final graph structure between network devices, the characteristics of the Kth node. XK That is, the node features of the final graph structure.

[0125] The graph structure is as follows:

[0126] ;

[0127] In the formula, K This represents the total number of layers in the depth map convolutional network model. K For natural numbers greater than 1, X K-1 The first term of the depth map convolutional network model represents the... K -1 Node features generated by depth map convolutional network layers DGC ( ) represents the function in the dynamic graph construction module used to determine whether there is a connection relationship between nodes. R n×n express n OK n A matrix of columns n This represents the total number of network devices.

[0128] Specifically, in this embodiment, the function It is implemented using a multilayer perceptron (MLP), which calculates the neighborhood weights between nodes in the graph structure and further uses a threshold function to determine whether there are connections between nodes, thus obtaining the final graph structure.

[0129] As an example, the first k -1 ( k ∈{2,3,……, K+ 1) Node features generated by depth map convolutional network layers X K-1 As the query matrix and key matrix, a function can be used at this time. DGC ( ) can be used to calculate the first k Nodes in a graph structure and (i.e., network equipment) and The adjacency weight between two points is defined by the following formula:

[0130] ;

[0131] In the formula, Sigmoid ( ) represents the Sigmoid activation function. Relu ( ) represents the ReLU activation function. and These represent the first and second layers of the depth map convolutional network model, respectively. k-1 Nodes generated by the depth map convolutional network layer and Node characteristics, and Indicates the first k The weight matrix of the depth map convolutional network layer has preset values. d This represents the preset node feature dimension. x This represents the dimension of the hidden layer in an MLP. T This represents the transpose symbol.

[0132] According to the k The adjacency weights between nodes in a graph structure can be derived from the adjacency weights of the nodes. k Graph structure A k It is an adjacency matrix, such as Figure 2 The adjacency matrix is ​​shown in the figure. Specifically, in this embodiment, the adjacency weights are used to determine the adjacency weights. Furthermore, a differentiable threshold function is used to determine whether a connection exists between network devices. Specifically, adjacency weights... It is normalized by a factor that is almost equal to itself but has no gradient, as shown below:

[0133] ;

[0134] In the formula, This indicates that during the training phase, the adjacent weights are used... The gradient separated in the middle, It is set to a preset minimum value, which is set to a very small value to prevent overflow.

[0135] In this embodiment, the adjacency matrix A k elements in The calculation method is as follows:

[0136] ;

[0137] In the formula, μ The preset threshold is set to 0.5 in this embodiment.

[0138] Furthermore, in this embodiment, a Hard Concrete distribution is introduced. Norm preserves key connections between nodes. During training, the model is pruned by causing weights to become zero to minimize redundant dependencies between network devices in the graph structure, while preventing the introduction of unnecessary information and overfitting. The expression for the norm is shown below:

[0139] ;

[0140] The implementation of this differentiable threshold function can regularize the gradient to a certain extent, thereby enabling the dynamic graph construction module to fine-tune the connection relationships between network devices.

[0141] Therefore, in this embodiment, the dynamic graph construction module can adaptively construct a graph structure between network devices so as to perform feature update steps between network devices based on the graph structure.

[0142] In this embodiment, the node characteristics of the graph structure are: X K ;in, X K The first term of the depth map convolutional network model represents the... K Node features generated by depth map convolutional network layers.

[0143] S1012. Obtain the node features of each depth graph convolutional network layer of the depth graph convolutional network model, and combine them into a node feature set.

[0144] Traditional graph convolutional network models achieve information aggregation between nodes through iterative propagation and transformation. Correspondingly, the traditional graph convolutional network model's first... k The expression for the node features generated by the graph convolutional network layer is:

[0145] ;

[0146] In the formula, k ∈{2,3,……, K+ 1}, This represents a normalized symmetric graph structure. , D' Representing the adjacency matrix A' k-1 The degree matrix, Represents the identity matrix. This represents the first generation of a traditional graph convolutional network model. k- Figure 1 shows the node features generated by the convolutional network layer. The first part represents the traditional graph convolutional network model. k- Figure 1 shows the trainable weight matrix of a convolutional network layer.

[0147] For the depth map convolutional network model proposed in this embodiment, the first... k The expression for the node features generated by the depth map convolutional network layer is:

[0148] ;

[0149] In the formula, k ∈{2,3,……, K+1}, This represents a normalized symmetric graph structure. , A k-1 The dynamic graph construction module calculates the first... k- 1. Diagram structure D Indicates the first k- 1 diagram structure A k-1 The degree matrix, I Represents the identity matrix;

[0150] The performance of graph convolutional network models is affected by the entanglement between graph convolutional transformations and propagation. Therefore, compared to traditional graph convolutional network models... k Compared to the expression for node features generated by graph convolutional network layers, the deep graph convolutional network model in this embodiment does not have a learnable weight matrix. .

[0151] Correspondingly, after After layer iteration, the node features of each depth map convolutional network layer of the depth map convolutional network model can be obtained, and a node feature set can be obtained accordingly. The expression of the node feature set is as follows:

[0152] ;

[0153] in, X 0 This represents the preset initial node features of the depth map convolutional network model. d This represents the preset node feature dimension; it is obvious that the node feature set... It contains node information from lower and deeper layers.

[0154] S1013. Obtain the projection vector, and based on the node feature set and the projection vector, obtain the implicit high-order dependencies between network devices.

[0155] In this embodiment, trainable projection vectors are used. c ∈ R d×1 The method adaptively determines which layers of the depth map convolutional network model are better at capturing node feature information. In this way, implicit high-order importance dependencies between network devices can be captured more effectively. Specifically, in step S1013 of this embodiment, the high-order dependencies are:

[0156] ;

[0157] In the formula, This indicates the higher-order dependency. Reshape ( ) represents a function used to adjust the dimensions of a matrix, used to... Reconstruct the three-dimensional tensor into a four-dimensional tensor for further computation; This indicates the importance of each convolutional network layer in the depth map, and is a value between 0 and 1. , c Represents the projection vector. c ∈ R d×1 , The features of each node are weighted and summed at different depth graph convolutional network layers. This weighted sum reflects the contribution of each layer to the node's feature representation.

[0158] S1014. Based on the higher-order dependency relationship and the node feature set, obtain the spatial correlation characteristics of deep coupling between network devices.

[0159] In step S1014, the spatial correlation feature is:

[0160] ;

[0161] In the formula, This indicates the spatial correlation feature. Represents the matrix multiplication symbol.

[0162] It should be noted that in this embodiment, after constructing the network device graph structure, a depth graph convolutional network model is used for information propagation between network devices. By employing a depth graph convolutional network model, the over-smoothing problem can be effectively alleviated, and the parameters of traditional graph convolutional network models can be reduced to prevent overfitting.

[0163] S1015. Based on the graph structure, the node features of the graph structure, and the spatial correlation features, a spatial modeling module matching multiple network devices is constructed.

[0164] S102. Based on the time series data generated by each network device, extract the time dependency relationship within a single time series data, generate the time feature representation of each network device, and then construct a time modeling module that matches multiple network devices.

[0165] In step S102, based on the time series data generated by each network device, the time dependencies within a single time series data point are extracted to generate a time feature representation for each network device, thereby constructing a time modeling module that matches multiple network devices, including:

[0166] S1021. Divide the time series data generated by each network device into multiple patches, and project the multiple patches of each network device into a high-dimensional space to obtain the high-dimensional time series data of each network device.

[0167] In step S1021, the time-series data generated by any network device is... After patching, the resulting data can be represented as follows: ;in, N This indicates the total number of patches. P The length of the patch is indicated and calculated as follows:

[0168] ;

[0169] In the formula, Indicates the floor function;

[0170] If the length of the time series data sequence is not divisible by the length of the patch, the last value at the end of the sequence will be padded before the patch is applied. S Next, maintain a consistent length. S Indicates the step size. S It is a natural number greater than 0. This approach ensures that the length of each patch is consistent and prevents information loss. If the step size... S Equal to the length of the patch P If the step size is [not specified], then the resulting patches are non-overlapping. S and the length of the patch P If they are not equal, then the step size... S This represents the non-overlapping region between two patches. By using patches, the transformer's ability to capture comprehensive semantic information in a sequence can be significantly improved, thus effectively modeling long-term dependencies in time series data.

[0171] Further, in step S1021, the high-dimensional time series data of any network device is represented as follows:

[0172] ;

[0173] In the formula, Z ∈ R N×d , Z= ( z 1, z 2,…, z i ,…, z N ), z i This refers to the first of a plurality of patches for any of the network devices. i One patch, z i ∈ R d , N This indicates the total number of patches. d This represents the preset node feature dimension.Z 'Indicates multiple patches for any of the network devices. Z '∈ R N×P , P Indicates the length of the patch. W Z This represents a pre-defined trainable weight matrix. W Z ∈ R P×d .

[0174] S1022. Based on the multiple patches for each network device, obtain multiple new patches for each network device using the self-attention module, and combine the new patches for each network device to obtain new network data for each network device.

[0175] Specifically, in step S1022, based on multiple patches for any network device, a new patch for any network device is obtained using a self-attention module, and the new patch for any network device is combined to obtain new network data for the network device, including:

[0176] S10221. Initialize relative position encoding parameters w pos = { w 1, w 2,…, w j-i ,…, w N-1};in, w j-i This indicates a patch for any of the network devices. i and patches j index, w j-i ∈ R d .

[0177] S10222. Calculate the attention weights and normalized attention weights among the patches of any network device based on the relative position encoding parameters; wherein, the patches of any network device and patches The attention weights between them are:

[0178] ;

[0179] In the formula, For patching any of the network devices i and patches j Attention weights between them z jThis refers to the first of a plurality of patches for any of the network devices. j One patch, z j ∈ R d , β Q and β K This represents the preset trainable weight matrix parameters. β Q ∈ R d×d , β K ∈ R d×d , a ij This indicates a patch for any of the network devices. i and patches j The level of attention to embedding a ij =w j-i ;in, and These represent the traditional attention mechanism for the interaction between queries and keys in a pre-defined self-attention mechanism. and These represent the relative position measures of the query and the key in the preset self-attention mechanism, respectively.

[0180] The patch for any network device and patches The normalized attention weights between them are:

[0181] ;

[0182] In the formula, For patching any of the network devices i and patches j Normalized attention weights between them μ ie This indicates a patch for any of the network devices. i and patches e Attention weights between them;

[0183] S10223. Based on multiple patches of the network device and the normalized attention weights between each patch, multiple new patches of the network device are calculated; specifically, in this embodiment, the normalized attention weights are applied to perform a weighted summation on each patch to calculate multiple new patches of the network device; wherein, the patch with respect to the multiple patches of the network device is... i The new patch corresponding to each patch is:

[0184] ;

[0185] In the formula, For any of the multiple patches for the network device, the first i The new patch corresponding to each patch. β V This represents the preset trainable weight matrix parameters. β V ∈ R d×d .

[0186] S10224. Combine multiple new patches for any one of the network devices to obtain new network data for the any one of the network devices; wherein, the new network data for any one of the network devices is represented as:

[0187] Z attn =( z '1, z '2,…, z ' i ,…, z ' N );

[0188] In the formula, Z attn ∈ R N×d .

[0189] Based on step S1022 of this embodiment, a new self-attention mechanism was developed, which combines relative position encoding and increases the interaction between the relative position encodings of each query, key and value in the self-attention mechanism, and can more accurately simulate the temporal order relationship of time series data.

[0190] S1023. Perform batch normalization on the sum of the new network data and high-dimensional time series data of each network device to obtain the initial time feature representation of each network device.

[0191] Specifically, in step S1023, the initial time characteristics of any network device are represented as follows:

[0192] Y = BatchNorm ( Z + Z attn );

[0193] In the formula, Y This represents the initial time characteristics of any of the network devices. BatchNorm ( ) is the batch normalization function.

[0194] It should be noted that in step S1023 of this embodiment, batch normalization is used instead of layer normalization, which can reduce the impact of outliers. Simultaneously, by introducing residual connections (high-dimensional time series)... Z attn This can prevent gradient vanishing, thereby learning deeper dependencies between network devices.

[0195] S1024. The initial time feature representations of each network device are enhanced using a feedforward neural network to obtain the enhanced time feature representations of each network device.

[0196] Specifically, in step S1024, the enhanced time characteristics of any network device are represented as follows:

[0197] ;

[0198] In the formula, For any network device, the enhanced time characteristic representation is... BatchNorm ( ) is the batch normalization function. ; Y This represents the initial time characteristics of any of the network devices; , For any network device, the enhanced time characteristic representation is... Relu ( ) represents the activation function. W 1 and W 2 represents the preset weight matrix. b 1 and b 2 represent the weight matrix respectively. W 1 and W The bias corresponding to 2.

[0199] S1025. The enhanced temporal feature representations of each network device are reprojected onto the initial dimension of the network data to obtain the temporal feature representations of each network device. Specifically, in this embodiment, a linear layer is used to reproject the enhanced temporal feature representations of each network device onto the same dimension as the input to obtain a univariate output.

[0200] Specifically, in step S1025, the time characteristics of any network device are represented as follows:

[0201] ;

[0202] In the formula, This represents the time characteristics of any of the network devices. Y 'This is the feature representation obtained by flattening the enhanced temporal feature representation of any of the network devices.' Y '∈ R 1×(N×d) , WLinear This represents the preset linear layer weight matrix, used to map the enhanced temporal feature representation to the target dimension. W Linear ∈ R (N×d)×d ; .

[0203] S1026. Based on the time characteristics of each network device, construct a time modeling module that matches multiple network devices.

[0204] In this embodiment, the time dependency relationship of each network device can be captured through the above steps S1021-S1026, and a multivariate output can be obtained by concatenating n single variables together. Y out ∈ R N×d , used to characterize time modeling modules that match multiple network devices.

[0205] S103. Based on the spatial modeling module and the temporal modeling module, a comprehensive spatiotemporal feature representation is generated, thereby constructing a spatiotemporal fusion module that matches multiple network devices; it should be noted that the spatiotemporal feature representation is used to connect the dependencies captured by the spatial modeling module and the temporal modeling module, and the spatiotemporal fusion module can serve as a bridge connecting the spatial modeling module and the temporal modeling module.

[0206] Specifically, in step S103, the spatiotemporal features are represented as follows:

[0207] ;

[0208] In the formula, This represents the spatiotemporal features. This parameter represents "spatial-temporal attention," which is a mechanism used to measure the impact of different time steps on each time series. softmax ( ) represents the activation function. Y out This refers to the time modeling module. H out This represents the spatial correlation characteristics in the spatial modeling module. W tp and W sp This represents the preset weight parameters. W tp R d×d , W sp Rd×d , T Indicates the transpose symbol. d This indicates the preset node feature dimension.

[0209] S104. The spatial modeling module and the temporal modeling module are trained collaboratively by the spatiotemporal fusion module to obtain the teacher model.

[0210] In step S104 of this embodiment, the spatiotemporal features are represented. It is used as input to a hybrid spatiotemporal neural network model that integrates spatial modeling and temporal modeling modules. This model consists of fully connected layers and ultimately produces prediction results. During collaborative training, the loss function is expressed as follows:

[0211] ;

[0212] That is, this embodiment uses predicted values ​​in combination. and the true value g t In timestamp t Root mean square deviation and norm l Using 0 as the loss function, the parameters of the hybrid spatiotemporal neural network model can be further optimized using stochastic gradient descent and backpropagation algorithms based on the expression of this loss function, and its parameters can be updated through the Adam optimizer to obtain the final teacher model.

[0213] S2. Obtain a network anomaly detection dataset; wherein the network anomaly detection dataset includes multiple training samples, and each training sample includes a target network and a label matching the target network.

[0214] S3. Randomly extract training samples from the network anomaly detection dataset, and input the training samples into the teacher model and each student model of the student model set, so as to obtain the prediction values ​​of the teacher model and each student model of the student model set for the training samples.

[0215] S4. Based on the predicted values ​​of the teacher model and each student model in the student model set for the training samples, calculate the loss function value of each student model in the student model set.

[0216] In step S4 of this embodiment, the loss function value of any student model is calculated using the cross-entropy loss function and the KL divergence.

[0217] Specifically, in step S4, the student model set... iThe loss function value for the student model is:

[0218] ;

[0219] In the formula, For the student model set, the first... i The loss function value of the student model. a , b and c 'Indicates the preset training weights, K ' represents the total number of student models in the student model set, where, The preset training coefficients are used to ensure that training is primarily guided by supervised learning, which is based on real labels and soft predictions from the teacher model. Indicates the first i Hard cross-entropy loss of a student model Indicates the first i The soft cross-entropy loss of the student model, Indicates the first i The predicted values ​​of the student model. In the student model set, the first... k The predicted values ​​of the student model. Indicates the first i KL divergence of a student model.

[0220] In this embodiment, introducing KL divergence as part of the loss function helps reduce the bias between student models.

[0221] S5. Calculate the stochastic gradient value of each student model based on the loss function value of each student model, and update each student model in the student model set through backpropagation based on the stochastic gradient value of each student model. For example, the parameters in the student model... , γ The preset learning rate is used to control the update step size. i ∈{0,1,…, N This yields a set of updated student models that includes multiple updated student models.

[0222] S6. Update the student model set to the updated student model set, and randomly extract training samples from the network anomaly detection dataset again. That is, return to step S3 for iterative training until the loss function value is less than the preset threshold or the preset maximum number of iterations is reached. Then output the final updated student model set, which is the updated student model set obtained in the last iteration.

[0223] S7. Use the final updated set of student models as the set of trained student models.

[0224] S8. Perform network anomaly detection processing on the target network based on the set of trained student models. Specifically, in this embodiment, during the anomaly detection process, the predicted values ​​of multiple trained student models in the set of trained student models can be combined, and a comprehensive detection result can be obtained through average fusion, weighted fusion, or voting mechanisms. No restrictions are imposed here.

[0225] For example, such as Figure 3 As shown, when the student model set includes two student models, it is assumed that the training samples contain... Sub-objects And the training samples may belong to M Each category will then be correspondingly associated with... N The label corresponding to each sub-object is denoted as ,in At this point, the training samples are first... Use a tokenizer tool such as Huggingface (a machine learning platform that provides datasets, pre-trained models, and courses) to segment the data and obtain its corresponding token. , that is Figure 3 The tokens in the model are then input into the teacher model. α Student model β 1 and β 2. Accordingly, the predicted values ​​output by these three models can be obtained. Z :

[0226] ;

[0227] in It is a model α , β 1 and β 2. Corresponding predicted output values ​​(combined composition) Figure 3 (model output vector in the model). F t , F 1 and F 2 represents the computation functions of the three models. Then, the training samples... X Category m The probability can be calculated as follows:

[0228] ;

[0229] ;

[0230] In the formula, , T 'Indicates the preset distillation temperature; Teacher model αFor training samples X Classify and determine its category. m The probability of; Student model β i For training samples X Classify and determine its category. m The probability. Therefore, the predicted value of the student model can be obtained. ,in .

[0231] To calculate the predicted values ​​of the two student models and The difference between (soft predicted values) was also calculated. and KL divergence between them.

[0232] At this point, for the first student model β 1. Its loss function contains three loss terms: , and ,in, This represents the hard cross-entropy loss of the first student model. ; This represents the soft cross-entropy loss of the first student model. ; Let KL divergence be the KL divergence. ;

[0233] Correspondingly, the loss function value for the first student model is:

[0234] .

[0235] Figure 3 In the middle, with the student model β 1 and β The corresponding loss functions for 2 are respectively used as Loss 1 and Loss 2 indicates that after multiple iterations of training, the final set of trained student models can be obtained, that is, the set of trained student models. β 1 and β The set of trained student models consists of 2 components.

[0236] This embodiment improves detection speed and solves the problem of a single knowledge source in knowledge distillation, resulting in high detection accuracy. Specifically, in implementation, this embodiment uses knowledge distillation to transfer knowledge from a teacher model with a large number of parameters and layers to a student model with a small number of parameters and layers. The trained student model is then used for network anomaly detection. Compared to traditional anomaly detection schemes, this embodiment avoids the problem of using deep neural networks with a large number of parameters for network anomaly detection, resulting in faster detection speed. Furthermore, this embodiment introduces a multi-student model structure based on response-based knowledge distillation, which helps alleviate the student model's dependence on the teacher model, solves the problem of a single knowledge source for the student model in knowledge distillation, and improves the accuracy of network anomaly detection by the trained student model.

[0237] Example 2:

[0238] This embodiment discloses a network anomaly detection system based on a knowledge distillation framework for implementing the network anomaly detection method based on the knowledge distillation framework in Embodiment 1; such as... Figure 4 As shown, the network anomaly rapid detection system based on the knowledge distillation framework includes:

[0239] The initial model acquisition module is used to acquire the teacher model and a set of student models, including multiple student models.

[0240] The model training module is communicatively connected to the initial model acquisition module and is used to acquire the network anomaly detection dataset; wherein, the network anomaly detection dataset includes multiple training samples;

[0241] The model training module is further configured to randomly extract training samples from the network anomaly detection dataset and input the training samples into the teacher model and each student model of the student model set, so as to obtain the prediction values ​​of the teacher model and each student model of the student model set for the training samples.

[0242] The model training module is further configured to calculate the loss function value of each student model in the student model set based on the prediction values ​​of the teacher model and each student model in the student model set for the training samples.

[0243] The model training module is also used to calculate the stochastic gradient value of each student model based on the loss function value of each student model, and to perform backpropagation update on each student model in the student model set based on the stochastic gradient value of each student model, so as to obtain an updated student model set including multiple updated student models.

[0244] The model training module is also used to update the student model set to the updated student model set, and randomly extract training samples from the network anomaly detection dataset again until the obtained loss function value is less than a preset threshold or the preset maximum number of iterations is reached, and then output the final updated student model set.

[0245] The model training module is also used to use the final updated student model set as the trained student model set.

[0246] An anomaly detection module is communicatively connected to the model training module and is used to perform network anomaly detection processing on the target network based on the trained student model set.

[0247] It should be noted that the working process, working details and technical effects of the network anomaly rapid detection system based on the knowledge distillation framework provided in this embodiment 2 can be found in embodiment 1, and will not be repeated here.

[0248] Example 3:

[0249] Based on Embodiment 1 or 2, this embodiment discloses an electronic device, which may be a smartphone, tablet computer, laptop computer, or desktop computer, etc. The electronic device may be referred to as a user terminal, portable terminal, desktop terminal, etc., and includes:

[0250] Memory, used to store computer program instructions; and,

[0251] A processor is configured to execute the computer program instructions to perform the operation of a network anomaly rapid detection method based on a knowledge distillation framework as described in any of Embodiment 1.

[0252] Example 4:

[0253] Based on any one of Embodiments 1 to 3, this embodiment discloses a computer program product, including a computer program or instructions, which, when executed by a computer, implements a rapid network anomaly detection method based on a knowledge distillation framework as described in any one of Embodiments 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0254] Obviously, those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.

[0255] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fast network anomaly detection method based on a knowledge distillation framework, characterized in that, include: Obtain the teacher model and a collection of student models, including multiple student models; Obtain a network anomaly detection dataset; wherein the network anomaly detection dataset includes multiple training samples; Training samples are randomly selected from the network anomaly detection dataset and input into the teacher model and each student model of the student model set, respectively, so as to obtain the prediction values ​​of the teacher model and each student model of the student model set for the training samples. Based on the predicted values ​​of the teacher model and each student model in the student model set for the training samples, the loss function value of each student model in the student model set is calculated. The stochastic gradient value of each student model is calculated based on the loss function value of each student model, and backpropagation is performed on each student model in the student model set based on the stochastic gradient value of each student model to obtain an updated student model set including multiple updated student models. The student model set is updated to the updated student model set, and training samples are randomly drawn again from the network anomaly detection dataset until the obtained loss function value is less than the preset threshold or the preset maximum number of iterations is reached, and then the final updated student model set is output. The final updated set of student models will be used as the set of trained student models. Based on the trained student model set, network anomaly detection processing is performed on the target network; Obtain the teacher model, including: Network data generated by multiple network devices is acquired, and a dynamic graph construction method is used in conjunction with a depth graph convolutional network model to learn the spatial dependencies between network devices from the network data, thereby constructing a spatial modeling module that matches the multiple network devices; wherein, the network data includes time series data generated by each network device; Based on the time series data generated by each network device, the time dependency relationship within a single time series data is extracted, the time feature representation of each network device is generated, and then a time modeling module matching multiple network devices is constructed. Based on the spatial modeling module and the temporal modeling module, a comprehensive spatiotemporal feature representation is generated, thereby constructing a spatiotemporal fusion module that matches multiple network devices; The spatiotemporal fusion module is used to collaboratively train the spatial modeling module and the temporal modeling module to obtain the teacher model. The method employs a dynamic graph construction approach, combined with a depth graph convolutional network model, to learn the spatial dependencies between network devices from the network data, thereby constructing a spatial modeling module that matches multiple network devices, including: The network data is input into the dynamic graph construction module, so that the dynamic graph construction module updates the graph structure layer by layer based on the deep graph convolutional network model, thereby obtaining the graph structure between network devices and the node features of the graph structure. The graph structure is as follows: A K =DGC(X K-1 )∈R n×n ; In the formula, K represents the total number of layers in the depth map convolutional network model, and K is a natural number greater than 1. K-1 R represents the node features generated by the (K-1)th depth graph convolutional network layer of the depth graph convolutional network model, DGC(·) represents the function used in the dynamic graph construction module to determine whether there is a connection relationship between nodes, and R n×n This represents an n x n matrix, where n is the total number of network devices; The node characteristics of the graph structure are: X K ; where X K This represents the node features generated by the Kth depth graph convolutional network layer of the depth graph convolutional network model; Obtain the node features of each depth graph convolutional network layer of the depth graph convolutional network model and combine them into a node feature set; Obtain the projection vector, and based on the node feature set and the projection vector, obtain the implicit high-order dependencies between network devices; Based on the higher-order dependencies and the node feature set, the spatial correlation characteristics of deep coupling between network devices are obtained; Based on the graph structure, the node features of the graph structure, and the spatial correlation features, a spatial modeling module matching multiple network devices is constructed. The expression for the node features generated by the k-th depth map convolutional network layer of the depth map convolutional network model is: In the formula, k∈{2,3,......,K+1}, This represents a normalized symmetric graph structure. A k-1 D represents the (k-1)th graph structure calculated by the dynamic graph construction module, and A represents the (k-1)th graph structure. k-1 The degree matrix, where I represents the identity matrix; The expression for the node feature set is: X=[X 0 ,X 1 ,...,X k ...,X K ]∈R n×(K+1)×d ; Among them, X 0 d represents the preset initial node features of the depth map convolutional network model, and d represents the preset node feature dimension. The higher-order dependency relationship is as follows: In the formula, The higher-order dependencies are represented by Reshape(·), which is a function used to adjust the matrix dimensions. S represents the importance of each convolutional network layer in the depth map, and S = sigmoid(X·c) ∈ R. n×(K+1)×1 c represents the projection vector, c∈R d ×1 ; The spatial correlation feature is as follows: In the formula, H out The spatial correlation feature is represented by ⊙, which represents the matrix multiplication symbol. The process involves extracting the temporal dependencies within individual time-series data generated by each network device, generating temporal feature representations for each network device, and then constructing a time modeling module that matches multiple network devices. This module includes: The time series data generated by each network device is divided into multiple patches, and the multiple patches of each network device are projected into a high-dimensional space to obtain the high-dimensional time series data of each network device. Based on multiple patches for each network device, multiple new patches for each network device are obtained using the self-attention module, and the new patches for each network device are combined and processed to obtain new network data for each network device. The sum of new network data and high-dimensional time series data of each network device is batch normalized to obtain the initial time feature representation of each network device; A feedforward neural network is used to enhance the initial time feature representation of each network device to obtain the enhanced time feature representation of each network device. The enhanced temporal feature representations of each network device are reprojected onto the initial dimension of the network data to obtain the temporal feature representations of each network device. Based on the time characteristics of each network device, a time modeling module matching multiple network devices is constructed. The high-dimensional time series data of any network device is represented as follows: Z=Z'W Z ; In the formula, Z∈R N×d Z = (z1, z2, ..., z i ,...,z N ), z i This represents the i-th d-th patch among multiple patches for any given network device, z i ∈R d N represents the total number of patches, d represents the preset node feature dimension, Z' represents multiple patches for any network device, and Z'∈R N×P P represents the length of the patch, W Z W represents the preset weight matrix. Z ∈R P×d ; Based on multiple patches for any network device, a new patch for any network device is obtained using a self-attention module, and the new patches for any network device are combined to obtain new network data for that network device, including: Initialize the relative position encoding parameter W pos ={W1,W2,...,W j-i ,...,W N-1 }; where W j-i W represents the index of patch i and patch j of any of the network devices d. j-i ∈R d ; The attention weights and normalized attention weights between patches of any network device are calculated based on the relative position encoding parameters; wherein, the attention weight between patch i and patch j of any network device is: In the formula, μ ij z represents the attention weight between patch i and patch j of any of the network devices. j Z represents the j-th patch among multiple patches for any given network device. j ∈R d ,β Q and β K This represents the preset weight matrix parameters, β. Q ∈R d×d ,β K ∈R d×d a ij a represents the degree of attention given to the embedding of patch i and patch j of any of the network devices. ij =W j-i ; The normalized attention weight between patch i and patch j of any network device is: In the formula, α ij μ is the normalized attention weight between patch i and patch j of any of the network devices. ie This represents the attention weight between patch i and patch e of any of the network devices; Based on multiple patches for any network device and the normalized attention weights between each patch, multiple new patches for any network device are calculated; wherein, the new patch corresponding to the i-th patch among the multiple patches for any network device is: In the formula, z i ' is the new patch corresponding to the i-th patch among multiple patches for any network device, β V This represents the preset weight matrix parameters, β. V ∈R d×d ; Multiple new patches for any one of the network devices are combined to obtain new network data for that network device; wherein, the new network data for any one network device is represented as follows: WITH attn =(z'1,z'2,…,z' i ,…,With' N ); In the formula, z attn ∈R N×d ; The initial time characteristics of any network device are represented as follows: Y=BatchNorm(Z+z attn ); In the formula, Y represents the initial time characteristics of any network device, and BatchNorm(·) is the batch normalization function; The enhanced time characteristics of any network device are represented as follows: In the formula, BatchNorm(·) is the batch normalization function, representing the enhanced temporal characteristics of any network device. This represents the initial time characteristics of any of the network devices; Let W1 and W2 be the enhanced temporal feature representation of any network device, where Relu(·) represents the activation function, W1 and W2 represent the preset weight matrices, and b1 and b2 represent the biases corresponding to the weight matrices W1 and W2, respectively. The time characteristics of any network device are represented as follows: In the formula, Let Y' be the temporal feature representation of any of the network devices, and Y' be the feature representation obtained by flattening the enhanced temporal feature representation of any of the network devices, where Y'∈R 1×(N×d) W Linear W represents the preset linear layer weight matrix. Linear ∈R (N×d)×d ; The spatiotemporal features are represented as follows: G=δ st H out ; In the formula, G represents the spatiotemporal feature. softmax(·) represents the activation function, Y out H represents the time modeling module. out W represents the spatial correlation characteristics in the spatial modeling module. tp and W sp W represents the preset weight parameters. tp ∈R d×d W sp ∈R d×d ,)T represents the transpose symbol, and d represents the preset node feature dimension; The loss function value of any student model is calculated using the cross-entropy loss function and the KL divergence. The loss function value of the i-th student model in the student model set is: In the formula, L i Let K' be the loss function value of the i-th student model in the student model set, where a, b, and c' represent preset training weights, and K' represents the total number of student models in the student model set. This represents the hard cross-entropy loss of the i-th student model. This represents the soft cross-entropy loss of the i-th student model. This represents the predicted value of the model for the i-th student. D represents the predicted value of the k-th student model in the student model set. KL Let KL divergence be the value of the i-th student model.

Images