An anomaly detection method integrating knowledge distillation and group learning

Abstract

This invention relates to an anomaly detection method integrating knowledge distillation and group learning, belonging to the field of Internet of Things (IoT) technology. The method includes the following steps: S1: Constructing an anomaly detection network based on attention and graph convolution in a 5G IoT scenario; S2: Training a local teacher model on the IoT edge server; S3: Aggregating teacher network parameters on the group network and distributing them to the edge server; S4: The edge server trains a local student network; S5: The student network detects abnormal traffic in the IoT in real time; S6: The group network periodically updates the global teacher model and the local student model. This invention enables real-time and effective anomaly detection of IoT device traffic, protecting user privacy and data security.

Description

Technical Field

[0001] This invention belongs to the field of Internet of Things (IoT) technology and relates to an anomaly detection method that integrates knowledge distillation and group learning. Background Technology

[0002] 5G Internet of Things (5G IoT) is an IoT application based on fifth-generation mobile communication technology. Due to the high speed, low latency, massive connectivity, and high-density connection characteristics of 5G networks, it can better meet the needs of IoT applications and is of great significance for achieving ultra-high-speed, highly reliable, and low-latency data transmission. Through the deep integration of IoT technology and 5G, 5G IoT is widely used in various fields, such as smart transportation, smart homes, smart healthcare, and smart factories. However, with the introduction of new technologies and the continuous expansion of the 5G IoT scale, it also faces new security threats. On the one hand, the access of a large number of IoT devices may cause network congestion, leading to abnormal device traffic. On the other hand, IoT nodes sometimes exhibit abnormal behavior, causing network interruptions, which may be caused by network attacks or device failures. Existing 5G IoT typically consists of a large number of sensor devices, each generating traffic data in time-series format. Therefore, intelligent analysis of the time-series traffic of 5G IoT devices to promptly detect abnormal traffic in the network is of great security significance for 5G IoT devices.

[0003] The widespread adoption of 5G networks has significantly increased the scale and complexity of IoT systems. However, the detection of device failures and abnormal traffic has become increasingly important and challenging. Traditional anomaly detection methods often fall short when dealing with large-scale, multi-dimensional, and real-time changing data. To address these issues, advanced machine learning and artificial intelligence technologies have been introduced to improve the accuracy and efficiency of anomaly detection.

[0004] Knowledge distillation is an effective machine learning technique that simplifies computational complexity by extracting knowledge from a complex teacher model and transferring it to smaller student models, while retaining the high performance of the teacher model. Swarm learning, on the other hand, is a distributed machine learning method that improves the robustness and accuracy of the overall model by combining the predictions of multiple models. In IoT anomaly detection, the fusion of knowledge distillation and swarm learning can fully leverage the strengths of each model to build an efficient, accurate, and stable anomaly detection framework to address the ever-changing and complex challenges of the 5G IoT environment. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide an anomaly detection method that integrates knowledge distillation and group learning.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An anomaly detection method that integrates knowledge distillation and group learning, the key of which includes the following steps:

[0008] S1: In the 5G IoT scenario, an anomaly detection network based on attention and graph convolution is constructed; the anomaly detection network based on attention and graph convolution consists of a time series K-means clustering module based on FastDTW, a variable long-term correlation capture module based on GCN, a neighborhood information extraction module based on multi-scale dilated convolution, and a long-term series reconstruction module based on LSH attention.

[0009] S2: Construct a group learning network. The participants in the group learning network consist of various edge servers of the Internet of Things. Deploy anomaly detection teacher and student models based on attention and graph convolution on each edge server and train the local teacher model.

[0010] S3: Each edge server shares local teacher model parameters through the group network. The virtual central server aggregates the network parameters of each teacher to form global teacher model parameters and distributes them to each IoT edge server.

[0011] S4: Each edge server receives the global teacher model parameters and uses the local dataset and the global teacher model to train the local student network;

[0012] S5: The local student network combines the abnormal situations of neighboring nodes to detect abnormal traffic in the Internet of Things in real time;

[0013] S6: Develop smart contracts to regularly update the global teacher model and local student model in the group network, ensuring the optimization and updating of the global teacher model and local student model.

[0014] Furthermore, the FastDTW-based time series K-means clustering module replaces Euclidean distance with FastDTW to measure the similarity between time series, and the K-means layer clusters time series variables.

[0015] Furthermore, the GCN-based long-term correlation capture module has a parallel multi-layer structure, with each layer consisting of a graph convolutional layer followed by another graph convolutional layer. The hidden neurons in the first graph convolutional layer are set to 2048, and the hidden neurons in the last convolutional layer are set to 512. Each convolutional layer undergoes layer normalization and Leaky ReLU activation, and the last convolutional layer is then processed through a residual structure.

[0016] Furthermore, the neighborhood information extraction module based on multi-scale dilated convolution has the following structure: first linear layer → first multi-scale extended convolutional layer → second linear layer → second multi-scale extended convolutional layer → third linear layer → third multi-scale extended convolutional layer; the hidden neurons in each linear layer are set to 512, the sparsity of the multi-scale extended convolutional layer is set to 1, 2, 3, and the receptive field is set to 3, 5, 7.

[0017] Furthermore, the long-term series reconstruction module based on LSH attention includes hash functions with multiple random projections and a self-attention mechanism; the hash function is defined as:

[0018]

[0019] Where q i It is the eigenvector of a multivariate time series at time t, r i b is a random vector drawn from the standard normal distribution. i It is a random offset drawn from a uniform distribution; a hash function is used to divide the row vectors of the query matrix, key matrix and value matrix of the sequence into multiple buckets according to similarity, and self-attention is performed within the bucket to effectively capture long-distance dependencies.

[0020] Furthermore, in S2, an anomaly detection teacher network and student network based on attention and graph convolution are deployed on each IoT edge server, and the teacher model is trained locally; the training loss for training the teacher model is defined as:

[0021]

[0022] Where M represents the sample size, and m represents the m-th sample. Indicates the reconstructed output. Represents the observed value. This indicates that the matrix has n rows and m columns.

[0023] Furthermore, in S3, the group network includes multiple IoT edge servers, each of which is equipped with an anomaly detection teacher network and student network based on attention and graph convolution. One edge server is randomly selected as a virtual central server, and the teacher model parameters trained on the other edge servers are sent to the virtual central server for aggregation. The aggregation process uses a weighted average method.

[0024]

[0025] In the formula φ global φ represents the global teacher model parameters. local Let α represent the local teacher model parameters of the edge server, s represent the s-th edge server, and α represent the parameters of the local teacher model of the edge server. s ∈[0,1] represents the weight of the aggregation.

[0026] Furthermore, in S4, each edge server receives the global teacher model from the virtual central server, uses the local dataset and the global teacher model to guide the training of the local student model, and defines the distillation loss of the local student model as:

[0027]

[0028] Where M represents the sample size, k represents the k-th sample, n represents the total number of rows in the matrix, i represents the i-th row of the matrix, m represents the total number of columns in the matrix, and j represents the j-th row of the matrix. This represents the output of the teacher model. This represents the output of the student model; the overall loss function of the student model is defined as the weighted sum of the distillation loss and the training loss:

[0029] L=α·L KD +β·L mse

[0030] Where α and β are weighting coefficients.

[0031] Furthermore, in S5, the student network is responsible for real-time detection of abnormal traffic from IoT devices. The degree of abnormality is quantified by an anomaly score. The anomaly score on the local edge server is defined as follows:

[0032]

[0033] Where n represents the total number of rows in the matrix, i represents the i-th row of the matrix, m represents the total number of columns in the matrix, and j represents the j-th row of the matrix. Indicates the observation window, This represents the reconstruction sequence of the student model. This matrix has n rows and m columns. To perform anomaly detection by combining the anomaly scores of neighboring devices, a weighted average method is used to comprehensively consider both the anomaly score of the device itself and the anomaly scores of its neighboring devices.

[0034]

[0035] Among them, AS′ i AS is the comprehensive anomaly score for device i. i Give a local anomaly score to device i. Let α be the average of the anomaly scores of device i's neighboring devices, and α∈[0,1] be the weighting coefficient.

[0036] Furthermore, in S6, due to the dynamic nature of the IoT environment, the detection accuracy of the global model may decrease over time, requiring continuous optimization of the anomaly detection framework on a regular basis. By formulating smart contracts, the edge server continuously collects new traffic data, performs preprocessing, expands the local dataset, and then uses the latest data to execute S2 to S6 to continuously train and update the local teacher model and student model. This ensures the optimization and updating of the global teacher model and the local student model, thereby achieving efficient anomaly detection and rapid response.

[0037] The beneficial effects of this invention are as follows: Addressing the problem of abnormal device traffic in IoT environments, this invention proposes an anomaly detection model, IoT-FKGDL, based on attention and graph convolution. This model introduces FastDTW as a distance metric between time-series variables and clusters highly correlated variables into the same cluster based on time-series K-Means. Within each cluster, a graph convolutional network is used to capture long-term correlations between variables. Secondly, multi-scale dilated convolution is used to extract local correlations of sequences, and combined with Locality Sensitive Hashing Attention, long-term correlations of multivariate sequences can be effectively captured. Furthermore, an IoT-FKGDL-SL anomaly detection framework based on a group learning architecture incorporating knowledge distillation is designed, establishing a global anomaly detection framework for IoT environments. This anomaly detection framework implements an efficient, secure, scalable, and privacy-preserving distributed learning method. It is particularly suitable for application scenarios requiring real-time response and distributed computing, such as IoT device traffic anomaly detection.

[0038] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0040] Figure 1 This is a schematic diagram of the process of the present invention;

[0041] Figure 2 For IoT-FKGDL anomaly detection model;

[0042] Figure 3 An IoT-FKGDL anomaly detection framework based on a group learning architecture that integrates knowledge distillation. Detailed Implementation

[0043] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0044] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0045] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0046] This invention provides an anomaly detection method that integrates knowledge distillation and group learning, comprising:

[0047] S1: In 5G IoT scenarios, construct an anomaly detection network based on attention and graph convolution, specifically including:

[0048] 1) The K-means clustering module for time series based on FastDTW uses FastDTW as a distance metric to cluster time series variables with high similarity into the same cluster. The similarity between each time series and the cluster center is:

[0049]

[0050] FastDTW(X i ,C j ) represents the time series X i and cluster center C j The FastDTW distance between them.

[0051] Calculate the average value of the elements within each cluster as the new cluster center:

[0052]

[0053] Where S j Indicates belonging to cluster C j The collection of all time series.

[0054] 2) The GCN-based long-term correlation capture module is used to capture the correlations between target variables within each cluster. The number of parallel GCN networks corresponds to the number of clusters. Each GCN network layer has a graph convolutional layer → graph convolutional layer structure; the first graph convolutional layer has 2048 hidden neurons, and the last convolutional layer has 512 hidden neurons; each convolutional layer undergoes layer normalization and LeakyReLU activation, and the last convolutional layer is then processed through a residual structure.

[0055] The propagation rule for each convolutional layer is as follows:

[0056]

[0057] in Indicates cluster C j The adjacency matrix of the time series variables within I, and I N It is the identity matrix, and λ is a learnable parameter. (Degree matrix) is a trainable weight matrix, and m represents the dimension of the hidden features of this layer. ReLU(·) is a non-linear activation function.

[0058] 3) A neighborhood information extraction module based on multi-scale dilated convolution is used to capture and analyze the long-term dependencies and dynamic changes of multivariate long-term series, thereby gaining a deeper understanding of data characteristics. Because data generated by IoT devices is continuous, anomalies typically occur in local areas. Anomalous data points are highly correlated with neighboring areas, but weakly correlated with more distant data points. This module expresses this adjacency relationship and uses it for anomaly detection. Its structure is as follows: first linear layer → first multi-scale dilated convolutional layer → second linear layer → second multi-scale dilated convolutional layer → third linear layer → third multi-scale dilated convolutional layer; the hidden neurons in each linear layer are set to 512, the sparsity of the multi-scale dilated convolutional layers is set to 1, 2, 3, and the receptive field is set to 3, 5, 7.

[0059] 4) The LSH-based self-attention long-time series reconstruction module addresses the high computational complexity and memory consumption inherent in self-attention mechanisms for long-time series modeling. As sequence length increases, computational and memory consumption quadratically, with memory limitations becoming a major bottleneck in processing long sequences. In practical applications, high computational and memory requirements lead to high latency, making it difficult to meet the real-time processing needs of IoT anomaly detection. This module includes hash functions with multiple random projections and a self-attention mechanism. The hash function is defined as:

[0060]

[0061] Where q i It is the eigenvector of a multivariate time series at time t, r i b is a random vector drawn from the standard normal distribution. i It is a random offset drawn from a uniform distribution; a hash function is used to divide the row vectors of the query matrix, key matrix and value matrix of the sequence into multiple buckets according to similarity, and self-attention is performed within the bucket to effectively capture long-distance dependencies.

[0062] Finally, the time series K-means clustering module based on FastDTW, the variable long-term correlation capture module based on GCN, the neighborhood information extraction module based on multi-scale dilated convolution, and the long-term series reconstruction module based on LSH attention are connected in sequence to form the anomaly detection network based on attention and graph convolution in this invention.

[0063] S2: Construct a group learning network and train a local teacher model, specifically including:

[0064] 1) The participants in the group learning network consist of various edge servers of the Internet of Things, on which attention-based and graph convolution-based anomaly detection teacher and student models are deployed.

[0065] 2) Each IoT edge server, as a participant in group learning, continuously monitors and collects traffic data, and the edge server preprocesses the collected data.

[0066] 3) Then, run the anomaly detection algorithm locally to train the teacher model. The training loss for the teacher model is defined as:

[0067]

[0068] Where M represents the sample size, and m represents the m-th sample. Indicates the reconstructed output. Represents the observed value. This indicates that the matrix has n rows and m columns.

[0069] S3: Each edge server shares local teacher model parameters through the cluster network. The virtual central server aggregates the network parameters of each teacher to form global teacher model parameters and distributes them to each IoT edge server, including:

[0070] 1) The parameters of the local teacher model are updated and shared with other participants in the group learning network. A randomly selected edge server is used as a virtual central server to aggregate the teacher model parameters uploaded from various edge servers, forming a global teacher model. A weighted average method is used to calculate the global teacher model parameters based on the amount of data per node or model performance, ensuring that the merged teacher model is more representative and robust. The aggregation process is represented as follows:

[0071]

[0072] In the formula φ global φ represents the global teacher model parameters. local Let α represent the local teacher model parameters of the edge server, s represent the s-th edge server, and α represent the parameters of the local teacher model of the edge server. s ∈[0,1] represents the weight of the aggregation.

[0073] 2) The virtual central server distributes the merged global teacher model to each edge node through the group network.

[0074] S4: Each edge server receives the global teacher model parameters and uses the local dataset and the global teacher model to train the local student network, specifically including:

[0075] 1) Each edge server receives the global teacher model parameters issued by the virtual center server through the group network.

[0076] 2) Each edge node uses its local dataset and the global teacher model to guide the training of a lightweight student model, defining the distillation loss for the local student model:

[0077]

[0078] Where M represents the sample size, k represents the k-th sample, n represents the total number of rows in the matrix, i represents the i-th row of the matrix, m represents the total number of columns in the matrix, and j represents the j-th row of the matrix. This represents the output of the teacher model. This represents the output of the student model. The overall loss function of the student model is defined as a weighted sum of the distillation loss and the training loss:

[0079] L=α·L KD +β·L mse

[0080] Where α and β are weighting coefficients.

[0081] S5: The local student network combines the anomalies of neighboring nodes to detect abnormal traffic in the Internet of Things in real time, specifically including:

[0082] 1) The anomaly score of the local edge server is defined as follows:

[0083]

[0084] Where n represents the total number of rows in the matrix, i represents the i-th row of the matrix, m represents the total number of columns in the matrix, and j represents the j-th row of the matrix. Indicates the observation window, This represents the reconstruction sequence of the student model. This indicates that the matrix has n rows and m columns.

[0085] 2) Anomaly detection is performed by combining the anomaly scores of neighboring devices, using a weighted average method to comprehensively consider both the anomaly score of the device itself and the anomaly scores of its neighboring devices:

[0086]

[0087] Among them, AS′ i AS is the comprehensive anomaly score for device i. i Give a local anomaly score to device i. Let α be the average of the anomaly scores of device i's neighboring devices, and α∈[0,1] be the weighting coefficient.

[0088] S6: Establish smart contracts to periodically update the global teacher model and local student model within the group network, ensuring their optimization and updates. This includes:

[0089] 1) Due to the dynamic nature of the IoT environment, the detection accuracy of the global model may decrease over time. Therefore, continuous optimization of the anomaly detection framework is necessary. Regularly executing S2-S5 involves using smart contracts to enable edge servers to continuously collect new traffic data, preprocess it, expand the local dataset, and then continuously train and update the local teacher and student models using the latest data, improving the model's adaptability and accuracy. The virtual central server is responsible for regularly collecting and aggregating the latest teacher model parameters, forming a new global teacher model, and distributing it. Ensuring the optimization and updating of the global teacher model enables efficient anomaly detection and rapid response.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An anomaly detection method integrating knowledge distillation and group learning, characterized in that: The method includes the following steps: S1: In the 5G IoT scenario, an anomaly detection network based on attention and graph convolution is constructed; the anomaly detection network based on attention and graph convolution consists of a time series K-means clustering module based on Fast Dynamic Time Warping (FastDTW), a variable long-term correlation capture module based on Graph Convolutional Network (GCN), a neighborhood information extraction module based on multi-scale dilated convolution, and a long-term series reconstruction module based on Locality Sensitive Hash (LSH) attention connected in sequence. S2: Construct a swarm learning network. Participants in the swarm learning network consist of various edge servers of the Internet of Things (IoT). On each edge server, deploy an anomaly detection teacher model and student model based on attention and graph convolution, and train the local teacher model. The training loss for training the local teacher model is defined as: Where M represents the sample size, and m represents the m-th sample. Indicates the reconstructed output. Represents the observed value. This indicates that the matrix has n rows and m columns. S3: Each edge server shares a local teacher model through a group learning network. The virtual central server aggregates the local teacher models to form a global teacher model and distributes it to each IoT edge server. S4: Each edge server receives the global teacher model from the virtual central server, trains its local student model using the local dataset and the global teacher model, and defines the distillation loss of the local student model as: Where M represents the sample size, k represents the k-th sample, n represents the total number of rows in the matrix, i represents the i-th row of the matrix, m represents the total number of columns in the matrix, and j represents the j-th row of the matrix. This represents the output of the teacher model. This represents the output of the student model; the overall loss function of the student model is defined as the weighted sum of the distillation loss and the training loss: L=α·L KD +β·L mse Where α and β are weighting coefficients; S5: The local student model combines the abnormal situations of neighboring nodes to detect abnormal traffic in the Internet of Things in real time. S6: Develop smart contracts so that the group learning network can regularly update the global teacher model and local student model, ensuring the optimization and updating of the global teacher model and local student model.

2. The anomaly detection method integrating knowledge distillation and group learning according to claim 1, characterized in that: The FastDTW-based time series K-means clustering module uses FastDTW to measure the similarity between time series and K-means layer clustering of time series variables.

3. The anomaly detection method integrating knowledge distillation and group learning according to claim 1, characterized in that: The GCN-based long-term correlation capture module has a parallel multi-layer structure, with each layer consisting of a graph convolutional layer followed by another graph convolutional layer. The first graph convolutional layer has 2048 hidden neurons, and the last graph convolutional layer has 512 hidden neurons. Each graph convolutional layer undergoes layer normalization and Leaky ReLU activation with leakage, and the last graph convolutional layer is then processed through a residual structure.

4. The anomaly detection method integrating knowledge distillation and group learning according to claim 1, characterized in that: The neighborhood information extraction module based on multi-scale dilated convolution has the following structure: first linear layer → first multi-scale extended convolutional layer → second linear layer → second multi-scale extended convolutional layer → third linear layer → third multi-scale extended convolutional layer; the hidden neurons in each linear layer are set to 512, the sparsity of the multi-scale extended convolutional layer is set to 1, 2, 3, and the receptive field is set to 3, 5, 7.

5. The anomaly detection method integrating knowledge distillation and group learning according to claim 1, characterized in that: The long-term series reconstruction module based on LSH attention includes hash functions with multiple random projections and a self-attention mechanism; the hash function is defined as: Where q i It is the eigenvector of a multivariate time series at time t, r i b is a random vector drawn from the standard normal distribution. i It is a random offset drawn from a uniform distribution; a hash function is used to divide the row vectors of the query matrix, key matrix, and value matrix of the sequence into multiple buckets according to their similarity.

6. The anomaly detection method integrating knowledge distillation and group learning according to claim 1, characterized in that: In S3, the swarm learning network includes multiple IoT edge servers. Each edge server is equipped with an anomaly detection teacher model and a student model based on attention and graph convolution. One edge server is randomly selected as a virtual central server, and the local teacher models trained on the other edge servers are sent to the virtual central server for aggregation. The aggregation process uses a weighted average method. In the formula φ global Represents the global teacher model, φ local Let α represent the local teacher model of the edge server, s represent the s-th edge server, and α represent the local teacher model of the edge server. s ∈[0,1] represents the weight of the aggregation.

7. The anomaly detection method integrating knowledge distillation and group learning according to claim 1, characterized in that: In S5, the student model is responsible for real-time detection of abnormal traffic from IoT devices. The degree of abnormality is quantified by an anomaly score, and the anomaly score on the local edge server is defined as follows: Where n represents the total number of rows in the matrix, i represents the i-th row of the matrix, m represents the total number of columns in the matrix, and j represents the j-th row of the matrix. Indicates the observation window, This represents the reconstruction sequence of the student model. This matrix has n rows and m columns; a weighted average method is used to comprehensively consider the anomaly score of the device itself and the anomaly scores of its neighboring devices. Among them, AS′ i AS is the comprehensive anomaly score for device i. i Give a local anomaly score to device i. Let α be the average of the anomaly scores of device i's neighboring devices, and α∈[0,1] be the weighting coefficient.

Images