Internet of vehicles anomaly detection lightweight method based on deep learning and knowledge distillation

Abstract

The invention discloses an Internet of Vehicles anomaly detection lightweight method based on deep learning and knowledge distillation, and the method comprises the steps: carrying out the preprocessing of a time sequence message of each vehicle in a data set through employing a sliding window strategy by employing a public Internet of Vehicles data set VeReMi Extension, obtaining a time window, and dividing the time window into a training set and a test set; constructing a teacher network, and performing reconstruction-based unsupervised learning training by using a large number of normal time windows in a training set; constructing a lightweight student network, and guiding the student network to realize knowledge distillation on the normal time window in the training set by using the trained teacher network; and deploying the trained lightweight student network and the tested preset threshold value to a roadside unit of the Internet of Vehicles, thereby realizing rapid Internet of Vehicles anomaly detection. According to the invention, the knowledge learned by the teacher network with high precision and large capacity is migrated to the lightweight student network, and rapid vehicle networking anomaly detection is realized while the detection precision is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle networking anomaly detection, and in particular to a lightweight vehicle networking anomaly detection method based on deep learning and knowledge distillation. Background Art

[0002] As an emerging wireless network technology, the Internet of Vehicles (IoV) is attracting more and more attention. IoV connects vehicles, roads and traffic infrastructure, providing vehicles with instant communication, information services and traffic management functions. By monitoring vehicle location, speed and traffic conditions in real time, IoV can improve the safety, efficiency and convenience of road traffic, thus providing strong support for the construction of intelligent transportation systems.

[0003] However, with the rapid development of the Internet of Vehicles, network security issues are becoming increasingly apparent. Since vehicle communications involve a large amount of privacy data and basic security information, such as location, driving route, etc., the Internet of Vehicles faces threats from various malicious attacks. For example, hackers may cause traffic accidents by tampering with the information sent by the vehicle, forging vehicle identity information to create false congestion, or consuming network resources by sending messages at a high frequency, thereby affecting the entire network. Therefore, anomaly detection systems are used to detect these anomalies and take some countermeasures against abnormal messages, such as not forwarding messages that are considered abnormal.

[0004] Deep learning has great potential in the field of anomaly detection. Most of the previous anomaly detection schemes for the Internet of Vehicles only focused on detecting one or several types of anomalies, while deep neural networks can detect multiple types of anomalies through training and have stronger robustness. However, in reality, anomalies are usually rare and masked by a large number of normal messages, which makes data labeling difficult and expensive. Therefore, we need to study anomaly detection in the Internet of Vehicles under unsupervised settings. The heterogeneity of anomalies makes the problem difficult, which means that we must not only identify known anomalies, but also do our best to detect anomalies that have never been seen.

[0005] Deep learning anomaly detection also faces a serious problem: most of the devices in the Internet of Vehicles are resource-constrained devices. We must reduce the resources required for anomaly detection methods as much as possible while ensuring detection performance to make deployment possible. Knowledge distillation, as an effective model lightweighting method in deep learning, can help us lightweight deep learning models and facilitate deployment on resource-constrained devices.

[0006] Based on the above discussion, a lightweight method for anomaly detection in Internet of Vehicles based on deep learning and knowledge distillation is invented, which has high practical application value. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and proposes a lightweight method for vehicle network anomaly detection based on deep learning and knowledge distillation. A teacher network with high detection accuracy is trained by unsupervised learning based on reconstruction, and then a lightweight student network is trained by knowledge distillation while maintaining high detection accuracy, so as to achieve rapid detection of time windows extracted from vehicle time series.

[0008] To achieve the above purpose, the technical solution provided by the present invention is: a lightweight method for anomaly detection in Internet of Vehicles based on deep learning and knowledge distillation, comprising the following steps:

[0009] 1) Using the public Internet of Vehicles dataset VeReMi Extension, the sliding window strategy is used to preprocess the time series messages of each vehicle in the dataset to obtain the time window. The preprocessed dataset is divided into a training set and a test set. The training set only contains normal time windows, while the test set contains normal time windows and abnormal time windows.

[0010] 2) Construct a teacher network based on Transformer and LSTM, wherein the Transformer is used as an encoder to capture the dependencies between various positions in the input time window to obtain a compressed representation of the time window information, and the LSTM is used as a decoder to effectively maintain long-distance dependencies through memory units and gating mechanisms, and to restore the information compressed by the Transformer encoding to input data through decoding; the teacher network uses a fully connected neural network as a linear embedding layer to replace the embedding layer in the original Transformer, mapping discrete input message features to a continuous vector space, while expanding the feature dimension of the input so as to better capture the hidden relationship between features later; the teacher network only uses the Transformer encoder to extract important information in the time window, while The decoder is not required to generate sequence output, so the decoder in the original Transformer is removed; the teacher network introduces a feedforward neural network layer after the Transformer and a fully connected network layer after the LSTM. The purpose of introducing the feedforward neural network layer after the Transformer is to integrate the features learned by the encoder and perform nonlinear transformation through layer-by-layer transmission to map high-dimensional data to low-dimensional representation, while the purpose of introducing the fully connected network layer after the LSTM is to map the decoded output to the dimension of the expected output; a large number of normal time windows in the training set are used to train the teacher network using an unsupervised learning method based on reconstruction, so that the teacher network learns the distribution of normal time windows and can restore normal time windows, but cannot restore abnormal time windows that have not been seen during training, thereby achieving the purpose of distinguishing abnormal time windows;

[0011] 3) Construct a lightweight student network with a similar structure to the teacher network, but with fewer layers, units, and parameters. Use the trained teacher network to guide the lightweight student network to implement knowledge distillation on the normal time window in the training set, increase the ability of the lightweight student network to reconstruct the normal time window, and obtain a trained lightweight student network.

[0012] 4) The trained lightweight student network is tested using a test set, and the threshold with the highest detection accuracy on the test set is selected as the preset threshold. The preset threshold and the trained lightweight student network are then deployed to the roadside unit in the Internet of Vehicles to perform anomaly detection tasks. The time window to be tested is reconstructed by the lightweight student network, and the reconstruction error of the time window is used as its anomaly score. Finally, the anomaly score is compared with the preset threshold. If the anomaly score exceeds the preset threshold, it is judged as an anomaly, otherwise it is judged as normal.

[0013] Further, the step 1) comprises the following steps:

[0014] 1.1) Obtain the public Internet of Vehicles dataset VeReMi Extension and perform data annotation on it. Messages in the dataset that have not been changed by noise or attacks are marked as normal messages, otherwise they are marked as abnormal messages;

[0015] 1.2) In the dataset, a basic safety message of the Internet of Vehicles carries the following contents: sending time, pseudonym ID, location x, location y, speed x, speed y, acceleration x, acceleration y, direction x and direction y. The above contents are preprocessed as follows: for each message, only the sending time, pseudonym ID, location x, location y, speed x and speed y are selected as input features. For all basic safety messages, the z-score normalization strategy is adopted, and the sliding window strategy is used. A sliding window with a window size of 50 and a step size of 10 is used to intercept the time window. If the time window contains an abnormal message, it is marked as an abnormal time window, otherwise it is a normal time window.

[0016] 1.3) The preprocessed data set is divided into a training set and a test set, wherein 90% of the normal time windows are used to construct the training set, and the remaining normal time windows and abnormal time windows are used to construct the test set according to the proportion of the original data set.

[0017] Further, in step 2), a teacher network is constructed and trained, including the following steps:

[0018] 2.1) Build a teacher network with large model capacity as a time window reconstructor. Its modules from low to high layers are: 1 layer of 16-head Transformer encoder, feedforward neural network layer, 4 layers of LSTM decoder and fully connected network layer; among them, the input normal time window W inputIt includes six features: sending time, pseudonym ID, location x, location y, speed x and speed y, and the time window size is 50, so W input The dimension is 50*6. After passing through the linear embedding layer in the Transformer encoder, it is mapped to a high-dimensional representation of 50*128. After passing through the entire Transformer encoder, the dimension is still 50*128; the feedforward neural network layer includes two fully connected layers, and the LeakyReLU activation function is used between the two fully connected layers. The first fully connected layer expands the dimension to 50*512, and the second fully connected layer compresses the dimension back to 50*128; each layer of the LSTM decoder contains 256 units. After passing through the LSTM decoder, the dimension is expanded to 50*256; finally, after passing through the fully connected network layer, the dimension is mapped to 50*2, and the final output reconstructs the time window W output It only contains two features: position x and position y;

[0019] 2.2) Since the input normal time window W input and output reconstruction time window W output The dimensions of are different, so only the input normal time window W is used input The position x and position y of the six features are used to reconstruct the output time window W output Calculate the reconstruction loss, where the input normal time window W input In the example, only the normal time window of position x and position y among the six features is taken as The dimension is 50*2; the output reconstruction time window is directly used as the reconstruction window of the input normal time window, denoted as The dimension is 50*2; construct the objective function L of the teacher network based on the reconstruction of unsupervised learning training T :

[0020]

[0021] In the formula, MAE stands for mean absolute error, The smaller it is, the closer the reconstruction time window is to the input normal time window. When training the teacher network, by minimizing the above objective function L T To optimize the parameters of the teacher network and enhance the ability of the teacher network to reconstruct normal time windows.

[0022] Further, the step 3) comprises the following steps:

[0023] 3.1) Build a lightweight student network with few parameters as a lightweight time window reconstructor. Its modules from low to high layers are: 1 layer of 2-head Transformer encoder, feedforward neural network layer, 1 layer of LSTM decoder and fully connected network layer; among them, the input normal time window Winput It includes six features: sending time, pseudonym ID, location x, location y, speed x and speed y, and the time window size is 50, so W input The dimension is 50*6. After passing through the linear embedding layer in the Transformer encoder, it is mapped to a high-dimensional representation of 50*64. After passing through the entire Transformer encoder, the dimension is still 50*64; the feedforward neural network layer includes two fully connected layers, and the LeakyReLU activation function is used between the two fully connected layers. The first fully connected layer expands the dimension to 50*256, and the second fully connected layer compresses the dimension back to 50*64; the LSTM decoder contains 256 units. After passing through the LSTM decoder, the dimension is expanded to 50*256; finally, after passing through the fully connected network layer, the dimension is mapped to 50*2, and the final output reconstructs the time window W output It only contains two features: position x and position y;

[0024] 3.2) Since the input normal time window W input and output reconstruction time window W output The dimensions of are different, so only the positions x and y of the six features in the input normal time window are used to calculate the reconstruction loss with the output reconstruction time window, where the input normal time window W input In the example, only the normal time window of position x and position y among the six features is taken as The dimension is 50*2; the output reconstruction time window is directly used as the reconstruction window of the input normal time window, denoted as The dimension is 50*2; the objective function of unsupervised learning training based on reconstruction of the student network is constructed

[0025]

[0026] In the formula, MAE stands for mean absolute error, The smaller it is, the closer the reconstruction time window is to the input normal time window. When training the lightweight student network, by minimizing the above objective function To optimize the parameters of the lightweight student network and enhance the ability of the lightweight student network to reconstruct the normal time window;

[0027] The lightweight student network not only needs to learn the ability to reconstruct normal time windows, but also needs to accept knowledge transferred from the teacher network. First, a knowledge distillation method based on feature transfer is defined, called FKD, and three features are defined: the output of the input data after passing through the Transformer encoder and the feedforward neural network layer is defined as the encoding feature EF, the output of the data after passing through the LSTM decoder is defined as the decoding feature DF, and the output of the data after passing through the fully connected network layer is defined as the output feature OF. The objective function of the lightweight student network knowledge distillation method FKD is constructed.

[0028]

[0029] In the formula, MSE is the mean square error, EF is T represents the encoding features of the teacher network, EF S Represents the encoding features of the lightweight student network, DF T Denotes the decoding features of the teacher network, DF S represents the decoding features of the lightweight student network, OF T represents the output features of the teacher network, OF S Represents the output features of the lightweight student network; since the mean square error of three features is used, it is multiplied by the coefficient in front

[0030] Finally, the objective function of the reconstruction-based unsupervised learning training in the lightweight student network is The objective function of the knowledge distillation method FKD Combine proportionally to construct the total objective function L for lightweight student network training S :

[0031]

[0032] In the formula, α and β are hyperparameters used to adjust the objective function With the objective function The weight between them, and α+β1; by minimizing the above total objective function L S , which enables the lightweight student network to accept feature knowledge output from the teacher network while learning to reconstruct normal time windows.

[0033] Furthermore, in step 4), the roadside unit processes the time series sent by each vehicle within the coverage area, converts it into a time window that can be input into the lightweight student network, reconstructs the time window through the lightweight student network, and uses the reconstruction error of the time window as its anomaly score

[0034]

[0035] In the formula, MAE represents the mean absolute error, Indicates the time window to be tested; express After processing, only the time window after the two features of position x and position y is retained; It means that after the input data is reconstructed by the lightweight student network, the output only contains the reconstructed time window of the two features of position x and position y; the normal time window conforms to the distribution of normal data used in training the lightweight student network, so it can be well reconstructed and has a low anomaly score; while the abnormal time window does not conform to the distribution of normal data used in training the lightweight student network, so it will deviate from the original data after reconstruction, resulting in a large anomaly score;

[0036] Finally, by comparing the anomaly score with the preset threshold θ, if the anomaly score of the time window exceeds the preset threshold θ, it is judged as abnormal, otherwise it is judged as normal. Finally, the roadside unit will not forward the time window judged as abnormal.

[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0038] 1. The present invention uses a reconstruction-based unsupervised learning method to train the teacher network and the lightweight student network, solving the problem of difficulty in labeling abnormal data and high labeling cost in supervised learning methods. At the same time, after training with the reconstruction-based unsupervised learning method, it can detect unknown anomalies and has the ability to generalize new and unseen anomalies, solving the problem that the traditional vehicle network anomaly detection method cannot detect unknown anomalies.

[0039] 2. The present invention proposes a feature-based knowledge distillation method FKD, which enhances the ability of the lightweight student network to reconstruct normal time windows by migrating the encoder features, decoder features and output features of the input data when passing through the teacher network, thereby solving the problem that traditional knowledge distillation methods are not suitable for reconstruction-based unsupervised learning.

[0040] 3. The present invention combines the reconstruction-based unsupervised learning method with the feature-based knowledge distillation method FKD, and deploys the optimized lightweight student network to the roadside unit of the Internet of Vehicles to perform anomaly detection tasks. The lightweight student network has the ability to detect unknown anomalies. At the same time, due to the reduction in the number of neural network layers or units, the number of parameters is reduced and the prediction speed is improved, which solves the technical problem that the number of parameters of deep neural networks is large and difficult to deploy on resource-constrained devices in the Internet of Vehicles, and achieves the technical effect of ensuring detection accuracy while improving detection speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1Flow chart of the method of the present invention.

[0042] Figure 2 Schematic diagram of teacher network and lightweight student network training.

[0043] Figure 3 This is the flow chart of time window anomaly detection.

[0044] Figure 4 Schematic diagram of calculating anomaly scores for lightweight student networks. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0046] The experimental platform of this embodiment is Python 3.9, Pytorch 1.13.1, and the computer configuration is: CPU model is Intel (R) Xeon (R) Gold 6230, memory is 12GB, and graphics card model is NVIDIA GeForce RTX 2080Ti.

[0047] like Figures 1 to 4 As shown, this embodiment discloses a lightweight method for anomaly detection in the Internet of Vehicles based on deep learning and knowledge distillation, which is roughly divided into four stages: the first stage is the acquisition and preprocessing of the data set, which mainly downloads the public data set, uses the sliding window method to generate the vehicle time window, selects the required features, and performs standardization operations; the second stage is the construction and training of the teacher network, which mainly builds a teacher network with a large model capacity and performs reconstruction-based unsupervised learning training; the third stage is the construction and training of the lightweight student network, which mainly builds a lightweight student network and uses knowledge distillation to transfer the knowledge of the teacher network to the lightweight student network; the fourth stage is the deployment of the model, which mainly deploys the trained lightweight student network to the roadside unit to perform anomaly detection. The specific steps are as follows:

[0048] 1) Using the existing Internet of Vehicles dataset VeReMi Extension, the time series of different vehicles are preprocessed to generate time windows, and the dataset is divided into training and test sets, which includes the following steps:

[0049] 1.1) Obtain the public Internet of Vehicles dataset VeReMi Extension and perform data annotation on it. Messages in the dataset that have not been changed by noise or attacks are marked as normal messages, otherwise they are marked as abnormal messages;

[0050] 1.2) In the dataset, a basic safety message of the Internet of Vehicles carries the following contents: sending time, pseudonym ID, location x, location y, speed x, speed y, acceleration x, acceleration y, direction x and direction y. The above contents are preprocessed as follows: for each message, only the sending time, pseudonym ID, location x, location y, speed x and speed y are selected as input features. For all basic safety messages, the z-score normalization strategy is adopted, and the sliding window strategy is used. A sliding window with a window size of 50 and a step size of 10 is used to intercept the time window. If the time window contains an abnormal message, it is marked as an abnormal time window, otherwise it is a normal time window.

[0051] 1.3) The preprocessed data set is divided into a training set and a test set, wherein 90% of the normal time windows are used to construct the training set, and the remaining normal time windows and abnormal time windows are used to construct the test set according to the proportion of the original data set.

[0052] 2) Construct a teacher network based on Transformer and LSTM, wherein the Transformer is used as an encoder to capture the dependencies between various positions in the input time window and obtain a compressed representation of the time window information. The LSTM is used as a decoder to effectively maintain long-distance dependencies through memory units and gating mechanisms, and to restore the information compressed by the Transformer encoding to input data through decoding. The teacher network uses a fully connected neural network as a linear embedding layer to replace the embedding layer in the original Transformer, mapping discrete input message features to a continuous vector space, while expanding the feature dimension of the input so as to better capture the hidden relationship between features later. The teacher network only uses the Transformer encoder to extract important information in the time window without the need for a decoder. The encoder generates sequence output, so the decoder in the original Transformer is removed; the teacher network introduces a feedforward neural network layer after the Transformer and a fully connected network layer after the LSTM. The role of the feedforward neural network layer after the Transformer is to integrate the features learned by the encoder and perform nonlinear transformation through layer-by-layer transmission to map high-dimensional data to low-dimensional representation, while the role of the fully connected network layer after the LSTM is to map the decoded output to the dimension of the expected output; a large number of normal time windows in the training set are used to train the teacher network using an unsupervised learning method based on reconstruction, so that the teacher network learns the distribution of normal time windows and can restore normal time windows, but cannot restore abnormal time windows that have not been seen during training, thereby achieving the purpose of distinguishing abnormal time windows; it includes the following steps:

[0053] 2.1) Build a teacher network with large model capacity as a time window reconstructor. Its modules from low to high layers are: 1 layer of 16-head Transformer encoder, feedforward neural network layer, 4 layers of LSTM decoder and fully connected network layer; among them, the input normal time window W input It includes six features: sending time, pseudonym ID, location x, location y, speed x and speed y, and the time window size is 50, so W input The dimension is 50*6. After passing through the linear embedding layer in the Transformer encoder, it is mapped to a high-dimensional representation of 50*128. After passing through the entire Transformer encoder, the dimension is still 50*128; the feedforward neural network layer includes two fully connected layers, and the LeakyReLU activation function is used between the two fully connected layers. The first fully connected layer expands the dimension to 50*512, and the second fully connected layer compresses the dimension back to 50*128; each layer of the LSTM decoder contains 256 units. After passing through the LSTM decoder, the dimension is expanded to 50*256; finally, after passing through the fully connected network layer, the dimension is mapped to 50*2, and the final output is the normal time window W output It only contains two features: position x and position y;

[0054] 2.2) Since the input normal time window W input And output normal time window W output The dimensions of are different, so only the input normal time window W is used input The position x and position y of the six features are compared with the output normal time window W output Calculate the reconstruction loss, where the input normal time window W input In the example, only the normal time window of position x and position y among the six features is taken as The dimension is 50*2; the output reconstruction time window is directly used as the reconstruction window of the input normal time window, denoted as The dimension is 50*2; the objective function L of the reconstruction-based unsupervised learning training of the teacher network is constructed T :

[0055]

[0056] In the formula, MAE stands for mean absolute error, The smaller it is, the closer the reconstructed normal time window is to the input normal time window. When training the teacher network, by minimizing the above objective function L T To optimize the parameters of the teacher network and enhance the ability of the teacher network to reconstruct normal time windows.

[0057] 3) constructing a lightweight student network with a similar structure to the teacher network but with fewer layers, units and parameters, using the trained teacher network to guide the lightweight student network to implement knowledge distillation for the normal time window in the training set, increasing the ability of the lightweight student network to reconstruct the normal time window, and obtaining an optimized lightweight student network; it includes the following steps:

[0058] 3.1) Build a lightweight student network with few parameters as a lightweight time window reconstructor. Its modules from low to high layers are: 1 layer of 2-head Transformer encoder, feedforward neural network layer, 1 layer of LSTM decoder and fully connected network layer; among them, the input normal time window W input It includes six features: sending time, pseudonym ID, location x, location y, speed x and speed y, and the time window size is 50, so W input The dimension is 50*6. After passing through the linear embedding layer in the Transformer encoder, it is mapped to a high-dimensional representation of 50*64. After passing through the entire Transformer encoder, the dimension is still 50*64; the feedforward neural network layer includes two fully connected layers, and the LeakyReLU activation function is used between the two fully connected layers. The first fully connected layer expands the dimension to 50*256, and the second fully connected layer compresses the dimension back to 50*64; the LSTM decoder contains 256 units. After passing through the LSTM decoder, the dimension is expanded to 50*256; finally, after passing through the fully connected network layer, the dimension is mapped to 50*2, and the final output is the normal time window W output It only contains two features: position x and position y;

[0059] 3.2) Since the input normal time window W input And output normal time window W output The dimensions of are different, so only the position x and position y of the six features in the input normal time window are used to calculate the reconstruction loss with the output normal time window, where the input normal time window W input In the example, only the normal time window of position x and position y among the six features is taken as The dimension is 50*2; the output reconstruction time window is directly used as the reconstruction window of the input normal time window, denoted as The dimension is 50*2; the objective function of unsupervised learning training based on reconstruction of the student network is constructed

[0060]

[0061] In the formula, MAE stands for mean absolute error, The smaller it is, the closer the reconstructed normal time window is to the input normal time window. When training the lightweight student network, by minimizing the above objective function To optimize the parameters of the lightweight student network and enhance the ability of the lightweight student network to reconstruct the normal time window;

[0062] The lightweight student network not only needs to learn the ability to reconstruct normal time windows, but also needs to accept knowledge transferred from the teacher network. First, a knowledge distillation method based on feature transfer is defined, called FKD, and three features are defined: the output of the input data after passing through the Transformer encoder and the feedforward neural network layer is defined as the encoding feature EF, the output of the data after passing through the LSTM decoder is defined as the decoding feature DF, and the output of the data after passing through the fully connected network layer is defined as the output feature OF. The objective function of the lightweight student network knowledge distillation method FKD is constructed.

[0063]

[0064] In the formula, MSE is the mean square error, EF is T represents the encoding features of the teacher network, EF S Represents the encoding features of the lightweight student network, DF T Denotes the decoding features of the teacher network, DF S represents the decoding features of the lightweight student network, OF T represents the output features of the teacher network, OF S Represents the output features of the lightweight student network; since the mean square error of three features is used, it is multiplied by the coefficient in front

[0065] Finally, the objective function of the reconstruction-based unsupervised learning training in the lightweight student network is The objective function of the knowledge distillation method FKD Combine proportionally to construct the total objective function L for lightweight student network training S :

[0066]

[0067] In the formula, α and β are hyperparameters used to adjust the objective function With the objective function The weight between them, and α+β1; by minimizing the above total objective function L S , which enables the lightweight student network to accept feature knowledge output from the teacher network while learning to reconstruct normal time windows.

[0068] The trained teacher network and the lightweight student network were subjected to multiple experiments on the test set, and the threshold with the best performance was selected. The experimental results of this experiment are described in detail below:

[0069] According to the final detection results of the network, the accuracy and speed of the network are evaluated from the indicators of accuracy, precision, recall and F1 score. The results are shown in Table 1.

[0070] Table 1 Network performance comparison

[0071]

[0072]

[0073] The results in the above table show that the comprehensive performance of the teacher network and the lightweight student network in the present invention are better than the previous IoV anomaly detection model, achieving high detection performance.

[0074] The comparison of the number of parameters and prediction speed of the teacher network, lightweight student network and previous models is shown in Table 2.

[0075] Table 2 Comparison of parameter quantities and prediction speed

[0076]

[0077] The results in the above table show that after knowledge distillation, the number of parameters of the student network is reduced, the reasoning time is significantly reduced, and the reasoning speed is improved.

[0078] 4) The trained lightweight student network is tested using the test set, and the threshold with the highest detection accuracy on the test set is selected as the preset threshold θ, which is deployed together with the lightweight student network to the roadside unit in the Internet of Vehicles for real-time anomaly detection.

[0079] The roadside unit creates a time window for the message sent by each vehicle with a window size of 50 and a step size of 10, and only retains six fields: sending time, pseudonym ID, location x, location y, speed x, and speed y. Finally, the time window is normalized using the z-score operation.

[0080] like Figure 4 As shown in Figure 1, the time window to be tested is input into the lightweight student network. After reconstruction by the lightweight student network, the anomaly score of the time window to be tested is calculated. The reconstruction error of the time window is used as its anomaly score.

[0081]

[0082] In the formula, MAE represents the mean absolute error, Indicates the time window to be tested; express After processing, only the time window after the two features of position x and position y is retained; It means that after the input data is reconstructed by the lightweight student network, the output only contains the reconstructed time window of the two features of position x and position y; the normal time window conforms to the distribution of normal data used in training the lightweight student network, so it can be well reconstructed and has a low anomaly score; while the abnormal time window does not conform to the distribution of normal data used in training the lightweight student network, so it will deviate from the original data after reconstruction, resulting in a large anomaly score;

[0083] The anomaly score is compared with the preset threshold θ. If the anomaly score of the time window exceeds the preset threshold θ, it is judged as abnormal, otherwise it is judged as normal. Finally, the roadside unit will not forward the time window judged as abnormal.

[0084] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A lightweight method for anomaly detection in Internet of Vehicles based on deep learning and knowledge distillation, characterized by: The following steps are involved: 1) Using the public Internet of Vehicles dataset VeReMi Extension, the sliding window strategy is used to preprocess the time series messages of each vehicle in the dataset to obtain the time window. The preprocessed dataset is divided into a training set and a test set. The training set only contains normal time windows, while the test set contains normal time windows and abnormal time windows. 2) Construct a teacher network based on Transformer and LSTM, wherein the Transformer is used as an encoder to capture the dependencies between various positions in the input time window to obtain a compressed representation of the time window information, and the LSTM is used as a decoder to effectively maintain long-distance dependencies through memory units and gating mechanisms, and to restore the information compressed by the Transformer encoding to input data through decoding; the teacher network uses a fully connected neural network as a linear embedding layer to replace the embedding layer in the original Transformer, mapping discrete input message features to a continuous vector space, while expanding the feature dimension of the input so as to better capture the hidden relationship between features later; the teacher network only uses the Transformer encoder to extract important information in the time window, while The decoder is not required to generate sequence output, so the decoder in the original Transformer is removed; the teacher network introduces a feedforward neural network layer after the Transformer and a fully connected network layer after the LSTM. The purpose of introducing the feedforward neural network layer after the Transformer is to integrate the features learned by the encoder and perform nonlinear transformation through layer-by-layer transmission to map high-dimensional data to low-dimensional representation, while the purpose of introducing the fully connected network layer after the LSTM is to map the decoded output to the dimension of the expected output; a large number of normal time windows in the training set are used to train the teacher network using an unsupervised learning method based on reconstruction, so that the teacher network learns the distribution of normal time windows and can restore normal time windows, but cannot restore abnormal time windows that have not been seen during training, thereby achieving the purpose of distinguishing abnormal time windows; 3) Construct a lightweight student network with a similar structure to the teacher network, but with fewer layers, units, and parameters. Use the trained teacher network to guide the lightweight student network to implement knowledge distillation on the normal time window in the training set, increase the ability of the lightweight student network to reconstruct the normal time window, and obtain a trained lightweight student network. 4) The trained lightweight student network is tested using a test set, and the threshold with the highest detection accuracy on the test set is selected as the preset threshold. The preset threshold and the trained lightweight student network are then deployed to the roadside unit in the Internet of Vehicles to perform anomaly detection tasks. The time window to be tested is reconstructed by the lightweight student network, and the reconstruction error of the time window is used as its anomaly score. Finally, the anomaly score is compared with the preset threshold. If the anomaly score exceeds the preset threshold, it is judged as an anomaly, otherwise it is judged as normal.

2. The lightweight method for anomaly detection in Internet of Vehicles based on deep learning and knowledge distillation according to claim 1 is characterized in that: The step 1) comprises the following steps: 1.1) Obtain the public Internet of Vehicles dataset VeReMi Extension and perform data annotation on it. Messages in the dataset that have not been changed by noise or attacks are marked as normal messages, otherwise they are marked as abnormal messages; 1.2) In the dataset, a basic safety message of the Internet of Vehicles carries the following contents: sending time, pseudonym ID, location x, location y, speed x, speed y, acceleration x, acceleration y, direction x and direction y. The above contents are preprocessed as follows: for each message, only the sending time, pseudonym ID, location x, location y, speed x and speed y are selected as input features. For all basic safety messages, the z-score normalization strategy is adopted, and the sliding window strategy is used. A sliding window with a window size of 50 and a step size of 10 is used to intercept the time window. If the time window contains an abnormal message, it is marked as an abnormal time window, otherwise it is a normal time window. 1.3) The preprocessed data set is divided into a training set and a test set, wherein 90% of the normal time windows are used to construct the training set, and the remaining normal time windows and abnormal time windows are used to construct the test set according to the proportion of the original data set.

3. The lightweight method for anomaly detection in Internet of Vehicles based on deep learning and knowledge distillation according to claim 1 is characterized in that: In step 2), the teacher network is constructed and trained, including the following steps: 2.1) Build a teacher network with large model capacity as a time window reconstructor. Its modules from low to high layers are: 1 layer of 16-head Transformer encoder, feedforward neural network layer, 4 layers of LSTM decoder and fully connected network layer; among them, the input normal time window W input It includes six features: sending time, pseudonym ID, location x, location y, speed x and speed y, and the time window size is 50, so W input The dimension is 50*6. After passing through the linear embedding layer in the Transformer encoder, it is mapped to a high-dimensional representation of 50*128. After passing through the entire Transformer encoder, the dimension is still 50*128; the feedforward neural network layer includes two fully connected layers, and the LeakyReLU activation function is used between the two fully connected layers. The first fully connected layer expands the dimension to 50*512, and the second fully connected layer compresses the dimension back to 50*128; each layer of the LSTM decoder contains 256 units. After passing through the LSTM decoder, the dimension is expanded to 50*256; finally, after passing through the fully connected network layer, the dimension is mapped to 50*2, and the final output reconstructs the time window W output It only contains two features: position x and position y; 2.2) Since the input normal time window W input and output reconstruction time window W output The dimensions of are different, so only the input normal time window W is used input The position x and position y of the six features are used to reconstruct the output time window W output Calculate the reconstruction loss, where the input normal time window W input In the example, only the normal time window of position x and position y among the six features is taken as The dimension is 50*2; the output reconstruction time window is directly used as the reconstruction window of the input normal time window, denoted as The dimension is 50*2; construct the objective function L of the teacher network based on the reconstruction of unsupervised learning training T : In the formula, MAE stands for mean absolute error, The smaller it is, the closer the reconstruction time window is to the input normal time window. When training the teacher network, by minimizing the above objective function L T To optimize the parameters of the teacher network and enhance the ability of the teacher network to reconstruct normal time windows.

4. The lightweight method for anomaly detection in Internet of Vehicles based on deep learning and knowledge distillation according to claim 1 is characterized in that: The step 3) comprises the following steps: 3.1) Build a lightweight student network with few parameters as a lightweight time window reconstructor. Its modules from low to high layers are: 1 layer of 2-head Transformer encoder, feedforward neural network layer, 1 layer of LSTM decoder and fully connected network layer; among them, the input normal time window W input It includes six features: sending time, pseudonym ID, location x, location y, speed x and speed y, and the time window size is 50, so W input The dimension is 50*6. After passing through the linear embedding layer in the Transformer encoder, it is mapped to a high-dimensional representation of 50*64. After passing through the entire Transformer encoder, the dimension is still 50*64; the feedforward neural network layer includes two fully connected layers, and the LeakyReLU activation function is used between the two fully connected layers. The first fully connected layer expands the dimension to 50*256, and the second fully connected layer compresses the dimension back to 50*64; the LSTM decoder contains 256 units. After passing through the LSTM decoder, the dimension is expanded to 50*256; finally, after passing through the fully connected network layer, the dimension is mapped to 50*2, and the final output reconstructs the time window W output It only contains two features: position x and position y; 3.2) Since the input normal time window W input and output reconstruction time window W output The dimensions of are different, so only the positions x and y of the six features in the input normal time window are used to calculate the reconstruction loss with the output reconstruction time window, where the input normal time window W input In the example, only the normal time window of position x and position y among the six features is taken as The dimension is 50*2; the output reconstruction time window is directly used as the reconstruction window of the input normal time window, denoted as The dimension is 50*2; the objective function of unsupervised learning training based on reconstruction of the student network is constructed In the formula, MAE stands for mean absolute error, The smaller it is, the closer the reconstruction time window is to the input normal time window. When training the lightweight student network, by minimizing the above objective function To optimize the parameters of the lightweight student network and enhance the ability of the lightweight student network to reconstruct the normal time window; The lightweight student network not only needs to learn the ability to reconstruct normal time windows, but also needs to accept knowledge transferred from the teacher network. First, a knowledge distillation method based on feature transfer is defined, called FKD, and three features are defined: the output of the input data after passing through the Transformer encoder and the feedforward neural network layer is defined as the encoding feature EF, the output of the data after passing through the LSTM decoder is defined as the decoding feature DF, and the output of the data after passing through the fully connected network layer is defined as the output feature OF. The objective function of the lightweight student network knowledge distillation method FKD is constructed. In the formula, MSE is the mean square error, EF is T represents the encoding features of the teacher network, EF S Denotes the encoding features of the lightweight student network, DF T Denotes the decoding features of the teacher network, DF S represents the decoding features of the lightweight student network, OF T represents the output features of the teacher network, OF S Represents the output features of the lightweight student network; since the mean square error of three features is used, it is multiplied by the coefficient in front Finally, the objective function of the reconstruction-based unsupervised learning training in the lightweight student network is The objective function of the knowledge distillation method FKD Combine proportionally to construct the total objective function L for lightweight student network training S : In the formula, α and β are hyperparameters used to adjust the objective function With the objective function The weight between them, and α+β=1; by minimizing the above total objective function L S , which enables the lightweight student network to accept feature knowledge output from the teacher network while learning to reconstruct normal time windows.

5. The lightweight method for anomaly detection in Internet of Vehicles based on deep learning and knowledge distillation according to claim 1 is characterized in that: In step 4), the roadside unit processes the time series sent by each vehicle within the coverage area and converts it into a time window that can be input into the lightweight student network. The time window is reconstructed through the lightweight student network, and the reconstruction error of the time window is used as its anomaly score. In the formula, MAE represents the mean absolute error, Indicates the time window to be tested; express After processing, only the time window after the two features of position x and position y is retained; It means that after the input data is reconstructed by the lightweight student network, the output only contains the reconstructed time window of the two features of position x and position y; the normal time window conforms to the distribution of normal data used in training the lightweight student network, so it can be well reconstructed and has a low anomaly score; while the abnormal time window does not conform to the distribution of normal data used in training the lightweight student network, so it will deviate from the original data after reconstruction, resulting in a large anomaly score; Finally, by comparing the anomaly score with the preset threshold θ, if the anomaly score of the time window exceeds the preset threshold θ, it is judged as abnormal, otherwise it is judged as normal. Finally, the roadside unit will not forward the time window judged as abnormal.

Images