Automobile heterogeneous network intrusion detection method based on QLoRA and knowledge distillation

Abstract

The invention relates to an automobile heterogeneous network intrusion detection method based on QLoRA and knowledge distillation, and belongs to the technical field of intelligent networked automobile on-board network intrusion detection safety. The technical problems that an existing method is difficult to give consideration to light weight and high precision and cannot adapt to resource constraints of a vehicle-mounted embedded environment are solved. According to the technical scheme, the method comprises the steps that self-adaptive mapping and feature extraction are conducted on original data from a CAN bus and the Ethernet, and a unified annotation data set is constructed; a knowledge distillation framework based on BERT is designed, google-bert is used as a teacher model, tiansz-bert is used as a student model, model parameters are compressed by adopting a QLoRA quantification technology, and partial layers are frozen; knowledge of the teacher model is transmitted to the student model through knowledge distillation, and training is carried out in combination with soft and hard label loss. The technical effects are that lightweight intrusion detection is realized, resource consumption is reduced, high detection precision is maintained, and various network attack threats can be identified.

Description

Technical Field

[0001] This invention belongs to the field of in-vehicle network intrusion detection security technology for intelligent connected vehicles, and relates to an intrusion detection method for heterogeneous automotive networks based on Quantization-aware Low-Rank Adaptation (QLoRA) and knowledge distillation. Background Technology

[0002] With the continuous improvement of vehicle intelligence and connectivity, the automotive network architecture has evolved from a single Controller Area Network (CAN) bus to a heterogeneous network environment where CAN bus and in-vehicle Ethernet coexist. The CAN bus is responsible for transmitting real-time control commands such as braking and steering, while the in-vehicle Ethernet undertakes large-volume data transmission tasks such as multimedia and diagnostics. The two types of networks have significant differences in protocol characteristics and data formats, forming a complex heterogeneous network architecture.

[0003] The openness and heterogeneity of such heterogeneous networks expose them to severe intrusion threats. For example, fuzzy attacks trigger logic errors in the Electronic Control Unit (ECU) by sending malformed messages to the CAN bus, while denial-of-service (DoS) attacks cause service disruptions by consuming in-vehicle Ethernet bandwidth. These attacks can tamper with control commands, interfere with the operation of critical systems, and seriously threaten driving safety.

[0004] Existing automotive network intrusion detection methods have significant shortcomings. Traditional rule-based detection methods rely on manually defined attack features, making them difficult to adapt to the complex data formats of heterogeneous networks and exhibiting weak ability to identify unknown attacks. Deep learning-based detection methods, such as Bidirectional Encoder Representations from Transformers (BERT) models, can automatically extract features, but they have a large number of parameters and high computational overhead, making them unsuitable for the limited computing power and memory resources of in-vehicle embedded devices. While model quantization techniques can compress the number of parameters, they are prone to accuracy loss; knowledge distillation can transfer model knowledge, but it lacks deep integration with quantization techniques, making it difficult to achieve an effective balance between lightweight design and detection accuracy.

[0005] Therefore, there is an urgent need in this field for an intrusion detection method for heterogeneous automotive networks that can balance lightweight design with high precision, in order to adapt to the resource constraints of the in-vehicle embedded environment and meet the requirements for real-time detection. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide an intrusion detection method for heterogeneous automotive networks based on QLoRA and knowledge distillation.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for intrusion detection in automotive heterogeneous networks based on QLoRA and knowledge distillation includes the following steps: S1. Obtain public datasets (Automotive Ethernet dataset, CAN Dataset for intrusion detection), and convert tagged CAN bus data (including normal messages and fuzzy attack messages) and Ethernet data (including normal traffic and DOS attack traffic) into text format. The specific process includes: S11. Load Data: Read the dataset file using the Python pandas library, parse the frame ID (hexadecimal), data field (8 bytes), timestamp (milliseconds), and data length code (DLC) fields of the CAN data, and parse the source IP, destination IP, transport layer port (TCP / UDP), packet size, and application layer protocol (SOME / IP / DoIP) fields of the Ethernet data. S12. Data Cleaning: Remove samples with empty timestamps, field values ​​exceeding protocol specifications (e.g., CAN frame ID exceeding the 0x000-0x7FF standard range, Ethernet packet size <64 bytes or >1500 bytes), and attack types undefined (not "normal", "fuzzy", or "dos"); fill missing values ​​in CAN data fields with 0x00; filter Ethernet data samples with abnormal IPs (non-vehicle private network segments 192.168.1.x / 24, 10.0.0.x / 24); S13. Feature Extraction: Extract the features of "frame ID occurrence frequency, period jitter value (deviation between actual period and standard period), and entropy value of each byte in the data field (measures data randomness)" from CAN data; extract the features of "source IP access frequency, number of port connection failures, packet length variation coefficient, and session establishment time" from Ethernet data. S14. Tag Mapping: Construct a mapping relationship between attack types and binary classification tags, where "normal" for CAN data is mapped to 0 (normal) and "fuzzy" is mapped to 1 (attack), and "normal" for Ethernet data is mapped to 0 (normal) and "dos" is mapped to 1 (attack), forming a unified tag system; S15. Feature Concatenation: Concatenate multi-dimensional features in the format of "protocol type + timestamp + core feature". For example, CAN data is concatenated as "CAN,1580000000,0x123,8,0.85,0.32" (protocol type, timestamp, frame ID, data length, period jitter value, and data domain entropy value, respectively), and Ethernet data is concatenated as "Ethernet,1580000000,192.168.1.100,8080,1200,0.25" (protocol type, timestamp, source IP, port, packet size, and length variation coefficient, respectively). S16. Format Conversion: Using the word segmenters of the teacher model and the student model, the concatenated feature text is converted into a token ID sequence that the model can input. The maximum sequence length is set to 64 (to adapt to the computing resources of the in-vehicle environment). Padding is applied to sequences that are not long enough, and sequences that exceed the length are truncated. S17. Dataset partitioning: The dataset is divided into training, validation, and test sets using stratified sampling in a 7:2:1 ratio to ensure that the distribution ratio of the "normal" and "attack" labels in the training, validation, and test sets is consistent, thus avoiding model bias caused by class imbalance. S2. Initialize and fine-tune the teacher model, construct the student model, perform QLoRA quantization and freeze the first two layers for fine-tuning to reduce the number of parameters. The specific process is as follows: S21. Teacher Model Initialization and Fine-tuning: A pre-trained Google-BERT model was selected as the teacher model. This model contains 12 Transformer encoding layers and a 12-head self-attention mechanism, with approximately 1.7G of parameters. The teacher model was fine-tuned using the training set obtained in S1. The optimizer used was ADAMW (learning rate set to 2e-5, weight decay coefficient set to 0.01), and the loss function used was the cross-entropy loss function.

[0009] Where N is the total number of samples in the training set, and N is the number of samples. The true label (0 or 1). The probability that sample i in the model output belongs to the attack category; the fine-tuning process iterates for 3 rounds, and the accuracy is evaluated on the validation set after each round, and the weight of the teacher model with the highest accuracy on the validation set is saved; S22. Student Model Layer Freeze: Freeze the first two Transformer encoding layers of the student model by setting the layer parameter `requires_grad=False`. Only the last four Transformer encoding layers, attention layer, and output layer are kept as trainable, reducing the scale of trainable parameters and reducing the computational overhead of training and inference in the vehicle environment. S23. QLoRA Quantization Processing: The frozen student model is quantized using QLoRA (Quantization-aware Low-Rank Adaptation) technology, specifically as follows: S231. Dynamic Range Calibration: Statistically analyze the activation value distribution of training set samples in each trainable layer of the student model, and calculate the quantization scaling factor α, using the following formula:

[0010] in is the activation value of a certain layer of the model, and b is the quantization bit width. Attention layers use 8-bit quantization to preserve accuracy, and fully connected layers use 4-bit quantization to compress the number of parameters. S232, Low-rank adaptation matrix insertion: Inserting the weight matrix of the trainable layer in the student model. (d is the dimension of the Transformer hidden layer) Insert low-rank matrix pairs (where r is the low-rank dimension), the adapted weight update formula is: , High-dimensional (through low-rank decomposition) ) Parameter update transformed into low dimension ( Matrix operations reduce computational complexity from O( The value decreases to O(2dr); S233, Quantization-Aware Training (QAT): Embedding quantization-dequantization operations during training, with the quantization formula as follows:

[0011] Quantize the floating-point activation value into an integer. The inverse quantization formula is:

[0012] Convert the quantized integer back to a floating-point value. Introducing quantization error loss means:

[0013] Where is the sample i The original activation value, The activation value after dequantization. Joint optimization with cross-entropy loss yields the following formula for total fine-tuning loss:

[0014] in To quantize error weights and balance classification accuracy with quantization error, the quantization fine-tuning is iterated for two rounds, and the quantized student model weights are saved. S3. Establish a knowledge distillation framework to transfer the knowledge from the teacher model to the quantified student model, thereby improving the accuracy of the student model in intrusion detection tasks. The specific process is as follows: S31. Construction of Distillation Loss Function: A joint loss function of "soft label loss + hard label loss" is adopted to achieve effective transfer of knowledge from the teacher model. The formula is as follows: S311, KL divergence soft label loss (distillation loss): measures the difference in the output probability distribution between the teacher model and the student model, and the formula is:

[0015] Where D is the training set. This represents the soft label probability that sample x, output by the teacher model, belongs to category c. The soft label probability output by the student model is T=5.0, which is the temperature parameter (used to smooth the probability distribution and enhance the transmission effect of fine-grained knowledge in the teacher model), and c=0 (normal) and c=1 (attack). S312, Cross-entropy Hard Label Loss (Supervised Loss): Optimizes the classification accuracy of the student model based on the true labels, with the following formula:

[0016] in For the true label of sample i, The probability that sample i belongs to the attack category is output by the student model; S313, Total Distillation Loss: Weighted fusion loss, formula is as follows:

[0017] Where α=0.7 is the soft label loss weight, to ensure that the teacher's model knowledge plays a dominant role in transmission, while retaining the supervisory role of the real labels; S32. Soft Label Generation for Teacher Model: Input the training set samples into the optimal teacher model stored in S21, and calculate the soft label probability using the softmax function. The formula is as follows:

[0018] in This is the logits output of the teacher model for category c, where T=5.0 is the temperature parameter; S33. Distillation Training Execution: Fix the teacher model parameters (do not participate in updates), input the training set samples and corresponding soft and hard labels into the quantized student model saved in S24; use the AdamW optimizer (learning rate set to 1e-4, weight decay coefficient of 0.05) to minimize the total distillation loss. Train the model for the target; iterate the training for 6 rounds, evaluate the accuracy of the student model on the validation set after each round, and save the weight of the student model with the highest accuracy on the validation set to avoid overfitting; S4. Model Performance Evaluation and Saving: The performance of the student model is evaluated before and after distillation. The optimization effect is compared using accuracy, precision, recall, and F1 score. The iteratively optimized student model is then saved. The specific process is as follows: S41. Performance Metric Calculation: Input the test set samples from S1 into the student models of "QLoRA Quantization Only (Before Distillation)" and "QLoRA Quantization + Knowledge Distillation (After Distillation)" respectively, and calculate the four core performance metrics. The formulas are as follows: Accuracy (Acc): Measures the overall correctness of the model's classification. The formula is:

[0019] Where TP represents the number of correctly identified attack samples, TN represents the number of correctly identified normal samples, FP represents the number of normal samples that were misidentified as attacks, and FN represents the number of attack samples that were misidentified as normal. Precision (Prec): Measures the accuracy with which the model identifies attack samples. The formula is:

[0020] Recall (Rec): Measures the model's ability to cover attack samples, and is calculated using the following formula:

[0021] F1 score: A metric that balances precision and recall; the formula is:

[0022] S42. Comparison of optimization effects: The F1 score of the student model after distillation is significantly improved compared with that before distillation, and the precision and recall are also significantly improved, ensuring the optimization effect of knowledge distillation on model performance. S43. Model Saving: Save the student model with the best performance after distillation (including QloRA quantization parameters, DistilBERT segmenter, and model weight file) to the specified directory to meet the real-time intrusion detection requirements of heterogeneous automotive networks.

[0023] In S1, for the obtained public dataset, it is first converted to a different number system, and then normalized. S21. Hexadecimal Conversion: This mainly targets the hexadecimal fields (frame ID, data field) of CAN data and the hexadecimal payload of Ethernet data. The conversion formula is as follows:

[0024] in n is the decimal value corresponding to the hexadecimal character, such as 10 for the character "A" and 15 for "F", and n is the length of the hexadecimal string; S22. Normalization Processing: For continuous features after number system conversion (such as timestamps, data lengths, data packet sizes, and periodic jitter values), select an appropriate normalization method based on the feature distribution characteristics: When the feature has no extreme outliers, use Min-Max normalization, with the following formula:

[0025] The feature values ​​are mapped to the [0,1] interval; when there are extreme outliers in the feature, such as extremely large data packets caused by Ethernet DoS attacks, Z-Score normalization is used, and the formula is:

[0026] Where μ is the characteristic mean. The standard deviation of the features is used to suppress the interference of outliers on the feature distribution; after normalization, it is ensured that the numerical scale of all continuous features is consistent, avoiding weight bias caused by differences in feature magnitude during model training.

[0027] In S21, the learning rate for fine-tuning the teacher model adopts a "cosine annealing decay" strategy, with an initial learning rate of 2e-5. After each training round, the learning rate is calculated using the following formula:

[0028] in Let be the learning rate in round t. The learning rate (T is the total number of fine-tuning rounds) is decayed to avoid parameter oscillations caused by an excessively high learning rate in the later stages of training, thereby improving the stability of the teacher model's fine-tuning.

[0029] A method for intrusion detection in automotive heterogeneous networks based on QLoRA-DistilBERT, characterized in that: in S23, the low-rank matrices A and B quantized by QLoRA are initialized using Xavier, and the initialization formula is:

[0030] Where U represents uniform distribution, d is the dimension of the Transformer hidden layer, and r is the low-rank dimension, ensuring the adaptability of the initialized low-rank matrix to the model weights and avoiding gradient vanishing or gradient exploding problems caused by improper parameter initialization in the early stage of training.

[0031] In S31, the value of the temperature parameter T is determined through a grid search, with a search range of [missing information]. The F1 score of the student model after distillation was used as the evaluation index. The reason for finally choosing T=5.0 is that when T<3, the probability distribution of the soft label output by the teacher model is too steep, and the student model has difficulty learning fine-grained knowledge; when T>7, the probability distribution is too smooth, and the difference between the soft label and the hard label is too small, which defeats the purpose of knowledge distillation; when T=5.0, the weighted fusion effect of KL divergence loss and cross-entropy loss is optimal, and the F1 score of the student model after distillation reaches the highest.

[0032] In S33, the distillation training process employs a "gradient accumulation" strategy. When the onboard training device has insufficient memory (e.g., ≤100MB), the batch size for a single training iteration (e.g., 32) is split into four micro-batches (each micro-batch contains eight samples). After calculating the loss gradient for each micro-batch, the model parameters are not updated. The gradients of all four micro-batches are accumulated before a unified parameter update. The gradient accumulation formula is:

[0033] Where K=4 is the micro-batch size. The loss gradient of the kth micro-batch; this strategy reduces memory usage to 1 / 4 of the original requirement without reducing training effectiveness, adapting to the limited memory resources of in-vehicle equipment.

[0034] The beneficial effects of this invention are as follows: (1) This invention can effectively support heterogeneous data formats of Controller Area Network (CAN) bus and vehicle Ethernet, and eliminates the differences between different protocols through a unified feature processing and tag mapping strategy. This design allows the method to be seamlessly applied to complex vehicle network environments without the need to customize detection rules for specific protocols, greatly improving the system's versatility and deployment flexibility.

[0035] (2) By employing quantization-aware low-rank adaptation technology to quantize and compress the parameters of the student model, and combining this with a partial network layer freezing strategy, this invention significantly reduces the number of model parameters and computational complexity while maintaining the core functionality of the model. This lightweight design enables the detection model to run efficiently on resource-constrained in-vehicle embedded devices, effectively reducing memory usage and energy consumption, and meeting the practical needs of real-time intrusion detection.

[0036] (3) This invention uses a knowledge distillation framework to transfer the rich knowledge of the teacher model to the lightweight student model, effectively compensating for the accuracy loss that may result from quantization compression. This method enables the student model to maintain its lightweight characteristics while possessing classification capabilities close to those of a large teacher model, and can accurately identify various network attack behaviors, such as fuzzy attacks and denial-of-service (DoS) attacks, thereby improving the overall reliability and robustness of detection.

[0037] (4) This invention optimizes resource allocation in the training and inference processes through a quantization-distillation approach. Quantization reduces model storage requirements, while knowledge distillation ensures efficient knowledge transfer, enabling the system to achieve rapid response even with limited computing resources. This efficiency improvement helps reduce the overall cost of the vehicle system and supports long-term stable operation.

[0038] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall process of the method of the present invention; Figure 2 A detailed flowchart illustrating the textification process for multi-source heterogeneous data. Figure 3 A schematic diagram illustrating the technical principle of QLoRA quantization freeze fine-tuning; Figure 4 A schematic diagram of the framework for knowledge distillation training; Figure 5 A simplified diagram for knowledge distillation. Detailed Implementation

[0040] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0041] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0042] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0043] Combination Figure 1 The present invention provides a detailed description of the specific implementation process of a QLoRA-DistilBERT-based automotive heterogeneous network intrusion detection method. The core of this invention lies in achieving synergistic optimization of lightweight automotive heterogeneous network intrusion detection model and detection accuracy through a technical path of "multi-source heterogeneous data textualization processing → teacher-student model construction and QLoRA quantization fine-tuning → freezing → knowledge distillation training → model performance evaluation and storage," thus adapting to the resource constraints of in-vehicle embedded devices.

[0044] I. Basic Description of Implementation Examples This embodiment uses publicly available datasets in the field of automotive network security (Automotive Ethernet dataset, CANDataset for intrusion detection) as the data source. These datasets cover CAN bus data (including normal messages and fuzzy attack messages) and Ethernet data (including normal traffic and DoS attack traffic). The two types of data follow the CAN2.0B protocol and the IEEE 802.3 Ethernet protocol, respectively, and have heterogeneous characteristics in frame structure, transmission rate, and data format.

[0045] II. Specific Implementation Steps S1. Multi-source heterogeneous data text processing like Figure 2 As shown, this step aims to convert the heterogeneous raw data from the CAN bus and Ethernet into a structured text sequence that the model can input, eliminating data format differences and constructing a unified labeled dataset. The specific process is as follows: S11, Data Loading and Field Parsing By reading the dataset file and parsing the core fields according to protocol type, the complete extraction of communication features is ensured. CAN data: Parse frame ID, data field, timestamp, and data length code; Ethernet data: Resolve source IP, destination IP, transport layer port, packet size, and application layer protocol identifier.

[0046] S12, Data Cleaning and Anomaly Filtering Remove invalid samples and handle outlier data to ensure data compliance and validity: Removal rules: Samples with empty timestamps, CAN frame IDs exceeding the 0x000-0x7FF standard range, Ethernet packet sizes less than 64 bytes (minimum frame length) or greater than 1500 bytes (MTU value), and attack types that are not defined (not "normal", "fuzzy", or "dos"); Missing value handling: For samples with missing data fields in the CAN data, fill them with "0x00"; Abnormal IP filtering: Directly filter IP samples in Ethernet data that do not belong to the vehicle's private network segment (such as 10.1.2.3) to avoid invalid data interfering with model training.

[0047] S13, Differential Feature Extraction Based on the protocol characteristics of the two types of data, core features that can accurately distinguish between normal and attack behaviors are extracted: CAN data characteristics: 1. Frame ID occurrence frequency: The number of times the same frame ID appears per unit of time (e.g., 10 times / second); 2. Period jitter value: The percentage deviation between the actual transmission period and the standard period. Formula: ; 3. Entropy values ​​of each byte in the data field: obtained using the Shannon entropy formula:

[0048] Where pi is the probability distribution of the i-th byte. The higher the entropy value, the stronger the randomness of the data, and the higher the probability of it being an attack packet. Ethernet data characteristics: 1. Source IP access frequency: The number of times the same source IP is accessed per unit of time (e.g., 5 times / minute); 2. Number of port connection failures: The number of times the target port connection times out or is rejected per unit of time (e.g., 3 times / minute); 3. Coefficient of variation for message length: The ratio of the standard deviation of message length to the mean (Formula: ); 4. Session establishment time: The time difference between the TCP SYN packet and the ACK packet (e.g., 0.5 seconds).

[0049] S14, Tag Mapping Construct a unified binary classification label system to eliminate label differences in heterogeneous data and ensure consistent model training objectives: CAN data: "normal" (normal message) → 0, "fuzzy" (fuzzy attack message) → 1; Ethernet data: "normal" (normal traffic) → 0, "dos" (DoS attack traffic) → 1.

[0050] S15, Feature splicing Multi-dimensional features are concatenated in a fixed format of "protocol type + timestamp + core features" to form a structured text sequence, ensuring the consistency of input data format; S16, Format Conversion A word segmenter using student and teacher models converts feature text into a sequence of token IDs that the model can input, adapting to in-vehicle computing resources: the maximum sequence length is set to 64 (to avoid memory overflow of in-vehicle devices). For sequences shorter than 64, padding with 0 values ​​is used; for sequences longer than 64, they are truncated from the end to ensure that the input sequences have a uniform length.

[0051] S2, Teacher-Student Model Construction and QloORA Quantization Freeze Fine-tuning like Figure 3 As shown, this step constructs a lightweight student model by first using QLoRA quantization to compress the number of parameters, and then freezing the parameters to reduce the number of trainable parameters. This process simultaneously controls the accuracy loss caused by quantization and freezing. The specific steps are as follows: S21. Teacher Model Initialization and Fine-tuning Model Selection: A pre-trained Google-BERT model was selected as the teacher model. This model contains 12 Transformer encoding layers, 12 self-attention mechanisms, approximately 1.7G of parameters, and possesses strong feature extraction and classification capabilities. Its fine-tuning configuration is as follows: 1. Optimizer: AdamW is used, employing a cosine annealing decay strategy. The optimization is performed by decaying the optimization after each training round, using the following formula:

[0052] in 1. Initial learning rate, t is the current round, T is the total number of fine-tuning rounds, and the weight decay coefficient is 0.01; 2. Loss function: Cross-entropy loss function, the formula is:

[0053] Where N is the total number of samples. For the true label of sample i, Output the probability that sample i belongs to the attack category for the model; S22, Student Model Construction and Initialization The pre-trained tiansz-bert model was selected as the student model. This model reduces the number of parameters to about 400M while retaining 97% of BERT's performance by removing the TokenTypeEmbeddings layer and simplifying the attention calculation logic, thus laying a lightweight foundation for subsequent quantization and freezing. After loading the pre-trained weights, the parameters of each layer of the model were initialized to ensure that the model has basic classification capabilities in the initial state.

[0054] S23. Student model Qlora quantization processing (quantization first) The QLoRA technique is used to quantize the complete student model, prioritizing the reduction of parameter count and computational complexity. The specific steps are as follows: S231. Dynamic Range Calibration: Statistically analyze the activation value distribution of training set samples across all Transformer encoding layers, attention layers, and fully connected layers of the student model, and calculate the quantization scaling factor α, with the following formula:

[0055] in a The layer activation value, b To quantize the bit width, different bit widths are set according to the functional characteristics of different layers: the attention layer uses 8-bit quantization (b=8, to ensure the accuracy of the attention mechanism in extracting fine-grained features), and the fully connected layer uses 4-bit quantization (b=4, to maximize the compression of parameters). S232, Low-rank adaptation matrix insertion: Inserting the weight matrix of the trainable layer in the student model. (d is the dimension of the Transformer hidden layer) Insert low-rank matrix pairs (where r is the low-rank dimension), the adapted weight update formula is: , High-dimensional (through low-rank decomposition) ) Parameter update transformed into low dimension ( Matrix operations reduce computational complexity from O( The value decreases to O(2dr); S233, Quantization-Aware Training (QAT): Embedding quantization-dequantization operations during training, with the quantization formula as follows:

[0056] Quantize the floating-point activation value into an integer. The inverse quantization formula is:

[0057] Convert the quantized integer back to a floating-point value. Introducing quantization error loss means:

[0058] Where is the sample i The original activation value, The activation value after dequantization. Joint optimization with cross-entropy loss yields the following formula for total fine-tuning loss:

[0059] in To quantize error weights and balance classification accuracy with quantization error, the quantization fine-tuning is iterated for two rounds, and the quantized student model weights are saved. S24. Freeze the student model layer (frozen later). Based on the quantized student model, non-critical layer parameters are further frozen to reduce the size of trainable parameters. The selection criteria for frozen layers are as follows: by analyzing the gradient contribution values ​​of each layer in the quantized student model, layers with gradient contribution values ​​below a preset threshold (e.g., 0.05) are identified as non-critical layers, such as the first two Transformer encoding layers, and are thus determined to be freezeable. S3, Knowledge Distillation Training like Figure 4 As shown, this step involves building a knowledge distillation framework to transfer high-quality knowledge from the teacher model to the student model, which is "quantified first and then frozen," thus compensating for the accuracy loss caused by quantization and freezing and improving intrusion detection accuracy. The specific process is as follows: S31. Construction of Distillation Loss Function: A joint loss function of "soft label loss + hard label loss" is adopted to achieve effective transfer of knowledge from the teacher model. The formula is as follows: S311, KL divergence soft label loss (distillation loss): measures the difference in the output probability distribution between the teacher model and the student model, and the formula is:

[0060] Where D is the training set. This represents the soft label probability that sample x, output by the teacher model, belongs to category c. The soft label probability output by the student model is T=5.0, which is the temperature parameter (used to smooth the probability distribution and enhance the transmission effect of fine-grained knowledge in the teacher model), and c=0 (normal) and c=1 (attack). S312, Cross-entropy Hard Label Loss (Supervised Loss): Optimizes the classification accuracy of the student model based on the true labels, with the following formula:

[0061] in For the true label of sample i, The probability that sample i belongs to the attack category is output by the student model; S313, Total Distillation Loss: Weighted fusion loss, formula is as follows:

[0062] Where α=0.7 is the soft label loss weight, to ensure that the teacher's model knowledge plays a dominant role in transmission, while retaining the supervisory role of the real labels; S32. Soft Label Generation for Teacher Model: Input the training set samples into the optimal teacher model stored in S21, and calculate the soft label probability using the softmax function. The formula is as follows:

[0063] in This is the logits output of the teacher model for category c, where T=5.0 is the temperature parameter; S33. Distillation Training Execution: Fix the teacher model parameters (do not participate in updates), input the training set samples and corresponding soft and hard labels into the quantized student model saved in S24; use the AdamW optimizer (learning rate set to 1e-4, weight decay coefficient of 0.05) to minimize the total distillation loss. Train the model for the target; iterate the training for 6 rounds, evaluate the accuracy of the student model on the validation set after each round, and save the weight of the student model with the highest accuracy on the validation set to avoid overfitting; S4. Model Performance Evaluation and Saving: The performance of the student model is evaluated before and after distillation. The optimization effect is compared using accuracy, precision, recall, and F1 score. The iteratively optimized student model is then saved. The specific process is as follows: S41. Performance Metric Calculation: Input the test set samples from S1 into the student models of "QLoRA Quantization Only (Before Distillation)" and "QLoRA Quantization Freeze + Knowledge Distillation (After Distillation)" respectively, and calculate the four core performance metrics. The formulas are as follows: Accuracy (Acc): Measures the overall correctness of the model's classification. The formula is:

[0064] Where TP represents the number of correctly identified attack samples, TN represents the number of correctly identified normal samples, FP represents the number of normal samples that were misidentified as attacks, and FN represents the number of attack samples that were misidentified as normal. Precision (Prec): Measures the accuracy with which the model identifies attack samples. The formula is:

[0065] Recall (Rec): Measures the model's ability to cover attack samples, and is calculated using the following formula:

[0066] F1 score: A metric that balances precision and recall; the formula is:

[0067] S42. Optimization Effect Verification Establish optimization performance evaluation criteria to verify the effectiveness of the collaborative optimization approach of "quantification followed by freezing + knowledge distillation": First verification: Compare the performance metrics of the "quantization only" and "quantization + freeze" models, requiring that the F1 value of the "quantization + freeze" model decrease by ≤10%, to ensure that the impact of freezing non-critical layers on accuracy is controllable; The second verification: compare the performance metrics of the "quantization + freezing" and "quantization + freezing + distillation" models. The "quantization + freezing + distillation" model is required to improve the F1 score by ≥30%, and improve the precision and recall by ≥30%, to ensure that knowledge distillation effectively compensates for the precision loss caused by quantization and freezing.

[0068] Figure 5 A simplified diagram for knowledge distillation.

[0069] III. Verification of Examples S31, Example Parameter Configuration Datasets: CAN Dataset for intrusion detection (containing normal messages and fuzzy attack messages) and Automotive Ethernet dataset (containing normal traffic and DoS attack traffic) were selected. Model parameters: The teacher model uses "google-bert", and the student model uses "tiansz-bert"; QloRA quantization low-rank dimension r=64, attention layer 8-bit quantization, fully connected layer 4-bit quantization; freeze the first 2 Transformer encoding layers of the student model; distillation temperature parameter T is determined to be 5.0 through grid search; Training configuration: 18 rounds of fine-tuning for the teacher model, 6 rounds of quantization fine-tuning for the student model, and 18 rounds of distillation training; during training, the model is divided into training set, validation set and test set using stratified sampling in a 7:2:1 ratio to ensure that the distribution ratio of the "normal" and "attack" labels in the training set, validation set and test set is consistent, and to avoid model bias caused by class imbalance. S32, Verification steps of the embodiment 1. Data preprocessing verification: Perform feature distribution verification on the training set and test set after S1 processing, check whether the distribution ratio of "normal" and "attack" labels is consistent, and whether the numerical range of core features (such as CAN data field entropy value and Ethernet packet length variation coefficient) conforms to the protocol specification, to ensure the effectiveness of data preprocessing; 2. Model Lightweighting Validation: The number of parameters and computational complexity (measured in floating-point operations per second) of the "original student model", "quantization-only model" and "quantization + freeze model" were statistically analyzed respectively. The model lightweighting effect at each stage was compared to confirm the reduction ratio of parameter number and computational complexity. 3. Progressive performance verification: In the order of "quantization only → quantization + freezing → quantization + freezing + distillation", the test set samples are input into the three types of models in sequence, and the accuracy, precision, recall and F1 score of each type of model are calculated and recorded to verify whether the performance changes at each stage meet the preset optimization standards. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intrusion detection in automotive heterogeneous networks based on QLoRA and knowledge distillation, characterized in that: Includes the following steps: The data preprocessing step involves acquiring and processing the raw data of the heterogeneous automotive network to generate a unified labeled dataset. The model building and quantization steps include initializing and fine-tuning the teacher model, building the student model and performing QLoRA quantization and freezing some parameters. The knowledge distillation process involves building a knowledge distillation framework and using the teacher model output and real labels to construct a joint loss function to train the student model. The system also includes performance evaluation and saving steps, which involve evaluating the performance of the student model and saving the optimized model.

2. The automotive heterogeneous network intrusion detection method based on QLoRA and knowledge distillation according to claim 1, characterized in that: The data preprocessing steps include number system conversion and normalization of the data; The number system conversion converts hexadecimal fields to decimal values. The conversion formula is as follows: ;in This represents the decimal value corresponding to a hexadecimal character. n The length of the string; The normalization process is applied to continuous features, and the Min-Max normalization formula is used when there are no extreme outliers: When extreme outliers exist, the Z-Score normalization formula is used. , where μ is the characteristic mean and σ is the characteristic standard deviation.

3. The automotive heterogeneous network intrusion detection method based on QLoRA and knowledge distillation according to claim 1, characterized in that: The teacher model fine-tuning employs a cosine annealing learning rate decay strategy, with the initial learning rate set to 2×10⁻⁶. -5 After each round of training, follow the formula Attenuation is performed, where For the first t Learning rate of the round, The initial learning rate, T This is the total number of fine-tuning rounds.

4. The automotive heterogeneous network intrusion detection method based on QLoRA and knowledge distillation according to claim 1, characterized in that: The low-rank adaptation matrices A and B in the QLoRA quantization process are initialized using Xavier, with the initialization formula being: and ,in U Indicates uniform distribution. d This represents the hidden layer dimension of the Transformer.

5. The automotive heterogeneous network intrusion detection method based on QLoRA and knowledge distillation according to claim 1, characterized in that: Temperature parameters in the knowledge distillation step T The range of values ​​was determined through grid search. Finally selected T =5.0 to optimize the probability distribution of soft tags.

6. The automotive heterogeneous network intrusion detection method based on QLoRA and knowledge distillation according to claim 1, characterized in that: The knowledge distillation training employs a gradient accumulation strategy. When device memory is insufficient, a single training batch is split into multiple micro-batches. After accumulating gradients, the parameters are updated uniformly. The gradient accumulation formula is as follows: ,in K This refers to the quantity in a micro-batch. For the first k The loss gradient of each micro-batch.

7. The automotive heterogeneous network intrusion detection method based on QLoRA and knowledge distillation according to claim 1, characterized in that: The feature extraction in the data preprocessing step includes extracting the frame ID occurrence frequency, periodic jitter value, and data domain entropy value from CAN data, and extracting the source IP access frequency, port connection failure number, message length variation coefficient, and session establishment duration from Ethernet data.

8. The automotive heterogeneous network intrusion detection method based on QLoRA and knowledge distillation according to claim 1, characterized in that: The QLoRA quantization process in the model construction and quantization steps includes dynamic range calibration, low-rank fitting matrix insertion, and quantization-aware training; dynamic range calibration calculates the quantization scaling factor. ,in As the activation value, b This is for quantization bit width; The formula for updating low-rank adaptation weights is: The quantization-perception training loss function is: ,in For cross-entropy loss, To account for quantization error loss, λ =0.

1.

9. The automotive heterogeneous network intrusion detection method based on QLoRA and knowledge distillation according to claim 1, characterized in that: The joint loss function in the knowledge distillation step is: ,in The KL divergence loss is given by the formula: , The cross-entropy hard-label loss is given by the formula: , α =0.7 is the weight parameter.

10. The automotive heterogeneous network intrusion detection method based on QLoRA and knowledge distillation according to claim 1, characterized in that: The performance evaluation and storage steps include calculating the accuracy rate. Accuracy Recall rate and F1 value ,in TP For a real example, TN For a true negative example, FP As a false positive example, FN This is a false negative.

Images