Vehicle-mounted heterogeneous network security detection method based on memory mechanism and MoE architecture

Abstract

The application relates to a vehicle-mounted heterogeneous network security detection method based on a memory mechanism and an MoE architecture, and belongs to the technical field of vehicle-mounted network security detection of intelligent networked vehicles. The method comprises the following steps: a teacher model of the MoE architecture is constructed by fine-tuning a pre-trained language model in a directional manner and incorporating a memory mechanism, the teacher model containing three special types of MoE architecture, namely, Ethernet Fuzzy attack detection, CAN bus DoS attack detection and abnormal packet positioning, so that accurate identification of vehicle-mounted heterogeneous network attacks and rapid tracing of abnormal packets are realized. Meanwhile, a student model optimization scheme of QLoRA fine-tuning and knowledge distillation cooperation is proposed, a lightweight MoE architecture aligned with the function of the teacher model is constructed, and the soft knowledge of the teacher model is migrated. The student model can still retain the core detection capability under the condition that the parameter quantity is reduced by more than 90%. The application can greatly improve the pertinence, real-time performance and positioning accuracy of vehicle-mounted heterogeneous network attack detection without additional deployment of hardware.

Description

Technical Field

[0001] This invention belongs to the field of vehicle network security detection technology for intelligent connected vehicles, and relates to a method for detecting intrusions into heterogeneous vehicle networks that integrates a memory mechanism and a hybrid expert (MoE) architecture. Background Technology

[0002] With the rapid development of automotive intelligence and connectivity technologies, in-vehicle networks have evolved from a single CAN bus architecture to a heterogeneous network system where CAN bus and in-vehicle Ethernet work together. The CAN bus handles the transmission of core control commands such as braking and steering, characterized by high real-time performance and reliability; while the in-vehicle Ethernet is responsible for transmitting large volumes of data such as multimedia data and diagnostic information, boasting advantages such as high bandwidth and high transmission speed. The significant differences in the characteristics of these two network protocols and data formats have created a complex heterogeneous network environment.

[0003] However, the openness and complexity of heterogeneous networks expose them to severe security threats. On the one hand, Ethernet is vulnerable to denial-of-service (DoS) attacks, where attackers can consume network bandwidth by sending massive amounts of invalid messages, leading to the paralysis of core services. On the other hand, the CAN bus is susceptible to fuzzy attacks, where attackers can forge legitimate frame IDs to send malicious messages, tamper with control commands, and seriously threaten driving safety. Furthermore, locating abnormal messages in heterogeneous networks is extremely difficult.

[0004] In summary, current in-vehicle heterogeneous network security detection technologies have the following problems: 1. Weak targeting of attack detection: Existing detection models mostly use a single network structure to process heterogeneous data, and do not design special detection modules for the characteristic differences of different attack types, resulting in insufficient accuracy in identifying specific threats such as DOS attacks and fuzzy attacks.

[0005] 2. Conflict between real-time performance and information redundancy: Traditional detection methods rely on sliding windows to extract time-series features. The data within the window is repeatedly calculated, which not only increases the computational cost but also causes detection delays, failing to meet the real-time response requirements of vehicle networks.

[0006] 3. Difficulty in balancing lightweight design and detection accuracy: High-precision detection models based on deep learning have a large number of parameters and high computational complexity, which cannot be adapted to the limited computing power and memory resources of automotive embedded devices; while traditional lightweighting methods such as quantization and pruning can easily lead to a significant drop in model performance, making it difficult to balance lightweight design and detection accuracy.

[0007] 4. Low accuracy in locating abnormal messages: Existing location methods rely on only a single feature (such as timestamp or protocol field) without combining historical message context information, resulting in large location errors and an inability to accurately pinpoint the time location and protocol level of abnormal messages.

[0008] In summary, there is an urgent need for an intrusion detection and abnormal packet location method that can adapt to heterogeneous in-vehicle network environments and is targeted, real-time, lightweight, and highly accurate, in order to solve the problems existing in the current technology. Summary of the Invention

[0009] In view of this, the purpose of this invention is to provide an in-vehicle heterogeneous network intrusion detection method that integrates memory mechanism and MoE architecture, so as to achieve accurate detection of Ethernet DOS attacks and CAN bus fuzzy attacks and efficient location of abnormal messages, and adapt to the resource constraints of in-vehicle embedded devices.

[0010] To achieve the above objectives, the present invention provides the following technical solution: A vehicle-mounted heterogeneous network security detection method based on memory mechanism and MoE architecture is characterized by acquiring heterogeneous raw data from CAN bus and Ethernet, preprocessing it into a structured feature dataset suitable for the model, and dividing it into training set, validation set and test set. A teacher model based on the MoE architecture with an integrated memory mechanism is constructed. The teacher model is fine-tuned based on a pre-trained language model. A hybrid expert architecture is built and three types of specialized expert modules are configured: an Ethernet DOS attack detection module, a CAN bus fuzzy attack detection module, and an abnormal message localization module. At the same time, a memory mechanism is embedded to replace the sliding window. The teacher model is trained using a prepared feature dataset. A lightweight MoE architecture student model is constructed that is functionally aligned with the teacher model. The student model fine-tunes its core parameters by using QLoRA low-rank adaptation on a pre-trained language model, and then transfers the soft knowledge of the teacher model through distillation technology to complete the training and optimization of the student model. After the teacher model and student model are trained, for actual vehicle-mounted heterogeneous network packets, by extracting features, linking the output results of the abnormal packet localization module in the teacher model with the historical packet context stored in the memory mechanism, the student model can realize the localization of abnormal packets in time series and protocol layer.

[0011] Furthermore, the preprocessing of heterogeneous network data in vehicles includes: Parse key fields of CAN and Ethernet data, remove invalid samples, fill missing values ​​in CAN data fields, and filter abnormal IPs in Ethernet data. Features of CAN data and Ethernet data are extracted separately. CAN data features include frame ID occurrence frequency, period jitter value, and data field entropy value. Ethernet data features include source IP access frequency, number of port connection failures, message length variation coefficient, and session establishment time. The tags are uniformly mapped to a binary classification format and concatenated according to the format of "protocol type + timestamp + core feature"; The feature text is converted into a token ID sequence, and a stratified sampling method is used to divide it into training set, validation set and test set.

[0012] Furthermore, the teacher model is fine-tuned based on a pre-trained language model. The hybrid expert architecture includes a gating network and three types of specialized expert modules. The fusion output of the hybrid expert architecture is represented as follows:

[0013] In the formula, To integrate the output results of the hybrid expert architecture, The weights of the i-th type of expert module output by the gating network are: These are the preprocessed structured features; This is the output of the Ethernet DoS attack detection module. This is the output of the CAN bus Fuzzy attack detection module. Output to the abnormal message location module; The gating network uses a two-layer fully connected network to output the weights of each expert module. The output of the gating network is represented as follows:

[0014] In the formula, To reduce Gaussian noise, avoid excessive calls to a single expert module; This is the output vector of the gating network.

[0015] Furthermore, the embedded memory mechanism includes employing a key-value pair storage structure, with the memory state updated as follows:

[0016] In the formula, for Always remember the state. For adaptive attenuation rate, for The state of historical memory at all times , For learnable projection matrices, For the first Input characteristics at any given time; Adaptive decay rate Dynamically adjusted based on the cosine similarity between the current input and historical memory:

[0017] In the formula, Use the Sigmoid activation function; Cosine similarity is used; the higher the similarity, the greater the decay rate, which accelerates the updating of similar information; the lower the similarity, the smaller the decay rate, which preserves differentiated information. for The state of historical memory at all times Historical state vector The transpose of .

[0018] Furthermore, the AdamW optimizer is used to train the teacher model, and a cosine annealing learning rate decay strategy is employed during training. The learning rate is expressed as:

[0019] In the formula, Let e ​​be the learning rate in round e. The initial learning rate is T, and the total number of fine-tuning rounds is T. The loss function used is cross-entropy loss:

[0020] In the formula, The total number of samples, For the true label of sample i, Output the probability that sample i belongs to the attack category for the model; The teacher model is trained iteratively for T rounds. After each round, the accuracy is evaluated on the validation set, and the weights of the teacher model with the best performance are saved.

[0021] Furthermore, the student model must be constructed with a hybrid expert architecture that is functionally aligned with the teacher model, ensuring that the three types of specialized expert modules correspond one-to-one. The QLoRA low-rank adaptation technique is used to fine-tune the core parameters of the student model, including dynamic range calibration, low-rank adaptation matrix insertion, and quantization-aware training. Dynamic range calibration includes statistically analyzing the distribution of activation values ​​in each trainable layer of the student model from the training set samples, and calculating the quantization scaling factor.

[0022] In the formula, This is the quantization scaling factor. For a single activation value of a certain layer in the student model, The maximum absolute value of the activation values ​​for this layer. This is for quantization bit width; Low-rank adaptation matrix insertion includes inserting the original weight matrix in the core layer. Side insertion of low-rank matrix pairs and Then the weight matrix is ​​updated as follows:

[0023] Low-rank decomposition reduces the computational complexity from O( ). The value decreases to O(2dr); Weight matrix Dimension size, The rank is the approximation of the lower rank. Quantization-aware training includes embedding quantization-dequantization operations, with the quantization representation as follows:

[0024] Inverse quantization is represented as:

[0025] In the formula, where, The quantized integer activation value. This is the quantization scaling factor. This is the floating-point activation value restored after dequantization; This is the floating-point activation value before quantization.

[0026] Furthermore, the student model is trained using knowledge distillation, including: First, construct the distillation loss function, which is a joint loss function using KL divergence soft-label loss and cross-entropy hard-label loss, expressed as:

[0027] In the formula, is the distillation loss function; b is the soft label loss weight, ensuring that the teacher's model knowledge plays a dominant role in transmission. For KL divergence soft label loss, The cross-entropy hard-label loss is used. Then, soft labels for the teacher model are generated. The training set samples are input into the saved optimal teacher model, and the soft label probabilities are calculated using the softmax function.

[0028] In the formula, The output of the teacher model for category c is the logits. For temperature parameters; Finally, the student model is trained using distillation. The teacher model parameters are fixed, and the training set samples along with their corresponding soft and hard labels are input into the quantized student model. The AdamW optimizer is used to minimize the total distillation loss. Train a student model for the target; iterate the training for several rounds, evaluate the accuracy of the student model on the validation set after each round, and save the weight of the student model with the highest accuracy on the validation set to avoid overfitting.

[0029] Furthermore, by combining timing characteristics, protocol characteristics, and historical context, the time location and protocol level of abnormal messages can be determined.

[0030] The timing localization of abnormal messages includes calculating the precise timestamp of the abnormal message based on the timestamp difference and feature matching score:

[0031] In the formula, For the precise timestamp of the abnormal message, Initial timestamp for attack detection. K represents the message transmission period, and K represents the backtracking window size. For the first Feature vectors of historical messages For abnormal feature vectors, The feature matching score is calculated using weighted cosine similarity.

[0032] In the formula, The weight coefficients for the core features For the first The first historical message in The values ​​on the core features of the class, For the abnormal feature vector at the th The values ​​on the core features of the class, ; Protocol-level location of abnormal messages includes calculating location confidence:

[0033] In the formula, To determine location reliability, These are the weighting coefficients; The degree of matching between the abnormal message and the standard protocol fields is obtained through the abnormal message location module of the teacher model; For anomaly features and historical context Similarity; For the protocol field of the exception message, This is a specification field for standard protocols.

[0034] Furthermore, the abnormal message location uses a binary search backtracking method to optimize search efficiency, and the location window is updated as follows:

[0035] In the formula, The upper and lower bounds of the current positioning window are defined, and d is the middle value of the window. Using the confidence threshold, the location range is quickly narrowed down using a binary search method.

[0036] The beneficial effects of this invention are as follows: (1) This invention constructs a teacher model of the MoE architecture containing three types of specialized expert modules: Ethernet DoS attack detection, CAN bus Fuzzy attack detection and abnormal message location. This model can accurately identify different attack characteristics, improve the detection targeting, and overcome the shortcomings of traditional single models with weak targeting.

[0037] (2) This invention replaces the traditional sliding window with an embedded memory mechanism, dynamically updates the historical state based on the adaptive decay rate, avoids repeated calculation of data within the window, reduces computational redundancy, significantly reduces detection delay, and improves real-time performance, thus meeting the real-time response requirements of the vehicle network.

[0038] (3) The present invention adopts a QLoRA quantization and knowledge distillation collaborative training strategy to transfer the soft knowledge of the teacher model to the student model with a significantly reduced number of parameters. While maintaining the core detection capabilities, it adapts to the resource constraints of vehicle-mounted embedded devices and achieves a balance between lightweight and detection accuracy.

[0039] (4) Combining message timing features, protocol features and historical context stored by memory mechanism, weighted cosine similarity matching and binary backtracking optimization are adopted, so that the present invention can accurately lock the timestamp and protocol level of abnormal messages, solving the problem of large positioning error in traditional methods.

[0040] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a vehicle-mounted heterogeneous network security detection method according to an embodiment of the present invention; Figure 2 A flowchart illustrating the process of text-based processing of multi-source heterogeneous data; Figure 3 A schematic diagram illustrating the principle of QLoRA quantization fine-tuning; Figure 4 A schematic diagram of the framework for knowledge distillation training; Figure 5 A simplified diagram of knowledge distillation; Figure 6 This is a schematic diagram of the MoE expert model structure. Detailed Implementation

[0042] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0043] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0044] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0045] Please see Figure 1 One embodiment of this invention provides a method for detecting heterogeneous network security in vehicles based on a memory mechanism and the MoE architecture. The core of this method lies in constructing a MoE architecture teacher model by fine-tuning a Google base model. This teacher model includes three specialized detection modules: Ethernet DoS attack detection, CAN bus fuzzy attack detection, and abnormal message localization. Then, by combining QLoRA low-rank quantization fine-tuning and knowledge distillation, the three specialized detection capabilities of the teacher model are completely transferred, creating a lightweight student model with over 90% fewer parameters while still retaining the core detection and localization capabilities of the teacher model. The student model adopts the same MoE architecture as the teacher model, reuses the same abnormal message localization module, and can accurately locate abnormal messages by linking historical context information. Simultaneously, it independently performs specialized detection of Ethernet DoS attacks and CAN bus fuzzy attacks, ultimately collaborating with the teacher model to achieve accurate identification and efficient response to anomalies in heterogeneous vehicle networks.

[0046] This embodiment uses publicly available datasets in the field of automotive network security (Automotive Ethernet dataset, CANDataset for intrusion detection) as the data source. These datasets cover CAN bus data (including normal messages and fuzzy attack messages) and Ethernet data (including normal traffic and DoS attack traffic). The two types of data follow the CAN2.0B protocol and the IEEE 802.3 Ethernet protocol, respectively, and have heterogeneous characteristics in frame structure, transmission rate, and data format.

[0047] The method provided in this embodiment is as follows: 1. Data preprocessing for vehicle-mounted heterogeneous networks like Figure 2 As shown, the heterogeneous raw data from CAN bus and Ethernet is converted into a structured feature dataset that the model can input, eliminating data format differences and extracting core attack features.

[0048] (1) Data loading and field parsing: The dataset file is read using the pandas library. For CAN data, the frame ID (hexadecimal), data field, timestamp (milliseconds), and data length code (DLC) fields are parsed. For Ethernet data, the source IP, destination IP, transport layer port, packet size, and application layer protocol (SOME / IP / DoIP) fields are parsed.

[0049] (2) Data cleaning and anomaly filtering: Invalid samples are removed, including those with empty timestamps, CAN frame IDs outside the 0x000-0x7FF range, Ethernet packet sizes less than 64 bytes or greater than 1500 bytes, and attack types that are not defined (not "normal", "dos", or "fuzzy").

[0050] Missing value handling: missing values ​​in the CAN data field are filled with 0x00; abnormal IPs in Ethernet data (non-vehicle private network segments 192.168.1.x / 24, 10.0.0.x / 24) are directly filtered.

[0051] (3) Core feature extraction: CAN data characteristics include: Frame ID occurrence frequency, the number of times the same frame ID appears per unit time; The period jitter value is calculated using the following formula: ); The entropy value of the data domain is calculated as follows: ,in Let k be the probability of occurrence of the byte value k. It is a byte value, which is a specific 8-bit binary value in the data field.

[0052] Ethernet data characteristics include: Source IP access frequency, the number of times the same source IP is accessed per unit of time; Number of port connection failures; The coefficient of variation of message length is calculated as follows: ; Session establishment time is the time difference between the TCP SYN packet and the ACK packet.

[0053] (4) Label mapping and feature concatenation: Construct unified binary labels: "normal" → 0, "dos" (Ethernet attack), "fuzzy" (CAN attack) → 1. Concatenate the data according to the format "protocol type + timestamp + core characteristics". For example, CAN data is concatenated as "CAN,1580000000,0x123,8,0.85,0.32", and Ethernet data is concatenated as "Ethernet,1580000000,192.168.1.100,8080,1200,0.25".

[0054] (5) Format conversion and dataset partitioning: The teacher model and student model's word segmenters are used to convert the feature text into token ID sequences. The maximum sequence length is set to 64. Padding is applied to sequences that are too short, and sequences that exceed the length are truncated. The dataset is divided into training set, validation set, and test set using stratified sampling in a 7:2:1 ratio to ensure that the "normal" and "attack" labels of the three datasets are distributed in a consistent manner and to avoid class imbalance.

[0055] 2. Construction and Training of a Teacher Model Based on MoE Architecture Integrating Memory Mechanisms like Figure 6 As shown, a specialized detection model for heterogeneous network attacks is constructed. The detection targeting is improved through the MoE architecture, and the memory mechanism is embedded to solve the redundancy and latency problems of the sliding window.

[0056] (1) Teacher model initialization: We fine-tuned the Google-BERT pre-trained model (12 Transformer coding layers, 12-head self-attention mechanism, 1.7 G parameters) to retain its strong feature extraction capabilities.

[0057] (2) MoE architecture setup: The core fusion output logic for constructing a gating network and three types of specialized expert modules is as follows:

[0058] in, The output results of the MoE architecture fusion The weights of the i-th type of expert module output by the gating network are: These are the preprocessed structured features. This is the output of the Ethernet DoS attack detection module. This is the output of the CAN bus Fuzzy attack detection module. Output for the abnormal message location module.

[0059] Gated network design: A two-layer fully connected network is used to output the weights of each expert module. Gaussian noise is introduced into the weight calculation. The output of the gated network is represented as follows:

[0060] in, This is a Gaussian noise term. This avoids excessive calls to a single expert module. This is the output vector of the gating network.

[0061] The targeted design of the three types of specialized expert modules: The Ethernet DoS attack detection module focuses on capturing timing characteristics such as sudden changes in traffic and abnormal connection requests.

[0062] The CAN bus Fuzzy attack detection module is designed based on a self-attention mechanism, which focuses on protocol features such as frame ID validity and data field consistency through attention weights.

[0063] The abnormal message location module is designed based on a self-attention mechanism and outputs the time confidence and protocol matching degree of the abnormal message.

[0064] (3) Embedded memory mechanisms: Using a key-value pair storage structure, the memory state is updated as follows:

[0065] in, for Always remember the state. For adaptive attenuation rate, for The state of historical memory at all times , The learnable projection matrix has dimensions of 768×512. for The input feature vector at time step 1. Adaptive decay rate. Dynamically adjusted based on the cosine similarity between the current input and historical memory:

[0066] in, It is the Sigmoid activation function. Cosine similarity is used; the higher the similarity, the greater the decay rate, which accelerates the updating of similar information; the lower the similarity, the smaller the decay rate, which preserves the differentiated information. for Transpose of the time-history memory state vector.

[0067] (4) Teacher model training: The AdamW optimizer was used with an initial learning rate of 2×10⁻⁶. -5 A cosine annealing learning rate decay strategy is adopted, and the learning rate is expressed as:

[0068] in, For the first Round learning rate, The initial learning rate is T=18, and the total number of fine-tuning rounds is T=18.

[0069] The loss function used is cross-entropy loss:

[0070] in, The total number of samples, For the true label of sample i, Output the probability that sample i belongs to the attack category for the model.

[0071] The teacher model is trained iteratively for 18 rounds. After each round, the accuracy is evaluated on the validation set, and the weights of the model with the best performance are saved.

[0072] 3. QLoRA fine-tuning and distillation-coordinated student model optimization like Figure 3 As shown, a lightweight student model is constructed, and through the collaborative optimization of QLoRA quantization and knowledge distillation, high-precision detection capability is retained while reducing the model size.

[0073] (1) Student model initialization: The tiansz-bert pre-trained model (6 Transformer encoding layers, 6-head self-attention mechanism, 400M parameters) was selected to construct the MoE architecture that is aligned with the teacher model's functionality, ensuring that the specialized detection modules correspond one-to-one.

[0074] (2) QLoRA quantization: ① Dynamic range calibration: Statistically analyze the activation value distribution of training set samples in each trainable layer of the student model, and calculate the quantization scaling factor.

[0075] in, This is the quantization scaling factor. This represents a single activation value of a layer in the model. The maximum absolute value of the activation values ​​for this layer. This is for quantization bit width.

[0076] ②Low-rank adaptation matrix insertion: the original weight matrix in the self-attention layer Side insertion of low-rank matrix pairs and Then the weight matrix is ​​updated as follows:

[0077] Low-rank decomposition reduces the computational complexity from O( ). ) decreased to O(2dr). Weight matrix Dimension size, It is the rank of the low-rank approximation.

[0078] ③ Quantization-Aware Training (QAT): Embedding quantization-dequantization operations, quantization is represented as:

[0079] Inverse quantization is represented as:

[0080] in, The quantized integer activation value. This is the quantization scaling factor. This is the floating-point activation value restored after dequantization. This is the floating-point activation value before quantization.

[0081] Dynamic range calibration calculates the appropriate quantization scaling factor by statistically analyzing the activation value distribution of each trainable layer in the student model, providing a precise numerical mapping benchmark for subsequent low-precision quantization. This lays the foundation for lightweighting the model while ensuring quantization accuracy. Low-rank adaptation matrix insertion reduces computational complexity from O(d²) to O(2dr) by inserting low-rank matrix pairs next to the original weight matrices of core layers such as the self-attention layer. With a parameter reduction of over 90%, only minor adjustments to the low-rank matrix are needed to fully preserve the core detection and localization capabilities of the teacher model, significantly reducing the storage and inference overhead of the student model. Quantization-aware training simulates quantization errors during training by embedding quantization-dequantization operations, allowing the model to actively adapt. This enables the student model to maintain detection performance comparable to the original floating-point model even when deployed in low-precision environments such as INT8. These three elements work together to achieve the core goals of the student model: "extremely lightweight, high detection accuracy, and deployable on vehicle-mounted devices."

[0082] (3) Knowledge distillation training like Figure 4and Figure 5 As shown, by building a knowledge distillation framework, the high-quality knowledge of the teacher model is transferred to the student model, which is "quantified first and then the MoE expert model is built", which makes up for the accuracy loss caused by quantization and improves the accuracy of intrusion detection.

[0083] ① Construction of distillation loss function: A joint loss function of "soft label loss + hard label loss" is adopted to achieve effective transfer of teacher model knowledge.

[0084] The KL divergence soft-label loss (distillation loss) measures the difference in probability distribution between the teacher's model and the student's model, expressed as:

[0085] in, For the training set, Samples output for the teacher model Category The probability of soft tags, The soft label probability output by the student model. This is a temperature parameter (used to smooth the probability distribution and enhance the transfer of fine-grained knowledge in the teacher model). Two categories are available. Indicates normal category, Indicates the attack category.

[0086] Cross-entropy hard-label loss (supervised loss) optimizes the classification accuracy of the student model based on the true labels:

[0087] in, For the sample The true label, Samples output for the student model The probability of belonging to an attack category.

[0088] The total distillation loss is a weighted fusion of the soft label loss and the hard label loss, as shown in the following formula:

[0089] Where b=0.7 is the soft label loss weight, which ensures that the teacher's model knowledge plays a dominant role in transmission, while retaining the supervisory role of the real labels; ② Teacher Model Soft Label Generation: Input the training set samples into the saved optimal teacher model, and calculate the soft label probability using the softmax function:

[0090] in, The output of the teacher model for category c is the logits. =5.0 is the temperature parameter.

[0091] ③ Distillation training execution: Fix the teacher model parameters (do not participate in updates), input the training set samples and corresponding soft and hard labels into the saved quantized student model; use the AdamW optimizer (learning rate set to 1e-4, weight decay coefficient of 0.05) to minimize the total distillation loss. Training is performed for the target; the training is iterated for 6 rounds, and the accuracy of the student model is evaluated on the validation set after each round. The weight of the student model with the highest accuracy on the validation set is saved to avoid overfitting.

[0092] ④ Model performance evaluation and saving: The performance of the student model is evaluated before and after distillation. The optimization effect is compared by comparing accuracy, precision, recall and F1 score, and the student model after iterative optimization is saved.

[0093] The performance metrics calculation method includes: inputting the divided test set samples into the student models of "QLoRA quantization only (before distillation)" and "QLoRA quantization + knowledge distillation (after distillation)" respectively, and calculating four types of performance metrics.

[0094] Accuracy: Measures the overall correctness of the model's classification.

[0095]

[0096] in, To correctly identify the number of attack samples, To ensure the correct number of normal samples are identified, The number of normal samples misclassified as attacks. The number of attack samples misclassified as normal; Precision: Measures the accuracy with which a model identifies attack samples.

[0097]

[0098] Recall: Measures a model's ability to cover attack samples.

[0099]

[0100] F1 score: A metric that balances precision and recall.

[0101]

[0102] 4. Abnormal message location By combining timing characteristics, protocol characteristics, and historical context, the system can accurately locate the time position and protocol level of abnormal messages.

[0103] In this method, CAN bus messages use frame IDs as the core protocol identifier, while Ethernet messages use IP addresses. This approach enables precise location of abnormal messages based on both time and protocol level, and optimizes location efficiency through a binary search backtracking method.

[0104] In the time dimension, based on temporal characteristics and historical context, candidate time intervals for abnormal messages are initially identified. Then, through binary backtracking and using location confidence as the judgment criterion, the location window is rapidly narrowed down iteratively, ultimately accurately pinpointing the timestamp of the abnormal message.

[0105] At the protocol level, frame ID matching is used for CAN bus messages, and IP address compliance is used for Ethernet messages. Combining protocol characteristics with historical context-based protocol behavior benchmarks, coarse protocol-level localization of abnormal messages is achieved. This method significantly improves localization efficiency while maintaining accuracy, ultimately outputting a set of abnormal CAN message IDs and a set of abnormal Ethernet message source IPs, achieving comprehensive, efficient, and accurate localization of abnormal messages in heterogeneous automotive networks.

[0106] (1) Time-series localization: Calculate the precise timestamp of the abnormal message based on the timestamp difference and feature matching score:

[0107] in, For the precise timestamp of the abnormal message, Initial timestamp for attack detection. For message transmission cycle (CAN bus) =10 ms, Ethernet =1 ms), K=50 is the backtracking window size. For the first Feature vectors of historical messages For abnormal feature vectors, This is the feature matching score. The feature matching score is calculated using weighted cosine similarity:

[0108] in, The weight coefficients are the core features, which include four categories: time-series features, data domain features, protocol features, and statistical features. For the first The first historical message in The values ​​on the core features of the class, For the abnormal feature vector at the th The values ​​on the core features of the class, .

[0109] (2) Protocol-level positioning: Calculate positioning reliability:

[0110] in, For location reliability (value range [0,1]), For the weighting coefficient, , The degree of matching between the abnormal message and the standard protocol fields. For anomaly features and historical context The similarity. For the protocol field of the exception message, This is a specification field for standard protocols.

[0111] (3) The binary search method is used to backtrack and optimize the positioning efficiency. The positioning window update is represented as follows:

[0112] in, The upper and lower bounds of the current positioning window are defined, and d is the middle value of the window. Using the confidence threshold, the location range is quickly narrowed down using a binary search method.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for detecting heterogeneous network security in vehicles based on memory mechanisms and MoE architecture, characterized in that, The heterogeneous raw data from CAN bus and Ethernet is acquired, preprocessed and converted into a structured feature dataset suitable for the model, and divided into training set, validation set and test set; A teacher model based on the MoE architecture with an integrated memory mechanism is constructed. The teacher model is fine-tuned based on a pre-trained language model. A hybrid expert architecture is built and three types of specialized expert modules are configured: an Ethernet DoS attack detection module, a CAN bus fuzzy attack detection module, and an abnormal message localization module. At the same time, a memory mechanism is embedded to replace the sliding window. The teacher model is trained using a prepared feature dataset. A lightweight MoE architecture student model is constructed that is functionally aligned with the teacher model. The student model fine-tunes its core parameters by using QLoRA low-rank adaptation on a pre-trained language model, and then transfers the soft knowledge of the teacher model through distillation technology to complete the training and optimization of the student model. After the teacher model and student model are trained, for actual vehicle-mounted heterogeneous network packets, by extracting features, linking the output results of the abnormal packet localization module in the teacher model with the historical packet context stored in the memory mechanism, the student model can realize the localization of abnormal packets in time series and protocol layer.

2. The method according to claim 1, characterized in that, Preprocessing of heterogeneous network data in vehicles includes: Parse key fields of CAN and Ethernet data, remove invalid samples, fill missing values ​​in CAN data fields, and filter abnormal IPs in Ethernet data. Features of CAN data and Ethernet data are extracted separately. CAN data features include frame ID occurrence frequency, period jitter value, and data field entropy value. Ethernet data features include source IP access frequency, number of port connection failures, message length variation coefficient, and session establishment time. The tags are uniformly mapped to a binary classification format and concatenated according to the format of "protocol type + timestamp + core feature"; The feature text is converted into a token ID sequence, and a stratified sampling method is used to divide it into training set, validation set and test set.

3. The method according to claim 1, characterized in that, The teacher model is fine-tuned based on a pre-trained language model. The hybrid expert architecture includes a gating network and three types of specialized expert modules. The fusion output of the hybrid expert architecture is represented as follows: In the formula, To integrate the output results of the hybrid expert architecture, The weights of the i-th type of expert module output by the gating network are: These are the preprocessed structured features; This is the output of the Ethernet DoS attack detection module. This is the output of the CAN bus Fuzzy attack detection module. Output to the abnormal message location module; The gating network uses a two-layer fully connected network to output the weights of each expert module. The output of the gating network is represented as follows: In the formula, To reduce Gaussian noise, avoid excessive calls to a single expert module; This is the output vector of the gating network.

4. The method according to claim 3, characterized in that, The embedded memory mechanism includes using a key-value pair storage structure, with the memory state updated as follows: In the formula, for Always remember the state. For adaptive attenuation rate, for The state of historical memory at all times , For learnable projection matrix For the first Input characteristics at any given time; Adaptive decay rate Dynamically adjusted based on the cosine similarity between the current input and historical memory: In the formula, Use the Sigmoid activation function; Cosine similarity is used; the higher the similarity, the greater the decay rate, which accelerates the updating of similar information; the lower the similarity, the smaller the decay rate, which preserves differentiated information. for The state of historical memory at all times State vector The transpose of .

5. The method according to claim 3, characterized in that, The teacher model was trained using the AdamW optimizer, with a cosine annealing learning rate decay strategy employed during training. The learning rate was expressed as: In the formula, For the first Round learning rate, The initial learning rate is T, and the total number of fine-tuning rounds is T. The loss function used is cross-entropy loss: In the formula, The total number of samples, For the true label of sample i, Output the probability that sample i belongs to the attack category for the model; The teacher model is trained iteratively for T rounds. After each round, the accuracy is evaluated on the validation set, and the weights of the teacher model with the best performance are saved.

6. The method according to claim 1, characterized in that, The student model must be constructed with a hybrid expert architecture that is functionally aligned with the teacher model, ensuring that the three types of specialized expert modules correspond one-to-one. The QLoRA low-rank adaptation technique is used to fine-tune the core parameters of the student model, including dynamic range calibration, low-rank adaptation matrix insertion, and quantization-aware training. Dynamic range calibration includes statistically analyzing the distribution of activation values ​​in each trainable layer of the student model from the training set samples, and calculating the quantization scaling factor. In the formula, This is the quantization scaling factor. For a single activation value of a certain layer in the student model, The maximum absolute value of the activation values ​​for this layer. This is for quantization bit width; Low-rank adaptation matrix insertion includes inserting the original weight matrix in the core layer. Side insertion of low-rank matrix pairs and Then the weight matrix is ​​updated as follows: Low-rank decomposition reduces the computational complexity from O( ). The value decreases to O(2dr); Weight matrix Dimension size, The rank is the approximation of the lower rank. Quantization-aware training includes embedding quantization-dequantization operations, with the quantization representation as follows: Inverse quantization is represented as: In the formula, where, The quantized integer activation value. This is the quantization scaling factor. This is the floating-point activation value restored after dequantization; This is the floating-point activation value before quantization.

7. The method according to claim 6, characterized in that, Knowledge distillation training is performed on the student model, including: First, construct the distillation loss function, which is a joint loss function using KL divergence soft-label loss and cross-entropy hard-label loss, expressed as: In the formula, is the distillation loss function; b is the soft label loss weight, ensuring that the teacher's model knowledge plays a dominant role in transmission. For KL divergence soft label loss, The cross-entropy hard-label loss is used. Then, soft labels for the teacher model are generated. The training set samples are input into the saved optimal teacher model, and the soft label probabilities are calculated using the softmax function. In the formula, The output of the teacher model for category c is the logits. For temperature parameters; Finally, the student model is trained using distillation. The teacher model parameters are fixed, and the training set samples along with their corresponding soft and hard labels are input into the quantized student model. The AdamW optimizer is used to minimize the total distillation loss. Train a student model for the target; iterate the training for several rounds, evaluate the accuracy of the student model on the validation set after each round, and save the weight of the student model with the highest accuracy on the validation set to avoid overfitting.

8. The method according to claim 1, characterized in that, By combining timing characteristics, protocol characteristics, and historical context, the time location and protocol level of abnormal messages can be determined.

9. The method according to claim 8, characterized in that, The timing localization of abnormal messages includes calculating the precise timestamp of the abnormal message based on the timestamp difference and feature matching score: In the formula, For the precise timestamp of the abnormal message, Initial timestamp for attack detection. K represents the message transmission period, and K represents the backtracking window size. For the first Feature vectors of historical messages For abnormal feature vectors, Feature matching score; Feature matching scores are calculated using weighted cosine similarity. In the formula, The weight coefficients for the core features For the first The first historical message in The values ​​on the core features of the class, For the abnormal feature vector at the th The values ​​on the core features of the class, ; Protocol-level location of abnormal messages includes calculating location confidence: In the formula, To determine location reliability, These are the weighting coefficients; The degree of matching between the abnormal message and the standard protocol fields is obtained through the abnormal message location module of the teacher model; For anomaly features and historical context Similarity; For the protocol field of the exception message, This is a specification field for standard protocols.

10. The method according to claim 9, characterized in that, The abnormal message location uses a binary search backtracking method to optimize search efficiency, and the location window is updated as follows: In the formula, The upper and lower bounds of the current positioning window are defined, and d is the middle value of the window. Using the confidence threshold, the location range is quickly narrowed down using a binary search method.

Images