Privacy protection federated learning method and system based on transformer and knowledge distillation

Abstract

This invention discloses a privacy-preserving federated learning method and system based on Transformer and knowledge distillation, belonging to the fields of artificial intelligence and cybersecurity technology. It includes using the attention mechanism of the Transformer model as a core component for local feature extraction to capture long-distance dependencies in data and improve feature representation quality; introducing the Paillier encryption protocol to achieve homomorphic encrypted transmission of model weights; and employing knowledge distillation technology on the central server to extract soft knowledge from the aggregated global model as a teacher model, feeding it back to the client to compress and optimize the model. This invention optimizes detection performance under Non-IID data by using Transformer as a local feature extractor, combining it with the Paillier encryption protocol, and employing knowledge distillation technology after aggregation on the central server, thus achieving a balance between data security and detection efficiency.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and cybersecurity technology, specifically relating to a privacy-preserving federated learning method and system based on Transformer and knowledge distillation. Background Technology

[0002] With the rise of IIoT (Industrial Internet of Things) and distributed cloud platforms, the massive amounts of data generated by edge devices (such as sensors and cloud nodes) have become core assets, but at the same time, they face serious privacy risks. Traditional centralized machine learning requires uploading all data to a central server for training, which not only increases security risks during data transmission but may also violate international privacy regulations. Especially in sensitive industries (such as chemical and power industries), data breaches could lead to significant economic losses or safety incidents.

[0003] To address these challenges, Federated Learning has emerged as a distributed learning paradigm. It allows clients to train models locally, sharing only model updates (such as weights) rather than the original data, thus protecting data privacy to some extent. This technology aims to solve privacy issues on mobile devices and is gradually expanding into industrial applications. However, in practical applications, Federated Learning still faces challenges such as performance degradation due to Non-IID (non-independent and identically distributed) data distribution, excessive computational overhead on resource-constrained edge devices, and the high accuracy requirements of intrusion detection tasks. Furthermore, advanced mechanisms such as attention mechanisms (e.g., Transformer models, which excel in natural language processing and computer vision, capturing long-range dependencies) and knowledge distillation techniques (used for model compression and generalization improvement) have not been fully integrated into the Federated Learning framework, making it difficult to balance privacy protection and detection performance.

[0004] Existing federated learning frameworks incorporate homomorphic encryption protocols, such as Paillier (a probabilistic public-key cryptography algorithm that supports homomorphic operations and is commonly used in privacy-preserving scenarios) or similar encryption mechanisms (e.g., FedAvg variants combined with homomorphic encryption). The core of these schemes is that the client trains the model locally (e.g., using CNNs or RNNs to extract features), then protects the model updates during transmission to a central server for aggregation via encryption protocols, and finally distributes the global model back to the client. This technology has been applied in some industrial IoT intrusion detection systems to address data privacy issues while supporting distributed training.

[0005] However, existing federated learning frameworks have significant shortcomings in terms of data privacy protection, model performance optimization, and applicability to resource-constrained devices. Specifically, existing technologies or scenarios suffer from the following key issues:

[0006] Low computational efficiency: Although homomorphic encryption supports operations in the encrypted state, it introduces high computational overhead, especially on resource-constrained edge devices, resulting in slow training iterations and failing to meet the requirements of real-time intrusion detection.

[0007] The generalization problem under non-IID data has not been effectively solved: in scenarios with uneven data distribution, the model performance deteriorates significantly, and the intrusion detection accuracy is difficult to consistently exceed 95%. In particular, in sensitive fields such as chemical and power industries, the weak generalization ability leads to an increase in false positives or false negatives.

[0008] Insufficient integration of mechanisms: Existing solutions mostly rely on traditional feature extractors (such as CNNs) and do not make full use of the attention mechanism of Transformer to improve feature quality; at the same time, there is a lack of knowledge distillation techniques to compress the model and improve the generalization of small models, resulting in the problem of insufficient computing power of edge devices, poor overall system robustness, and inability to achieve a balance between privacy protection and detection performance. Summary of the Invention

[0009] To address the problems mentioned in the background, this invention proposes a privacy-preserving federated learning method and system based on Transformer and knowledge distillation. By using Transformer as a local feature extractor, combined with Paillier encryption protocol, and a central server for aggregation, knowledge distillation technology is employed to optimize detection performance under Non-IID data, achieving a balance between data security and detection efficiency.

[0010] Technical Solution: To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0011] A privacy-preserving federated learning method based on Transformer and knowledge distillation includes using the attention mechanism of the Transformer model as a core component for local feature extraction to capture long-distance dependencies in data and improve the quality of feature representation; introducing the Paillier encryption protocol to achieve homomorphic encrypted transmission of model weights; and employing knowledge distillation technology on the central server to extract soft knowledge from the aggregated global model as a teacher model, feeding it back to the client to compress and optimize the model. The method includes the following steps:

[0012] S1. Data Acquisition and Preprocessing: Acquire data from local sensors and preprocess it into a sequence format;

[0013] S2. Feature Extraction: Deploy the Transformer model as a local feature extractor on each client device to perform feature extraction;

[0014] S3. Local Model Training: Train the intrusion detection model based on extracted features and optimize the loss function;

[0015] S31. Input the output of the self-attention mechanism and the original input, and perform residual connection on them;

[0016] S32. Perform residual connection on the input of S31 and normalize the residual results;

[0017] S33. Add a feedforward network to the normalized result of S32;

[0018] S34. Perform residual connection and layer normalization on the output of the second-level linear transformation in S33 and the normalization result in S32;

[0019] S35, Output high-dimensional feature representation;

[0020] S4. Encrypted transmission: After the client completes local training, it encrypts the model weights using the Paillier encryption protocol and then transmits the weights to the central server through a secure channel.

[0021] S5. Central Server Aggregation: After receiving the encrypted weights from multiple clients, the central server performs an aggregation operation to generate a global model.

[0022] S6. Knowledge Distillation Feedback: The central server decrypts the aggregated global model as the teacher model, then applies knowledge distillation technology to extract knowledge, and encrypts the extracted knowledge to feed back to each client to update the local model.

[0023] The distillation process includes generating soft labels: the teacher model calculates the softmax output on a shared pseudo-dataset, and the softening probability distribution of the temperature parameter t is as follows:

[0024] ,

[0025] in, represents the softening probability of the i-th category of the soft label; t represents the temperature parameter; This represents the score of the j-th category in the logits vector, where j represents the category index in the summation;

[0026] Then, the student model minimizes the difference with the teacher's soft labels using KL divergence loss, specifically:

[0027] ,

[0028] Where LOSS represents the total loss function; CE represents the cross-entropy; KL represents the divergence; and α represents the balance factor. Indicates the true label; This represents the predicted probability distribution of the student model; The soft-label probability distribution represents the teacher model;

[0029] S7. Iterative optimization: Perform closed-loop iterations of the above steps until the global model converges.

[0030] As a preferred option, the specific implementation process in S2 is as follows:

[0031] The encoder layer of the Transformer model calculates the dependency weights between input sequences using a multi-head self-attention mechanism. Specifically, it performs dot product attention calculations using query, key, and value matrices.

[0032] ,

[0033] Where softmax() represents the normalization function; Q represents the query matrix, and K represents the key matrix. The key dimension is represented by T, which represents the transpose of the matrix.

[0034] As a preferred option, the specific implementation process in S4 is as follows:

[0035] The model weights are encrypted using the Paillier encryption protocol. The specific formula for calculating the ciphertext c is as follows:

[0036] ,

[0037] Where r represents a random number; Indicates plaintext message; n represents the modulus; g represents the cardinality; The base g is raised to the power of m, used to encode the plaintext message m; This represents a random number r raised to the power of n, used to introduce randomness.

[0038] As a preferred option, the specific implementation process in S5 is as follows:

[0039] Leveraging Paillier's homomorphic properties, a weighted average is applied to the encryption weights. The specific calculation formula is as follows:

[0040] ,

[0041] in, This represents the client-side weighting coefficient; Represents the global model weights; This represents the local model weights of the i-th client; Indicates the encryption function;

[0042] Then, initialize the global model, iteratively accumulate the encryption contribution, and finally output the encrypted global model.

[0043] As a preferred approach, the global model is initialized, the encryption contribution is iteratively accumulated, and the final output of the encrypted global model is as follows:

[0044] Step 1: Initialize the global model: global model weights Initialize to a zero vector;

[0045] Step 2, Iteratively accumulate encryption contribution: For each client, the server calculates the client's encryption contribution, then adds noise to the encryption contribution, and finally accumulates it into the global model;

[0046] Step 3: Output the encrypted global model.

[0047] A privacy-preserving federated learning system based on Transformer and knowledge distillation, implementing the privacy-preserving federated learning method based on Transformer and knowledge distillation described in any of the above items, including a client and a central server;

[0048] The client includes local sensors, a feature extractor, a local trainer, and an encrypted transmitter;

[0049] Raw data is collected from local sensors and preprocessed into sequence format; features are extracted from the preprocessed data using a feature extractor; an intrusion detection model is trained based on the extracted high-dimensional features using a local trainer, and the loss function is optimized; finally, the trained model weights are homomorphically encrypted using the Paillier encryption protocol using an encrypted transmitter, and then transmitted to the central server through a secure channel.

[0050] The central server includes an aggregator, a knowledge distiller, and a feedback distributor;

[0051] Encrypted weights are received from multiple clients. An aggregator is used to perform a weighted average aggregation of the encrypted weights, leveraging the homomorphic properties of Paillier, to generate an encrypted global model. A knowledge distiller is used to decrypt the aggregated global model as a teacher model. Knowledge distillation is applied to shared pseudo-datasets or anonymous samples to generate soft labels and intermediate representations, extracting knowledge to improve model generalization and compression. A feedback distributor is used to encrypt the extracted knowledge using the Paillier protocol and distribute it back to each client through a secure channel to update the local model and optimize performance under Non-IID data.

[0052] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0053] (1) This invention constructs a privacy-preserving federated learning framework. By using Transformer as a local feature extractor (utilizing its attention mechanism to improve feature quality), combined with the Paillier encryption protocol (supporting homomorphic operations to ensure that model weights are aggregated in an encrypted state without leaking original information), and using knowledge distillation technology after aggregation by a central server (using the global model as the teacher model to feed back to the client to improve the generalization ability of small models), the detection performance under Non-IID data is optimized, achieving a balance between data security and detection efficiency. This concept not only solves the bottleneck of existing technologies, but is also applicable to scenarios such as industrial IoT and distributed cloud platforms, and can be extended to privacy-sensitive fields such as medical care and finance, ultimately significantly reducing the risk of data leakage and improving the accuracy of intrusion detection (>95%).

[0054] (2) This invention achieves a balance between privacy protection and detection performance by using Transformer as a local feature extractor, combining it with Paillier encryption protocol to transmit model weights, and then using knowledge distillation technology to feed back to each client model after aggregation on a central server. The method and system of this invention can effectively solve key problems existing in the prior art, including the performance degradation of federated learning under Non-IID data, the high computational overhead of edge devices, the low computational efficiency caused by encryption protocols, and the lack of full integration of attention mechanisms with federated learning and knowledge distillation, thus completing the closed loop of the invention logic: from the identification of the background problem to the proposal of the technical solution, and then to the verification of the actual effect. Specifically:

[0055] Addressing privacy risks and enhancing data security: Existing technologies, such as traditional centralized machine learning, require data to be uploaded to a central server, increasing transmission security risks and potentially violating privacy regulations (e.g., GDPR). Even with encryption protocols (like homomorphic encryption), computational efficiency is often low, and privacy during model updates is not fully guaranteed. This invention supports homomorphic operations through the Paillier encryption protocol, performing model aggregation in an encrypted state, avoiding decryption operations and leakage of original weights, thus significantly reducing the risk of data leakage. Experimental verification shows that the privacy leakage risk is reduced to 1.0 through differential privacy metric, ensuring the security of data in industrial control systems (such as chemical and power industries). This effect achieves a high level of privacy protection without sacrificing computational efficiency, making it suitable for sensitive industries.

[0056] Optimizing model performance and generalization ability on Non-IID data: This invention innovatively integrates the attention mechanism of Transformer for local feature extraction, which can capture long-distance dependencies in the data and improve the quality of feature representation. Combined with knowledge distillation technology, the global model is used as a teacher model to extract knowledge (soft labels or intermediate representations) and feed it back to the client, improving the generalization ability of the small model. Experimental results: On Non-IID datasets (such as the KDD Cup 99 variant), the intrusion detection accuracy reaches 96.5%, an 8% improvement compared to traditional federated learning. This effect solves the performance bottleneck on Non-IID data, improves the robustness and detection efficiency of the system (accuracy >95%), especially on resource-constrained edge devices.

[0057] Reducing computational overhead and improving applicability to edge devices: This invention optimizes small models through efficient feature extraction and knowledge distillation using Transformer, reducing the computational power required for local training; the homomorphic nature of the Paillier protocol ensures efficient aggregation without additional decryption overhead. This addresses the issue of insufficient computing power on edge devices, improving computational efficiency while maintaining high-precision intrusion detection, making it suitable for distributed environments such as industrial IoT and distributed cloud platforms.

[0058] Expanding Applicability and Enhancing Business Value: This invention framework is applicable to intrusion detection in the Industrial Internet of Things (IIoT) and distributed cloud platforms, and can be extended to other privacy-sensitive fields such as healthcare and finance. Through the fusion of multiple mechanisms (Transformer, Federated Learning, and Knowledge Distillation), the overall robustness and applicability of the system are improved. This effect completes the invention's logical loop: from addressing the core issue of balancing data privacy and efficiency to achieving practical technical effects of improved accuracy and reduced risk, thus promoting secure applications in sensitive industries (such as chemical and power sectors). Attached Figure Description

[0059] Figure 1 This is the network topology diagram of the present invention;

[0060] Figure 2 This is a data processing flowchart of the present invention;

[0061] Figure 3 This is a timing diagram of the present invention;

[0062] Figure 4 This is a performance comparison curve of the model of the present invention;

[0063] Figure 5 This is a comparison chart of the final accuracy under different degrees of data heterogeneity in this invention;

[0064] Figure 6 This is the privacy-utility trade-off analysis diagram of the present invention;

[0065] Figure 7 This is a comparison chart of the client-side local computing overhead of the present invention;

[0066] Figure 8 This is a flowchart of the Transformer local feature extraction process of the present invention;

[0067] Figure 9 This is a flowchart of the central server knowledge distillation and feedback process of the present invention;

[0068] Figure 10 This is the privacy-accuracy trade-off curve of the present invention. Detailed Implementation

[0069] The present invention will be further illustrated below with reference to specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0070] Example 1

[0071] Based on the comprehensive needs of privacy protection and performance optimization in federated learning in distributed environments (such as industrial IoT and distributed cloud platforms), this invention aims to construct an efficient and secure framework through the deep integration of multiple mechanisms. Specifically, this concept uses the attention mechanism of the Transformer model as the core component for local feature extraction to capture long-distance dependencies in data and improve the quality of feature representation. Simultaneously, the Paillier encryption protocol is introduced to achieve homomorphic encrypted transmission of model weights, ensuring that the aggregation process does not leak original information. Furthermore, knowledge distillation technology is employed on the central server to extract soft knowledge (including soft labels and intermediate representations) from the aggregated global model as the teacher model, feeding back to the client to compress the model and optimize generalization. The core of this concept lies in the synergy between mechanisms: Transformer provides high-quality local features, the encryption protocol ensures secure transmission, and knowledge distillation addresses the limitations of Non-IID data distribution and edge computing power, thereby achieving a dynamic balance between privacy protection and intrusion detection efficiency (accuracy > 95%). This concept is not a simple superposition, but rather forms a closed-loop system through iterative optimization cycles (such as repeated local training-encryption-aggregation-distillation), suitable for intrusion detection tasks in sensitive scenarios such as chemical and power industries.

[0072] The terms used in this embodiment are explained below:

[0073] LLM: Large Language Models, such as generative AI models like ChatGPT.

[0074] MoE: Mixture of Experts, an architecture that dynamically routes to multiple expert sub-models.

[0075] PU Learning: Positive-Unlabeled Learning, a semi-supervised learning method that requires only positive samples and unlabeled data.

[0076] Jailbreak Attack: A form of bypassing AI model security restrictions, including prompt injection and malicious instruction bypass.

[0077] Prompt Injection is a type of attack that bypasses model restrictions through input.

[0078] Malicious Instruction Bypass: A type of attack that bypasses security mechanisms by inputting commands.

[0079] The Transformer architecture is a deep learning model based on a self-attention mechanism for sequence processing.

[0080] Gating Network: A component in MoE that dynamically routes inputs to expert sub-models.

[0081] Experts: Expert sub-models are multiple independent sub-networks in MoE that are used to process specific input slices.

[0082] Reliable Negative Extraction: An algorithm for selecting high-confidence negative samples from unlabeled data.

[0083] AdvBench: Adversarial Benchmark, a public dataset for testing adversarial attacks.

[0084] API: Application Programming Interface, used for system integration.

[0085] CNN: Convolutional Neural Network, used for text feature extraction.

[0086] RNN: Recurrent Neural Network, used for processing sequence data.

[0087] LSTM: Long Short-Term Memory, a variant of RNN used to capture long dependencies.

[0088] GRU: Gated Recurrent Unit, a simplified variant of RNN.

[0089] Bi-RNN: Bidirectional RNN, which combines forward and backward processing to enhance contextual understanding.

[0090] GNN: Graph Neural Network, used for extracting graph-structured data.

[0091] GAT: Graph Attention Network, a variant of GNN.

[0092] GAN: Generative Adversarial Network, used for data generation and discrimination.

[0093] RL: Reinforcement Learning, which optimizes decision-making through reward signals.

[0094] BPE: Byte Pair Encoding, a method for word segmentation.

[0095] POS Tagging: Part-of-Speech Tagging, used to enhance text semantic processing.

[0096] ELMo: Embeddings from Language Models, a dynamic embedding method.

[0097] IIoT: Industrial Internet of Things.

[0098] Non-IID: Non-Independent and Identically Distributed.

[0099] GDPR: General Data Protection Regulation, a type of data privacy regulation.

[0100] Federated Learning: A distributed learning paradigm that allows clients to train locally and share model updates.

[0101] Paillier: The Paillier encryption protocol is a public-key encryption scheme that supports homomorphic operations and is used for privacy protection.

[0102] FedAvg: Federated Averaging, a model aggregation algorithm in federated learning.

[0103] KDD Cup 99: The KDD Cup 99 dataset is a benchmark dataset for intrusion detection.

[0104] CKKS: Cheon-Kim-Kim-Song, a fully homomorphic encryption scheme that supports multiplication and addition operations.

[0105] BFV: Brakerski-Fan-Vercauteren, a lattice-based homomorphic encryption scheme that supports integer arithmetic.

[0106] PHE: Partial Homomorphic Encryption, which supports specific operations such as multiplication.

[0107] DP: Differential Privacy, a mechanism for quantifying privacy protection.

[0108] MPC: Multi-Party Computation, used for joint computation with privacy protection.

[0109] KD: Knowledge Distillation, a model compression and knowledge transfer technique.

[0110] ViT: Vision Transformer, a variant of Transformer used for image processing.

[0111] BERT: Bidirectional Encoder Representations from Transformers, a pre-trained Transformer model.

[0112] TTF: Tab Transformer, a variant of Transformer for tabular data.

[0113] AE: Autoencoder, used for feature compression and data reconstruction.

[0114] ECA-Net: Efficient Channel Attention Network, a variant of CNN with enhanced attention.

[0115] ResNet: Residual Network, a type of CNN architecture.

[0116] Shamir: The Shamir secret-sharing scheme is a protocol that splits secrets among multiple parties.

[0117] ElGamal: The ElGamal encryption scheme is a public-key encryption scheme that supports multiplicative homomorphism.

[0118] Skefl: The Skefl protocol, a lightweight homomorphic encryption framework for federated learning.

[0119] Laplace: The Laplace mechanism is a mechanism for adding noise in differential privacy.

[0120] FedSplit: The FedSplit framework, a variant of federated learning that features phased dynamic aggregation.

[0121] FedSSD: Federated Selective Self-Distillation, a local self-distillation method.

[0122] Data-free KD: Data-free Knowledge Distillation, which uses synthetic data for distillation.

[0123] FedGen: Federated Generator, a generator framework for data-independent distillation.

[0124] FD: Federated Distillation, a method for knowledge transfer between clients.

[0125] AdaptiveKD: Adaptive Knowledge Distillation, which combines clustering and other mechanisms.

[0126] FedAsync: Federated Asynchronous, a variant of asynchronous federated learning.

[0127] HIPAA: Health Insurance Portability and Accountability Act, the U.S. medical privacy law.

[0128] CAN: Controller Area Network, used for vehicle communication.

[0129] The framework of this application consists of multiple client devices, a central server, and a communication module. The client devices integrate a Transformer-based local model for data processing; the central server is responsible for weight aggregation and knowledge distillation calculations; and the communication module embeds the Paillier encryption protocol to support secure transmission under homomorphic operations.

[0130] The privacy-preserving federated learning method based on Transformer and knowledge distillation provided in this embodiment has the following specific implementation steps:

[0131] S1. Data Acquisition and Preprocessing: Acquire data from local sensors and preprocess it into a sequence format;

[0132] First, the local dataset (such as traffic logs or video stream data in the Industrial Internet of Things) is preprocessed, including tokenization and positional encoding, to generate the input embedding vector.

[0133] S2. Local Feature Extraction: Deploy the Transformer model as a local feature extractor on each client device (such as an edge sensor or cloud node); for example... Figure 8 As shown.

[0134] Then, the Transformer's encoder layers (multiple stacked layers, typically 6-12 layers) compute dependency weights between input sequences through a multi-head self-attention mechanism, such as using a query, key, and value matrix for dot product attention computation:

[0135]

[0136] in, is a normalization function used to transform an input vector into a probability distribution such that the sum of all elements is 1, and each element is between 0 and 1; Q is the query matrix, and K is the key matrix. Let T be the key dimension, and T represent the transpose operation, used to interchange the rows and columns of the key matrix K (i.e., K...). T ), so that matrix multiplication (dot product) can be performed with the query matrix Q.

[0137] This mechanism captures long-distance dependencies, such as temporal patterns in intrusive data.

[0138] S3. Local Model Training: Train the intrusion detection model based on extracted features and optimize the loss function;

[0139] Add a feed-forward network and layer normalization to output a high-dimensional feature representation for training downstream intrusion detection models (such as classification heads).

[0140] In the encoder layer of the Transformer model, after the multi-head self-attention mechanism is computed, the output needs further processing to enhance the model's non-linear expressiveness and stability. Therefore, a feed-forward network (FFN) and layer normalization (LN) are added. These components are standard parts of the Transformer architecture, used to transform the intermediate representation of the self-attention output into a higher-quality, high-dimensional feature representation, facilitating its use in downstream tasks such as intrusion detection model training.

[0141] The core objective of this process is:

[0142] 1. By introducing nonlinear transformations through feedforward networks, the expressive power of features is enhanced.

[0143] 2. The vanishing / exploding gradient problem is mitigated by layer normalization and residual connections (Add & Norm), ensuring stable training.

[0144] 3. Output high-dimensional feature representations, typically with dimensions equal to the hidden layer size of the model (e.g., 512 or 768). These features capture long-distance dependencies and semantic information of the input sequence.

[0145] The specific operating steps are as follows:

[0146] S31. Input the output of the self-attention mechanism and the original input, and perform residual connection on them;

[0147] The self-attention mechanism outputs an intermediate tensor, denoted as AttentionOutput, with the shape (batch_size, sequence_length, hidden_dim), where hidden_dim is the hidden dimension (e.g., 512); batch_size represents the number of data samples; and sequence_length represents the length of a single sample.

[0148] S32. Perform residual connection on the input of S31 and normalize the residual results;

[0149] Residual connection: The self-attention output is added to the original input to preserve the information flow. The specific formula is as follows:

[0150] ;

[0151] in, This represents the residual connection result; The intermediate tensor represents the output of the self-attention mechanism; Input is the embedding vector before entering self-attention, i.e., the original input.

[0152] Layer normalization is applied to normalize the residual connection results and stabilize their distribution. The layer normalization formula is as follows:

[0153]

[0154]

[0155]

[0156] in, Indicates to Perform normalization. This represents the i-th input vector. It is the input vector (calculated along the last dimension), i.e., the result of the residual connection; d is the mean, and d is the hidden dimension. It is variance. It is a small constant (usually 1e-5) to prevent division by zero. and These are learnable parameters (initialized to 1 and 0).

[0157] The output is denoted as Norm1.

[0158] Objective: To prevent gradient problems and provide standardized inputs for feedforward networks.

[0159] S33. Add a feed-forward network to the normalized result of S32.

[0160] Input: Norm1 from the output of S32.

[0161] A feedforward network is a two-layer fully connected network (MLP) that uses an activation function (usually ReLU or GELU) to introduce nonlinearity.

[0162] The first layer of linear transformation expands the input to a higher dimension (typically 4 times the hidden dimension, such as 2048) to increase expressive power.

[0163] Activation: Apply the ReLU activation function.

[0164] The second-level linear transformation projects the dimension back to the hidden dimension.

[0165] The output is denoted as FFNOutput.

[0166] Objective: To enhance the nonlinear representation of features and capture more complex patterns.

[0167] The formula for adding a feed-forward network to the normalized result of S32 is as follows:

[0168]

[0169] in, This represents the output of the feedforward network; Norm1 is the input (from the normalization result of the first layer). It is the first layer weight matrix (shape: hidden_dim×intermediate_dim, for example 512×2048); It is the first-level bias term (shape: intermediate_dim); It is the ReLU activation function. It is the second layer weight matrix (shape: intermediate_dim × hidden_dim); It is the second-level bias term (shape: hidden_dim).

[0170] In this embodiment, the ReLU activation function can be replaced with GELU:

[0171] ,

[0172] in, It is the standard normal cumulative distribution function; Norm1 is the input (from the first layer of normalization results).

[0173] S34. Perform residual connection and layer normalization on the output of the second-level linear transformation in S33 and the normalization result in S32;

[0174] Add residual join: ;

[0175] Wherein, Residual2 represents the residual connection result; FFNOutput represents the output of the second-level linear transformation in S33; and Norm1 represents the output of the normalized residual result in S32.

[0176] Application layer normalization: Normalize the residual connection results, and denote the output as Norm2.

[0177] Objective: To further stabilize the training and output the final high-dimensional feature representation.

[0178] The formula for layer normalization is:

[0179]

[0180]

[0181]

[0182] in, Indicates to Perform normalization. This represents the i-th input vector. It is the input vector (calculated along the last dimension), i.e., the residual connection result Residual2; d is the mean, and d is the hidden dimension. It is variance. It is a small constant (usually 1e-5) to prevent division by zero. and These are learnable parameters (initialized to 1 and 0).

[0183] S35. Output high-dimensional feature representation:

[0184] Norm2 is the high-dimensional feature representation of this encoder layer, which can be directly used in downstream intrusion detection models (such as by adding a classification head).

[0185] If it is a multi-layer Transformer, then Norm2 is used as the input of the next layer.

[0186] In the context of intrusion detection, these features can be fed into a simple linear classifier that optimizes the cross-entropy loss function.

[0187] The entire process is performed layer by layer and implemented in the Transformer model on the local client. The computational overhead mainly comes from matrix multiplication, which can be executed efficiently using frameworks such as PyTorch (e.g., using GPU acceleration on edge devices).

[0188] In this embodiment, the complete Transformer sublayer formula (combining residuals and normalization) is:

[0189]

[0190] In practice: all operations are batch processed, supporting parallel computing.

[0191] parameter , , , , , Optimize through backpropagation.

[0192] For intrusion detection, the high_dim_features can be flattened and connected to a softmax classifier with the cross-entropy loss function. The specific calculation formula is as follows:

[0193]

[0194] in, This represents the one-hot encoding of the real label (1 for class i, 0 for others). This represents the softmax probability of the i-th class predicted by the model. C represents the number of classes.

[0195] Example parameters: hidden_dim=512, intermediate_dim=2048, number of layers=6, ensuring runtime <1 second / iteration on edge devices.

[0196] Local training employs gradient descent optimization (such as the Adam optimizer) to update model weights, but does not upload the original data. Specific parameters for this step include the number of attention heads (e.g., 8 heads) and the hidden dimension (e.g., 512) to ensure efficient operation on resource-constrained devices.

[0197] Specifically, the high-dimensional feature representation output is fed into a simple classification head (e.g., a linear layer), using the cross-entropy loss function as the optimization objective. This is suitable for classification problems in intrusion detection (such as binary or multi-class intrusion classification). The loss function is optimized using gradient descent (e.g., the Adam optimizer), updating only the local model weights without uploading the original data.

[0198] These aspects align with the Transformer feature extraction (adding a feedforward network and layer normalization to output high-dimensional features), parameter settings (e.g., 8 attention heads, 512 hidden dimensions), and efficient operation requirements in this application. The entire process is executed locally on the client side, supporting resource-constrained devices (such as edge sensors), and efficiency is ensured through mini-batch training and parameter optimization. The specific optimization process is as follows:

[0199] 1. Obtain features and labels: Obtain high-dimensional feature representations from S3, flatten or pool the features, and input the true labels from the local dataset;

[0200] Obtain a high-dimensional feature representation (Norm2, with shape (batch_size, sequence_length, hidden_dim), e.g., hidden_dim=512) from S3.

[0201] Flatten or pool the features (e.g., by average pooling or by taking the [CLS] token representation) to obtain a fixed-dimensional vector for downstream classification.

[0202] Input the true label (y) of the local dataset, such as the category label in intrusion detection (one-hot encoded, e.g., normal / intrusion).

[0203] 2. Construct a classification head: Add a linear classification layer to map high-dimensional features to the number of categories;

[0204] Add a linear classification layer: map the high-dimensional features to the number of classes C (e.g., C=2 for binary classification).

[0205] Output logits(z): ;

[0206] Where logits(z) represents the raw output of the classification head (a linear layer); Represents a logits vector; Represents the weight matrix (hidden_dim × C); This indicates the bias term. This represents the refined, high-dimensional numerical representation of the final output of the Transformer encoder.

[0207] 3. Calculate the predicted probability;

[0208] Applying the softmax function to logits yields the predicted probability distribution, specifically:

[0209]

[0210] in, This represents the softmax probability of the i-th class predicted by the model; Let z represent the original output score (i=1,2,3,…,C) corresponding to the i-th class in the logits vector z.

[0211] 4. Calculate the loss function: Use cross-entropy loss to quantify the difference between the prediction and the true label.

[0212] Cross-entropy loss function The specific optimization objective is as follows:

[0213]

[0214] in, This represents the one-hot encoding of the real label (1 for class i, 0 for others). This represents the softmax probability of the i-th class predicted by the model. C represents the number of classes. This formula measures the distributional dissimilarity, is applicable to classification tasks, and facilitates the model's learning of intrusion patterns.

[0215] 5. Optimization and Update: Use the Adam optimizer to calculate gradients and backpropagate to update model weights (including Transformer parameters and classification head).

[0216] Adam optimizer update (example simplified formula for gradient descent):

[0217] Adam combines momentum and adaptive learning rate, and the update rule is as follows:

[0218]

[0219] in, This represents the model weights (including the attention head and hidden layers). This represents the learning rate (e.g., 1e-3). , These represent the first-order and second-order momentum estimates, respectively. This represents a small constant (1e-8). This represents the value of the model weights at the t-th iteration (or time step).

[0220] In this embodiment, Adam is specified to efficiently handle resource-constrained devices.

[0221] Iteratively train the local dataset until the local epoch ends or the loss converges (e.g., fix 5-10 epochs to adapt to resource-constrained devices).

[0222] 6. Monitoring and Termination:

[0223] Evaluate the loss after each mini-batch to ensure efficiency on edge devices (small batch size, such as 32, to avoid memory overflow).

[0224] The original data is not uploaded; it is transmitted only after S4 encryption weights are applied.

[0225] Calculation process:

[0226] Initialization: Load the local dataset (Non-IID data, such as traffic logs) and set the parameters (number of attention heads = 8, hidden dimension = 512, learning rate = 1e-3, batch_size = 32).

[0227] Forward propagation: Input batch data, extract features through Transformer → calculate logits with classification head → obtain p with softmax → calculate Loss.

[0228] Backpropagation: Calculate the gradient (∂Loss / ∂θ) based on Loss, and adjust the weights using Adam.

[0229] Iterative loop: For each epoch (e.g., 5 rounds): Traverse the local dataset batch.

[0230] Calculate the average loss (monitoring decline).

[0231] S4. Encrypted transmission: After the client completes local training, it encrypts the model weights (tensor form) using the Paillier encryption protocol, and then transmits the weights to the central server through a secure channel.

[0232] Paillier is a public-key homomorphic encryption scheme that generates a public key (n, g) and a private key (λ, μ). Here, n represents the modulus, which is the product of two large prime numbers p and q, i.e., n = p × q. It is part of the public key and defines the modulus range (mod n²) for encryption operations. g represents the base or generator, which is part of the public key. It is a randomly chosen integer belonging to the multiplicative group coprime to n² modulo n², typically taking the value n+1 or other integers whose order is a multiple of n. It is used to encode plaintext information and support homomorphic properties. λ represents Carmichael's totient function, which is lcm(p-1, q-1) (lcm represents the least common multiple of p-1 and q-1). It is part of the private key and is used in the decryption process to ensure the security of the scheme. μ represents the modular inverse, which is... The inverse modulo n, i.e.:

[0233] ,

[0234] ,

[0235] in, As part of the private key, it is used to recover the plaintext from the ciphertext; n represents the modulus; L represents the modulo operation, that is, performing a modulo n² operation on the encrypted value. Representing the base g Power of 1.

[0236] The encryption process is as follows: For weight m, the specific formula for calculating ciphertext c is:

[0237] ;

[0238] Where r is a random number; This indicates a plaintext message, specifically referring to model weights in the context of this application, i.e., the numerical data to be encrypted; n represents the modulus; g represents the base or generator. This represents the base g raised to the power of m, used to encode the plaintext information m (model weights). This represents a random number r raised to the power of n, used to introduce randomness and enhance the security of encryption (probabilistic encryption).

[0239] This protocol supports additive homomorphism and scalar multiplication homomorphism, facilitating subsequent aggregation. The additive homomorphism is as follows:

[0240] ,

[0241] Here, 'a' represents a plaintext value, which in the Paillier encryption addition homomorphic demonstration is the value or message to be encrypted (e.g., a portion of the model weights), used to show that the product of two encrypted values ​​corresponds to the plaintext addition operation; 'b' represents another plaintext value, similar to 'a', which serves as the second plaintext input for encryption in the addition homomorphic method, used to calculate... .

[0242] Paillier encryption supports scalar multiplication homomorphism, meaning that for a plaintext scalar k (a constant integer) and encrypted ciphertext Enc(m), multiplication in the encrypted state can be achieved through exponentiation without decryption. This property plays an important role in federated learning aggregation, such as in calculating weighted contributions. At that time, it can be executed directly in the encrypted domain.

[0243] The calculation formula is:

[0244]

[0245] Where k represents a scalar constant, m represents the plaintext message, Enc(m) represents the Paillier encrypted ciphertext of m, and n represents the modulus (as defined in the public key definition).

[0246] Calculation process (using the square-multiplication algorithm to efficiently implement exponentiation, avoiding direct calculation of large exponents):

[0247] Step 1, Initialization: Take the ciphertext c=Enc(m), the result res =1 (corresponding to Enc(0), but actually the unit element).

[0248] Step 2, Binary decomposition: Convert the scalar k into a binary representation.

[0249] Step 3, Iterative Calculation:

[0250] Iterate through the binary bits starting from the highest bit of k.

[0251] every time: .

[0252] If the current bit is 1, then (Multiplied by Enc(m)).

[0253] Step 4, Output: ;k represents a scalar constant;m represents a plaintext message.

[0254] Efficiency considerations: In actual implementation, a large integer library is used for acceleration; for floating-point k1, quantization is performed first (e.g., k1*10). d (Convert to integer, then dequantize); the process is applied during aggregation on a central server to ensure privacy.

[0255] After encryption, the weights are transmitted to the central server through a secure channel. In the specific implementation, the key length is set to 1024 bits or more to balance security and computational overhead; before transmission, the weights can be quantized and compressed (such as 8-bit quantization) to reduce the amount of data.

[0256] S5. Central Server Aggregation: After receiving the encryption weights from multiple clients, the central server performs an aggregation operation to generate a global encryption model.

[0257] Utilizing Paillier's homomorphic property, a weighted average is applied to the encryption weights. The formula for calculating the global weight is:

[0258] ,

[0259] in, The client weight coefficient (based on data volume or performance) is represented by homomorphic multiplication and addition. The global model weights are aggregated results generated by the central server after weighted averaging of encrypted weights from multiple clients. They are used as a teacher model for knowledge distillation. This represents the local model weights of the i-th client, which are the model parameters that the client encrypts and uploads to the central server after completing local training. This refers to the encryption function, specifically the encryption operation of the Paillier encryption protocol, which is used to perform homomorphic encryption on model weights to ensure privacy is protected during transmission and aggregation without revealing the original weight information.

[0260] This step generates an encrypted global model, avoiding the risk of centralized decryption. The specific algorithm is similar to FedAvg, but integrates homomorphic operations: first, the global model is initialized, then the encryption contribution is iteratively accumulated, and finally, the encrypted global model is output.

[0261] This step primarily describes leveraging the homomorphic properties of the Paillier encryption protocol to perform a weighted average aggregation of the weights of encrypted models uploaded by multiple clients, generating a global model in encrypted form. The core of this process is a variant of the FedAvg (Federated Averaging) algorithm, but with integrated homomorphic operations to ensure the entire aggregation is performed within the cryptographic domain, preventing the decryption operation from leaking the original weight information. The specific steps are as follows:

[0262] Step 1: Initialize the global model;

[0263] Global model weights Initialize it as a zero vector (in the cryptographic domain, it represents Enc(0), but in practice, Enc(0) can be taken as 1 if g = n+1).

[0264] formula (For all weighted elements); This represents the encryption function.

[0265] Process: The server generates a zero-encryption tensor with the same structure as the local model as the starting point. This avoids the risk of leakage introduced by plaintext initialization.

[0266] Step 2: Iteratively accumulate encrypted contributions:

[0267] For each client i (i=1 to N), the server calculates the client's cryptographic contribution. Then, it is accumulated into the global model.

[0268] Using Paillier homomorphism, the formula for scalar multiplication is:

[0269] ,

[0270] in, Indicates the encryption function; n represents the modulus; This represents the client-side weighting coefficient; This represents the local model weights of the i-th client; assuming... It is an integer; if It's a floating-point number, so you can first quantize it to an integer, such as by multiplying it by a precision factor of 10. k .

[0271] Addition accumulation:

[0272]

[0273] in, Indicates the encryption function; n represents the modulus; Represents the global model weights; This represents the client-side weighting coefficient; This represents the local model weights of the i-th client.

[0274] If it is a weighted average, it needs to be divided by the total weight. However, since the encrypted field does not support division, this can be handled after decryption or adjusted. This is the pre-normalized value.

[0275] In this embodiment, noise can be selectively introduced to enhance differential privacy. Specifically, before accumulation, for each... Add Laplace noise. The specific formula is:

[0276] ,

[0277]

[0278] ,

[0279] in, Indicates the encryption function; This represents the noisy or perturbed model weights. The tilde is often used to represent approximate or perturbed values; in the context of differential privacy in a document, it represents the original... The result after adding noise η is used to prevent the original data from being reverse-engineered through weights; This represents the local model weights of the i-th client. represents the noise term, sampled from the Laplace distribution, used to achieve differential privacy. b represents the scale parameter, which controls the degree of noise dispersion (the larger the b, the greater the noise and the stronger the privacy protection, but the model accuracy may decrease slightly). Indicates sensitivity, Represents a constant. =1.0.

[0280] Noise addition utilizes homomorphism: .

[0281] Process: Iterate through N clients, accumulating the values ​​one by one. The weights are high-dimensional tensors, so homomorphic operations are applied independently to each element (parallelizable).

[0282] Step 3: Output the encrypted global model:

[0283] After the iteration is completed, This is the final encrypted global model, which is directly used in subsequent steps (such as decryption before knowledge distillation, or further operations in the encrypted state).

[0284] If normalization is required, then: (in, Represents the global model weights; This represents the client-side weighting coefficient; (Represents the local model weights of the i-th client), which can be divided after decryption (server-private only), or pre-defined. Designed for However, scalar multiplication requires handling fractions (through fixed-point quantization).

[0285] During the aggregation process, noise (such as differential privacy mechanism, ε=1.0) can be introduced to further enhance privacy.

[0286] S6. Knowledge Distillation Feedback: The central server decrypts the aggregated global model (using a private key) as the teacher model, then applies knowledge distillation technology to extract knowledge, and encrypts the extracted knowledge before feeding it back to each client to update their local models; for example... Figure 9 As shown.

[0287] The distillation process includes generating soft labels: the teacher model calculates the softmax output on a shared pseudo-dataset (or anonymized samples uploaded by the client), and the softening probability distribution of the temperature parameter t (e.g., t=5) is as follows:

[0288] ,

[0289] in, represents the softened probability for class i of the soft labels. It is the softmax output of the teacher model after temperature softening, used to extract knowledge and feed it back to the student model, providing richer distribution information than hard labels; t represents the temperature parameter, a positive real number (e.g., t=5), used to control the degree of softening of the softmax distribution; when t>1, the distribution is smoother (softened), facilitating knowledge transfer; when t=1, it degenerates into standard softmax. This represents the score for class j in the logits vector, where j represents the summation index over classes, and it iterates through all classes.

[0290] Subsequently, the student model (client-side local model) minimizes the difference between itself and the teacher's soft labels using KL divergence loss, specifically as follows:

[0291] ,

[0292] Here, LOSS represents the total loss function, which is the optimization objective of the student model during knowledge distillation. It minimizes the difference between the soft labels of the student model and the teacher model by combining cross-entropy loss and KL divergence loss, while also considering the influence of the true labels, and is used to update the client's local model; CE represents cross-entropy, KL represents divergence, and α represents the balance factor. Indicates the true label; This represents the predicted probabilities of the student model, which is the softened probability output by the softmax function of the client-side local model (student model), used for comparison with the true labels and the teacher's soft labels; This represents the soft label probability distribution of the teacher model, which is the aggregated softmax output of the global model (teacher model) softened by the temperature parameter t. It provides rich distribution knowledge to feed back into the optimization of the student model. At the same time, intermediate representations (such as the output of the attention layer) can be distilled to transfer feature knowledge.

[0293] The extracted knowledge (soft labels or intermediate vectors) is encrypted and fed back to each client to update the local model. This step specifically optimizes the Non-IID distribution and improves student generalization through teacher guidance.

[0294] Based on the existing Paillier protocol framework in this application (supporting additive homomorphism and scalar multiplication), and adapting to the characteristics of knowledge distillation output (soft labels are probability vectors, and intermediate vectors are high-dimensional tensors), the specific process of encrypting the extracted knowledge is as follows:

[0295] Step 1: Quantization: Soft tags or intermediate vectors are usually floating-point numbers (range 0-1), which need to be quantized to integers for compatibility with Paillier (integer encryption). For example, using fixed-point quantization: multiply by a precision factor. (k=6, to ensure accuracy). The formula is:

[0296]

[0297] in, This indicates that the decrypted plaintext message is the original m (if there is no perturbation) recovered from the ciphertext c. This represents the i-th component or instance of the query vector in the attention mechanism. k represents a scalar constant used for weight scaling operations.

[0298] Step 2, element-by-element encryption: The soft tag / intermediate vector is a tensor. Paillier encryption is applied independently to each element, and parallel computing is used to accelerate the process.

[0299] For each element of the extracted knowledge (quantized soft label value or intermediate vector value):

[0300]

[0301] Where m represents the plaintext message, specifically the model weight, i.e., the numerical data to be encrypted; g represents the cardinality (the public key part, usually g=n+1); and r represents a random number (from...). Uniform sampling provides probabilistic security. n represents the modulus (public key, n = p × q, where p and q are large prime numbers). The base g is raised to the power of m, used to encode the plaintext message m; This represents a random number r raised to the power of n, used to introduce randomness.

[0302] Encryption loop: Iterate through tensor elements, generate a random value r, and calculate c.

[0303] Step 3, Packaging and Transmission: After encryption, an encrypted tensor is formed and distributed through a secure channel (TLS encryption).

[0304] Transmission: Pack the encrypted tensors into a serialized format (such as JSON or Protocol Buffers) and send them through a secure channel.

[0305] Overhead considerations: Encryption time is proportional to the vector dimension and can be optimized through batch processing.

[0306] Step 4, Client Decryption and Integration: The client uses its private key to decrypt the data, dequantizes it, and integrates it into the local loss function.

[0307] Client-side feedback integration process: After receiving the encrypted information, the client decrypts it using its private key, specifically:

[0308]

[0309]

[0310]

[0311]

[0312] Where m represents the plaintext message, specifically the model weights, i.e., the numerical data to be encrypted. λ represents the Carmichael function value; This represents a function used to calculate the least common multiple of two or more integers; p and q represent prime numbers; It represents the modular inverse, which is The inverse modulo n; n represents the modulus. Representing the base g Power of 1. This is part of the private key and is used to recover the plaintext from the ciphertext. n represents the modulus. This represents the ciphertext c. Power of 1.

[0313] Inverse quantization: .

[0314] in, This represents the original value of "knowledge" extracted through knowledge distillation. It is a scaling factor. This represents the decrypted plaintext message. k represents a scalar constant.

[0315] Update the local model: Integrate the decrypted knowledge into the loss function and fine-tune the local Transformer model using gradient descent.

[0316] Process: Receive - Decrypt element by element - Dequantize - Calculate loss - Optimizer update.

[0317] This step is used to update the local model. Specifically, it optimizes the Non-IID distribution and improves student generalization through teacher guidance.

[0318] S7. Iterative optimization: Perform closed-loop iterations of the above steps until the global model converges.

[0319] The above steps form a closed-loop iteration until the global model converges (e.g., based on a validation set accuracy threshold > 95% or a fixed number of rounds).

[0320] In this embodiment, the convergence criterion is:

[0321] Step 1: Use validation set accuracy as the primary metric: After each round of aggregation, the central server evaluates the global model using a shared validation set (anonymous samples). The calculation formula is:

[0322]

[0323]

[0324] in, This is the client-side weighting coefficient (based on data volume). Let be the precision of the i-th client on the local validation set; N represents the total number of clients.

[0325] Threshold judgment: If the global accuracy is >95% and there is no significant improvement for 3 consecutive rounds (e.g., improvement <0.5%), then convergence is achieved.

[0326] Step 2, the calculation process for the early stopping mechanism is as follows:

[0327] Initialization: Set the patience parameter (patience=5, threshold for the number of consecutive rounds without improvement) and the best accuracy parameter (best_acc=0).

[0328] After each iteration: Calculate the current global precision (acc_current).

[0329] Determine if acc_current > best_acc, then update best_acc = acc_current and reset the counter; otherwise, increment the counter by 1.

[0330] If the counter > patience, or the number of iteration rounds > max_rounds (e.g., 100), then stop iterating.

[0331] Step 3, Loss Convergence Supplement Formula: Monitoring Global Loss ;

[0332] in, The client-side local loss (cross-entropy); This represents the client weighting coefficient. N represents the total number of clients.

[0333] if (ε=0.001, for 3 consecutive rounds), then convergence occurs.

[0334] in, This represents the global loss value calculated at the end of the t-th iteration. This represents the global loss. t represents the training time step or round.

[0335] This represents the global loss value calculated at the end of the (t-1)th iteration (the iteration preceding the tth iteration). This represents the global loss. t-1 represents the previous training round.

[0336] Asynchronous mode extension: In the FedAsync variant, convergence can be determined based on the average accuracy of the sliding window (e.g., the average of the last 10 rounds is >95%).

[0337] After each iteration, the client uses feedback knowledge to fine-tune its local Transformer model, while the central server monitors aggregation stability. Specific implementations include synchronous / asynchronous aggregation modes (such as the FedAsync variant) and convergence checks (such as early stopping mechanisms).

[0338] This embodiment also provides a privacy-preserving federated learning system based on Transformer and knowledge distillation, which implements the above-mentioned privacy-preserving federated learning method based on Transformer and knowledge distillation.

[0339] 1. Application scenarios;

[0340] This invention is applied to privacy-sensitive distributed environment products, for example:

[0341] 1) Industrial Internet of Things (IIoT) Platforms: Such as smart factory monitoring systems. In industries like chemical or power, edge devices (such as sensors) collect traffic logs or video stream data for real-time intrusion detection. Product Example: A software platform called "SecureIIoT Guardian" is deployed on an edge server in the factory, eliminating the need to upload sensitive production data to the cloud.

[0342] 2) Distributed cloud platforms: such as multi-cloud management tools, enabling intrusion detection across power grids or cloud nodes. Product example: "FedSecure Cloud," a cloud security service product suitable for enterprise-level distributed systems, helps administrators monitor intrusion events across geographically distributed nodes while complying with privacy regulations such as GDPR.

[0343] 3) Extended Scenarios: It can be extended to medical image analysis platforms (processing patient data) or financial transaction monitoring systems, and is applicable to any privacy protection scenario involving Non-IID data distribution and edge computing. This solution addresses the data centralization risks of traditional products and improves its applicability to resource-constrained devices (such as low-power sensors).

[0344] Through these scenarios, the product helps users achieve a "zero-trust" security model in sensitive industries, namely, localized data processing, while ensuring detection accuracy >95%.

[0345] 2. Functional characteristics;

[0346] The features focus on privacy protection, performance optimization, and ease of use, specifically including:

[0347] 1) Privacy Protection: Utilizes the Paillier encryption protocol to ensure data integrity during model weight transmission. Product Features: Supports homomorphic encrypted aggregation, generating a global model without decryption; integrates differential privacy mechanisms (…). =1.0), quantifying privacy risks. Users can view privacy compliance reports in the product, such as the "Data Breach Risk Assessment" module, which displays the current privacy compliance status. value.

[0348] 2) Intrusion Detection Performance: Feature extraction is based on the Transformer attention mechanism, combined with knowledge distillation to optimize generalization of Non-IID data. Product Features: Detection accuracy reaches 96.5% (e.g., on the KDD Cup 99 dataset), supports real-time alerts (e.g., abnormal traffic detection); knowledge distillation compresses model size (reducing parameters by 50%), suitable for low-computing scenarios on edge devices. Functional modules include an "Intrusion Alert Center" that displays detection results and confidence levels.

[0349] 3) Federated Learning Optimization: Iterative closed-loop training, supporting synchronous / asynchronous modes. Product Features: Automatically handles Non-IID distribution bias, improving system robustness; experiments show a 20% reduction in iteration time, suitable for large-scale clients (such as hundreds of edge nodes).

[0350] 4) Extensions and Integration: The product supports parameter adjustments (such as 1024-bit encryption key length, distillation temperature t=5) and can be integrated with third-party APIs (such as cloud service interfaces). Other features: Multi-user role support (administrator / operator), log auditing, and performance dashboards.

[0351] These features differentiate the product from traditional intrusion detection software (such as tools based on centralized ML), emphasizing distributed privacy balance and achieving the core values ​​of "efficiency, security, and scalability".

[0352] The privacy-accuracy trade-off curve of this invention is as follows: Figure 10 As shown.

[0353] 3. Operation method;

[0354] The product employs a modular design, allowing users (such as system administrators or security engineers) to configure, train, and monitor it through a graphical user interface (GUI). The operation process is divided into three stages: deployment, training iteration, and result viewing, emphasizing automation to lower the user barrier. The following description illustrates the interaction process.

[0355] 1) Deployment phase:

[0356] User login to the product web dashboard (main interface: the left navigation bar includes "Client Management", "Server Configuration" and "Training Settings").

[0357] Operation: Click the "Add Client" button and enter the edge device information (such as IP address, data type: traffic log or video stream). The product automatically deploys a Transformer-based local model (for example, select "BERT variant" or "ViT visual variant").

[0358] Interaction: A configuration wizard UI pops up, allowing you to set parameters such as the number of attention heads (default 8 heads) and the hidden dimension (default 512). After confirmation, the system generates a Paillier public / private key pair and embeds it into the communication module.

[0359] Example UI: A form interface with a drop-down menu to select a scenario (chemical / power) and a preview of the framework structure diagram (showing embedded layers - Encoder Layers - output).

[0360] 2) Training iteration phase:

[0361] Operation: Click the "Start Federated Training" button, and the system enters closed-loop iteration. Features are extracted locally on the client (Transformer processes the data and captures temporal dependencies), and encrypted weights are uploaded to the server.

[0362] Interactive Process: A real-time progress bar UI displays each step (e.g., "Local extraction in progress: Client 1 50% complete"). After aggregation on the server, knowledge distillation is performed to feed back to the client. Users can monitor the dashboard: a graph shows the change in accuracy (target > 95%) and adjust parameters.

[0363] Example interaction diagram: Assume a flowchart UI where users can click on nodes to view details, such as clicking "knowledge distillation" to bring up a dialog box displaying the loss function; and allow manual intervention (such as pausing iteration).

[0364] Automation: The system supports an early stop mechanism, which automatically stops when the accuracy threshold is reached; users can choose synchronous mode (all clients upload simultaneously) or asynchronous mode (FedAsync variant).

[0365] 3) Results viewing and maintenance phase:

[0366] Operation: After training converges, check the "Detection Report" module to see intrusion alerts (such as "Abnormal traffic detected, accuracy 96.5%) and performance metrics (iteration time reduced by 20%).

[0367] Interaction: Export logs or reports; if a problem is detected, the user can restart the iteration or adjust the encryption key length.

[0368] Example UI: The dashboard homepage displays a list of real-time alerts (in table format: time, client, detection type, confidence level). Clicking on an alert will take you to a detailed view, including feature visualization.

[0369] The hardware and software environment of this invention is set to a distributed architecture, suitable for resource-constrained edge devices and centralized servers, supporting high concurrency and secure communication. The hardware environment includes multiple client devices (edge ​​nodes) and a central server; the software environment is based on programming languages ​​such as Python, integrating deep learning frameworks (such as PyTorch for Transformer implementation) and cryptographic libraries (such as python-paillier for the Paillier protocol).

[0370] 1. Hardware environment:

[0371] 1) Client devices: Edge devices, such as IIoT sensor nodes or cloud edge nodes (e.g., Raspberry Pi or industrial embedded devices). Each client is equipped with a CPU / GPU (minimum configuration: 4-core CPU, 2GB RAM) for local model training. Multiple clients (N≥2) are supported, suitable for distributed scenarios.

[0372] 2) Central Server: A high-performance server (e.g., Intel Xeon processor, ≥16GB RAM, NVIDIA GPU optional) responsible for model aggregation and knowledge distillation. The server must support a high-bandwidth network interface (≥1Gbps) to handle the transmission of encrypted data.

[0373] 3) Communication module: Integrated network interface between the client and server, supporting TCP / IP protocol. Hardware-wise, it can use wireless / wired networks (such as 5G or Ethernet) to ensure low-latency transmission.

[0374] 2. Software environment:

[0375] 1) Operating System: The client uses a lightweight Linux distribution (such as Ubuntu ARM version); the server uses Ubuntu Server.

[0376] 2) Core libraries and frameworks:

[0377] Deep learning: PyTorch (for implementing Transformer models and supporting attention mechanisms).

[0378] Encryption protocol: Paillier library (supports homomorphic addition operations, key length ≥ 1024 bits).

[0379] Federated Learning Framework: A custom implementation of a variant based on the FedAvg algorithm.

[0380] Knowledge distillation: Integrates Softmax temperature parameter (t=5 by default).

[0381] Data processing: NumPy / Pandas is used for preprocessing, suitable for Non-IID datasets (such as KDD Cup 99).

[0382] 3) Deployment method: Containerized deployment, which is easy to extend to cloud platforms.

[0383] The hardware environment of this invention emphasizes edge computing to reduce the risks of data-centric transmission; the software integrates Transformer (attention mechanism to capture long-distance dependencies) and Paillier (homomorphic encryption to avoid decryption) to achieve a balance between privacy and performance, making it suitable for sensitive industries such as chemical / power.

[0384] like Figure 1 The diagram shown is a network topology diagram of this application; the topology diagram shows a star structure: the client connects to the server through an encrypted channel to avoid point-to-point leakage.

[0385] The system of this invention includes a front-end (client) and a back-end (central server), employing an iterative optimization mechanism. The front-end is responsible for local processing, while the back-end is responsible for global coordination. The entire logic is based on a federated learning paradigm, integrating Transformer extraction and knowledge distillation feedback.

[0386] 1. Front-end implementation logic (client-side):

[0387] 1) Module composition: feature extractor (Transformer-based), local trainer, and encrypted transmitter.

[0388] 2) Implementation steps:

[0389] Data input: Collect data from local sensors (such as traffic logs) and preprocess it into a sequence format.

[0390] Feature extraction: The Transformer model (multi-layer Encoder, including Self-Attention and Feed-Forward layers) is used to capture long-distance dependencies and improve the representation quality of Non-IID data.

[0391] Local training: Train an intrusion detection model (such as a classifier) ​​based on extracted features and optimize the loss function.

[0392] Encrypted transmission: The model weights are encrypted using the Paillier protocol (the public key is distributed by the server) and transmitted to the server.

[0393] The front end uses Transformer as a feature extractor to solve the problem of insufficient generalization of traditional CNNs on Non-IID data; edge devices only require small models and have low computing power overhead.

[0394] Backend implementation logic (central server):

[0395] 1) Module composition: Aggregator, Knowledge Distiller, Feedback Distributor.

[0396] 2) Implementation steps:

[0397] Receive encrypted weights: collected from multiple clients.

[0398] Aggregation: Utilize Paillier homomorphism to perform weighted averaging and generate a global model without decryption.

[0399] Knowledge distillation: The global model acts as the teacher model, generating soft labels (temperature t=5) to extract knowledge.

[0400] Feedback: Distribute the distilled knowledge back to the client to update the local model.

[0401] The backend integrates knowledge distillation to optimize the generalization of small models (improving accuracy by 8%); homomorphic encryption ensures privacy and security during the aggregation process, making it suitable for scenarios with high accuracy requirements (>95%).

[0402] like Figure 2 The diagram shows the data processing flow.

[0403] The data processing workflow employs a closed-loop iterative approach: from local data collection to global optimization, it repeats until convergence (e.g., accuracy > 95% or iterations > 50 rounds). The workflow emphasizes privacy protection: data never leaves the client; only encrypted weights are transmitted.

[0404] 1. Detailed process:

[0405] 1) Local data collection and extraction: The client collects Non-IID data and uses Transformer to extract features.

[0406] 2) Training and Encryption: After local training, the weights are transmitted in encrypted form.

[0407] 3) Aggregation and distillation: The server aggregates to generate a global model and distills knowledge.

[0408] 4) Feedback and iteration: Knowledge is returned to the client to update the model.

[0409] 5) Evaluation: In intrusion detection tasks, measure accuracy and privacy.

[0410] like Figure 3As shown, the timing diagram illustrates the parallel operation of multiple clients, emphasizing the asynchronous nature of encrypted transmission. The timing diagram of this invention does not involve data decryption, minimizing privacy risks; distillation further enhances overall performance.

[0411] Based on the above description, this invention achieves efficient and privacy-preserving federated learning in terms of technical implementation. Experiments have verified that the accuracy reaches 96.5% on the KDD Cup 99 dataset, making it suitable for scenarios such as IIoT intrusion detection.

[0412] The experimental verification process in this application aims to evaluate the system's performance in terms of privacy protection, model performance, and computational efficiency. The experiments employ a simulated distributed environment to model intrusion detection tasks in an Industrial Internet of Things (IIoT) scenario. The detailed experimental procedure is as follows:

[0413] 1. Dataset preparation:

[0414] A variant of the KDD Cup 99 dataset (a classic intrusion detection benchmark dataset containing 41-dimensional features and various attack types, such as DoS (Denial of Service) attacks and Probe attacks) is used. To simulate a Non-IID (non-independent and identically distributed) data distribution, the dataset is split into multiple clients (N=10 clients), with each client's data subset unevenly distributed in terms of attack type and sample size (e.g., client 1 is biased towards DoS attacks, client 2 towards normal traffic).

[0415] The NSL-KDD dataset is introduced as an additional validation dataset to reduce redundant samples and improve generalization testing.

[0416] Data preprocessing: Normalize feature values, convert to sequence format (sequence length = 128), and add positional encoding. The training / validation / test set ratio is 7:2:1, with a total sample size of approximately 500,000.

[0417] Prepare a shared pseudo dataset for knowledge distillation: Use a GAN generator to synthesize anonymized samples (approximately 10,000) to avoid leaking real data.

[0418] 2. Experimental environment setup:

[0419] Hardware: Client-side emulated edge devices (Raspberry Pi 4, 4GB RAM, CPU-only); central server uses NVIDIA RTX 3080 GPU, 16GB RAM.

[0420] Software: PyTorch 1.12 framework implementation of Transformer model (hidden dimension = 512, number of attention heads = 8, number of layers = 6); Paillier cryptography library (python-paillier, key length = 1024 bits); optimizer Adam (learning rate = 1e-3); knowledge distillation temperature t = 5, balance factor α = 0.5.

[0421] Parameter settings: Federated learning rounds = 50; local epochs = 5; differential privacy budget ε = 1.0 (add Laplace noise); client participation rate = 100% (synchronous mode), or 50% (asynchronous FedAsync variant).

[0422] Comparison baselines: (1) Traditional FedAvg (no Transformer, no distillation); (2) FedAvg+Paillier (no distillation); (3) FedAvg+Transformer (no distillation).

[0423] 3. Evaluation indicators:

[0424] Performance: Intrusion detection accuracy, recall, and F1 score.

[0425] Privacy: Differential privacy budget ε (the smaller the budget, the stronger the privacy), using privacy auditing tools to quantify the risk of leakage.

[0426] Efficiency: Local computation overhead (FLOPs / iteration time, in seconds); communication overhead (data transfer volume, in MB); overall iteration time.

[0427] Non-IID heterogeneity: controlled by the Dirichlet distribution parameter α (equilibrium factor) (α=0.1 for high heterogeneity, α=1.0 for low heterogeneity).

[0428] 4. Experimental Procedure:

[0429] Initialization: Randomly initialize model weights and generate Paillier key pairs.

[0430] Iterative training: Perform steps S1-S7. Evaluate the global model's performance on the shared validation set every 10 rounds.

[0431] Convergence judgment: Use the early stopping mechanism (patience=5). If the accuracy improvement is <0.5% or >95% for 5 consecutive rounds, stop the iteration.

[0432] Repeated experiment: run multiple times independently and take the mean ± standard deviation.

[0433] Extended testing: Repeated at different heterogeneity (α=0.1, 0.5, 1.0) and privacy budget (ε=0.5, 1.0, 2.0).

[0434] The experimental results are based on the KDD Cup 99 variant dataset, with N=10 clients and 50 rounds. The following is a summary of key data (mean ± standard deviation).

[0435] Results and Analysis:

[0436] (1) Model performance comparison;

[0437] The FedTD framework achieved state-of-the-art performance on the test set, with significantly higher accuracy, recall, and F1 score than all baseline models. Specific numerical comparisons are shown in Table 1 below.

[0438] Table 1 Model Performance Comparison

[0439]

[0440] The results show that simple encryption (FedAvg + Paillier) introduces a slight performance loss. The Transformer structure (FedAvg + Transformer) significantly improves the model's capabilities, demonstrating its advantage in capturing long-range dependencies. Furthermore, this invention, FedTD, combines the feature extraction capabilities of Transformer, the privacy protection capabilities of Paillier, and the generalization enhancement capabilities of knowledge distillation, achieving optimal performance with an accuracy improvement of over 8% compared to the basic FedAvg, fully demonstrating its innovation and effectiveness.

[0441] (2) Performance under different degrees of data heterogeneity;

[0442] The Dirichlet parameter α was adjusted to simulate data distributions with varying degrees of heterogeneity. As data heterogeneity increased (α value decreased), the performance of all models decreased, but the FedTD framework consistently maintained the highest accuracy and strongest robustness.

[0443] High heterogeneity (α=0.1): FedTD=94.8%±1.0% compared to FedAvg=82.5%±1.5% (an improvement of 12.3%).

[0444] Medium heterogeneity (α=0.5): FedTD=95.7%±0.9% compared to FedAvg=86.0%±1.3% (an improvement of 9.7%).

[0445] Low heterogeneity (α=1.0): FedTD=96.5%±0.8% compared to FedAvg=88.2%±1.2% (an improvement of 8.3%).

[0446] Experiments demonstrate the crucial role of knowledge distillation in mitigating the distribution problem of Non-IID data. The soft-label knowledge provided by the teacher model effectively feeds back to each client, guiding the local model to learn more generalizable features, thus maintaining excellent performance even in highly heterogeneous environments.

[0447] (3) Privacy-utility trade-off analysis;

[0448] There is a natural trade-off between the strength of privacy protection (measured by differential privacy budget ε) and model utility (accuracy). The performance variation of FedTD under different ε settings was evaluated.

[0449] ε=0.5 (strong privacy): Accuracy=94.2%±1.1%, extremely low risk of privacy leakage (<0.01%).

[0450] ε=1.0 (equilibrium): Accuracy=96.5%±0.8%, privacy leakage risk is controllable (<0.05%).

[0451] ε=2.0 (weak privacy): Accuracy=97.1%±0.7%, with a high risk of privacy leakage (<0.1%).

[0452] The results show that by flexibly adjusting the ε value, the framework of this invention can provide customizable solutions according to the different privacy and security requirements of actual application scenarios. In most applications requiring privacy protection (ε=1.0), this invention can maintain extremely high detection accuracy while ensuring high security.

[0453] (4) Computational and communication efficiency analysis;

[0454] Because of its integration with Transformer, the FedTD framework has a slightly higher FLOPs (floating-point operations per second) per iteration (1.2 GFLOPs) than the CNN-based FedAvg baseline (0.9 GFLOPs). However, thanks to knowledge distillation's optimization of model convergence, the iteration time required by FedTD (0.85 seconds ± 0.1 seconds) is actually about 20% shorter than FedAvg (1.1 seconds ± 0.15 seconds). In terms of communication, Paillier encryption and quantization compression increase the amount of data transmitted per round to 15MB, which is an increase compared to the baseline (12MB), but in return, it provides extremely strong privacy protection.

[0455] The results demonstrate the efficiency of the FedTD framework. It doesn't simply pile up complex modules; instead, through collaborative optimization between components (such as distillation to accelerate convergence), it achieves a net gain in computational efficiency while improving performance and protecting privacy, making it ideal for resource-constrained edge computing environments.

[0456] In summary, the FedTD framework proposed in this invention significantly outperforms existing federated learning baseline methods in terms of privacy protection, model performance, and computational efficiency. It not only addresses the performance degradation caused by Non-IID data but also provides provable privacy guarantees through homomorphic encryption and differential privacy, while optimizing overall efficiency through techniques such as knowledge distillation. The deep integration of the Transformer, Paillier, and knowledge distillation components generates significant synergistic effects, validating the enormous application potential of this invention in privacy-sensitive scenarios such as intrusion detection in the Industrial Internet of Things.

[0457] The core of this invention is a privacy-preserving federated learning system and method based on Transformer and knowledge distillation, aiming to construct a distributed learning framework that balances data security and detection efficiency. This framework is applicable to intrusion detection systems in Industrial Internet of Things (IIoT) and distributed cloud platforms. By processing data locally on the client side, encrypting and transmitting model updates, and aggregating and feeding back knowledge at a central server, it achieves data privacy protection (avoiding raw data leakage and complying with regulations such as GDPR), model generalization optimization under Non-IID (non-independent and identically distributed) data, and high-precision intrusion detection (accuracy > 95%). The overall process includes local feature extraction, encrypted transmission, server aggregation, knowledge distillation feedback, and iterative optimization, forming a closed-loop system that addresses the bottlenecks of traditional federated learning in terms of privacy, efficiency, and computational overhead.

[0458] This invention addresses the shortcomings of existing federated learning frameworks combined with homomorphic encryption by providing specific improvements:

[0459] 1) Solving the problem of low computational efficiency: Existing homomorphic encryption schemes result in high overhead. This invention addresses this by using the parallel attention computation of Transformer ("O("n")). 2 The complexity of the model is reduced by a high-efficiency GPU implementation, and Paillier optimizations (such as batch encryption) reduce the local / central computing burden. Knowledge distillation further compresses the model size (reducing parameters by more than 50%), enabling faster training on edge devices. Experiments show that the iteration time is reduced by 20% compared to traditional solutions.

[0460] 2) Solving the generalization problem under Non-IID data: Existing technologies show significant performance degradation. This invention uses Transformer to capture cross-client data dependencies and improve feature robustness; knowledge distillation balances the distribution bias through soft knowledge transfer, achieving an accuracy of 96.5% on the KDD Cup 99 Non-IID dataset, an improvement of 8%, thus solving the bottleneck.

[0461] 3) Addressing the issue of insufficient mechanism integration: Existing solutions lack deep integration. This invention organically integrates attention mechanism, federated learning, and distillation: Transformer provides high-quality input, encryption ensures secure transmission, and distillation optimizes output, achieving end-to-end balance; in sensitive scenarios such as power grids, the detection accuracy is >95%, and the privacy risk is reduced to ε=1.0, extending to fields such as medicine.

[0462] Through these detailed technical implementations, the present invention not only solves the root cause of existing problems, but also ensures the operability and high performance of the system in actual deployment.

[0463] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A privacy-preserving federated learning method based on Transformer and knowledge distillation, characterized in that: This includes using the attention mechanism of the Transformer model as a core component of local feature extraction to capture long-distance dependencies in data and improve the quality of feature representation; The Paillier encryption protocol is introduced to achieve homomorphic encrypted transmission of model weights; knowledge distillation technology is used on the central server to extract soft knowledge from the aggregated global model as the teacher model, which is then fed back to the client to compress and optimize the model, including the following steps: S1. Data Acquisition and Preprocessing: Acquire data from local sensors and preprocess it into a sequence format; S2. Feature Extraction: Deploy the Transformer model as a local feature extractor on each client device to perform feature extraction; S3. Local Model Training: Train the intrusion detection model based on extracted features and optimize the loss function; S31. Input the output of the self-attention mechanism and the original input, and perform residual connection on them; S32. Perform residual connection on the input of S31 and normalize the residual results; S33. Add a feedforward network to the normalized result of S32; S34. Perform residual connection and layer normalization on the output of the second-level linear transformation in S33 and the normalization result in S32; S35, Output high-dimensional feature representation; S4. Encrypted transmission: After the client completes local training, it encrypts the model weights using the Paillier encryption protocol and then transmits the weights to the central server through a secure channel. S5. Central Server Aggregation: After receiving the encrypted weights from multiple clients, the central server performs an aggregation operation to generate a global model. S6. Knowledge Distillation Feedback: The central server decrypts the aggregated global model as the teacher model, then applies knowledge distillation technology to extract knowledge, and encrypts the extracted knowledge to feed back to each client to update the local model. The distillation process includes generating soft labels: the teacher model calculates the softmax output on a shared pseudo-dataset, and the softening probability distribution of the temperature parameter t is as follows: ， wherein, denotes the softening probability of the i-th class of soft labels; t denotes a temperature parameter; denotes the score of the j-th class in the logits vector, j denotes the class index in the sum; Then, the student model minimizes the difference with the teacher's soft labels using KL divergence loss, specifically: ， Where LOSS represents the total loss function; CE represents the cross-entropy; KL represents the divergence; and α represents the balance factor. Indicates the true label; This represents the predicted probability distribution of the student model; The soft-label probability distribution represents the teacher model; S7. Iterative optimization: Perform closed-loop iterations of the above steps until the global model converges.

2. The privacy-preserving federated learning method based on Transformer and knowledge distillation according to claim 1, wherein: In S2, the specific implementation process is as follows: The encoder layer of the Transformer model calculates the dependency weights between input sequences using a multi-head self-attention mechanism. Specifically, it performs dot product attention calculations using query, key, and value matrices. ， wherein softmax() represents a normalization function; Q represents a query matrix, K represents a key matrix, denotes a key dimension, and T represents a transpose operation of a matrix.

3. The privacy-preserving federated learning method based on Transformer and knowledge distillation according to claim 1, characterized in that: In S4, the specific implementation process is as follows: The model weights are encrypted using the Paillier encryption protocol. The specific formula for calculating the ciphertext c is as follows: ， Where r represents a random number; Indicates plaintext message; n represents the modulus; g represents the cardinality; This represents the base g raised to the power of m, used to encode the plaintext message m; This represents a random number r raised to the power of n, used to introduce randomness.

4. The privacy-preserving federated learning method based on Transformer and knowledge distillation according to claim 1, characterized in that: In S5, the specific implementation process is as follows: Leveraging Paillier's homomorphic properties, a weighted average is applied to the encryption weights. The specific calculation formula is as follows: ， wherein, denotes a client weight coefficient; denotes a global model weight; denotes a local model weight of the i-th client; denotes an encryption function; Then, initialize the global model, iteratively accumulate the encryption contribution, and finally output the encrypted global model.

5. The privacy-preserving federated learning method based on Transformer and knowledge distillation according to claim 4, characterized in that: The specific content of the encrypted global model, which is initialized, iteratively accumulated, and finally output in encrypted form, is as follows: Step 1, Initialize global model: global model weights initialized to zero vector; Step 2, Iteratively accumulate encryption contribution: For each client, the server calculates the client's encryption contribution, then adds noise to the encryption contribution, and finally accumulates it into the global model; Step 3: Output the encrypted global model.

6. A privacy-preserving federated learning system based on Transformer and knowledge distillation, implementing the privacy-preserving federated learning method based on Transformer and knowledge distillation in any one of claims 1 to 5, characterized in that: Including client and central server; The client includes local sensors, a feature extractor, a local trainer, and an encrypted transmitter; Raw data is collected from local sensors and preprocessed to convert it into a sequence format; The preprocessed data is used to extract features using a feature extractor; an intrusion detection model is trained based on the extracted high-dimensional features using a local trainer, and the loss function is optimized; finally, the trained model weights are homomorphically encrypted using the Paillier encryption protocol using an encrypted transmitter, and then transmitted to the central server through a secure channel. The central server includes an aggregator, a knowledge distiller, and a feedback distributor; Encrypted weights are received from multiple clients. An aggregator is used to perform a weighted average aggregation of the encrypted weights, leveraging the homomorphic properties of Paillier, to generate an encrypted global model. A knowledge distiller is used to decrypt the aggregated global model as a teacher model. Knowledge distillation is applied to shared pseudo-datasets or anonymous samples to generate soft labels and intermediate representations, extracting knowledge to improve model generalization and compression. A feedback distributor is used to encrypt the extracted knowledge using the Paillier protocol and distribute it back to each client through a secure channel to update the local model and optimize performance under Non-IID data.

Images