A network attack detection method integrating transformer-federated learning-knowledge distillation

Abstract

The present invention discloses a network attack detection method integrating Transformer-federated learning-knowledge distillation, which collects network attack data sets of industrial Internet systems, uses Transformer deep learning models to extract and classify features of the data sets, and each local server performs federated learning training to upload the data to the central server using the Transformer encryption protocol. After the central server decrypts the overall data, it performs knowledge distillation feedback training on non-IID distribution data and returns to further perform decision evaluation and prediction until the detection meets the performance requirements and ends the program. The present invention effectively solves the defects of poor encryption performance and low detection accuracy of the current industrial control system, and builds an intrusion detection network framework based on Transformer-federated learning-knowledge distillation in a way that strengthens the privacy protection of the chemical industrial control system, improves the security of model parameters during training, and realizes the security protection of the chemical industrial control network system.

Description

Technical Field

[0001] The present invention relates to the fields of chemical industry interconnection system, network attack detection and process safety technology of complex industrial interconnection system, and specifically to a network attack detection method integrating transformer-federated learning-knowledge distillation. Background Art

[0002] Network attack detection technology is a crucial task in the security of chemical interconnection systems. Industrial control systems control important equipment and process flows in industrial processes. With the development of network information technology, industrial systems are becoming more and more dependent on network communications and information technology while realizing intelligent and green development. This also makes industrial interconnection systems face various risks and threats from the network. At the same time, for a large number of sensitive data sources involved in the interconnection system, including but not limited to production process parameters, equipment operating status, system operating parameters, etc., if they are invaded by malicious attackers and cause data leakage or damage, it will lead to serious economic losses and security risks.

[0003] In order to solve the network security problems in chemical control systems, the main methods currently on the market include intrusion detection systems (IDS), intrusion prevention systems (IPS), firewall technology, encryption technology, etc. Intrusion detection systems (IDS) detect potential attacks by monitoring network traffic and system activities, and can be divided into network intrusion detection systems (NIDS) and host intrusion detection systems (HIDS). Intrusion prevention systems (IPS) monitor network traffic to detect and prevent potential attack behaviors, and take active defense measures based on intrusion detection. Firewall technology is widely used to monitor and control traffic in and out of the network to prevent unauthorized access. They can be configured to identify and prevent potential attacks. Encryption technology is mainly used to protect data confidentiality and integrity, including encryption of communication data and encryption of stored data.

[0004] Although many technologies continue to emerge in people's vision, the domestic field of technology is still limited compared with foreign countries. At present, domestic industrial safety control still has problems such as expensive communication overhead, easy leakage of sensitive information, high security risks, and low detection accuracy when the system is non-IID distribution. In order to improve the accuracy of network attack detection in chemical safety systems, it is of great significance to carry out a network attack detection method that integrates transformer-federated learning-knowledge distillation.

[0005] In view of this, it is necessary to provide a new technology to solve at least part of the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide a network attack detection method integrating transformer-federated learning-knowledge distillation to address the problems of poor encryption performance, waste of communication resources and low detection accuracy when data is non-IID distribution in current network attack detection technology.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is: a network attack detection method integrating transformer-federated learning-knowledge distillation, which comprises the following steps:

[0008] S1: Introduce multi-party attention learning mechanism to design Transformer model, and establish intrusion detection model of chemical interconnected control system based on deep learning; use chemical industry network attack dataset to mark local industrial data and perform data preprocessing, and then input it into Transformer model for training to realize classification of detection system;

[0009] S2: Based on the results of system classification, the original chemical industry interconnection system data set is sorted and partitioned using known classification labels. The data in each partition is used to form a data subset using the federated learning framework for local model training to obtain weights. The qualified local model weights are screened out according to the designed threshold range and uploaded to the central server for aggregation to obtain the optimal global prediction weight.

[0010] S3: Introduce an unlabeled open dataset into each client dataset to achieve efficient classification of classes. To avoid generating low-quality logits on unlabeled open datasets, integrate knowledge distillation technology on the basis of federated learning framework, use large models to feed back small models, introduce multiple mechanisms to jointly improve the quality of logits in unlabeled regions, and thus form a Transformer-federated learning-knowledge distillation framework;

[0011] S4: Design a communication protocol of the Paillier cryptographic system between the local client and the central server. The local prediction weights uploaded by the participants will be encrypted using the Paillier encryption algorithm, and then transmitted to the central server using a secure channel. The system will then be decrypted using the Paillier decryption algorithm to complete the update of the system parameter model. This protocol protects the security and privacy of the model parameters during the training process.

[0012] S5: Based on the Transformer-federated learning-knowledge distillation framework model, a model fusion strategy is introduced. The knowledge distillation technology is used to transfer the trained global prediction model knowledge to the local sub-server. The performance of the integrated model is evaluated through the model integration method. The effectiveness of the model is proved through the calculation and performance comparison of various indicators.

[0013] Furthermore, the intrusion detection model of the chemical industry interconnected control system based on deep learning is constructed in S1, including:

[0014] S1-1. Clean the data set of cyber attacks on chemical interconnected control systems, identify duplicate data in the data set and process missing values;

[0015] S1-2. Based on the preprocessed data set, the network data of the chemical interconnected control system is organized into sequence data and the event position corresponding to each time step in the sequence is encoded using position coding. The constructed input sequence is input as training data into the transformer model for training. The structure formed by the attention mechanism and the neural network is used to achieve accurate feature extraction.

[0016] Furthermore, the formula used to identify duplicate data in the dataset and handle missing values ​​is as follows:

[0017] Duplicates = {x i ∈X|count(x i )>1}

[0018] X clean =X\Duplicates

[0019] Missing={x i ∈X|is_missing(x i )}

[0020] X clean =X\Missing

[0021] Duplicates represents duplicate records or duplicate data items in a data set, Missing represents missing data items in a data set, the is_missing function determines whether there are missing values ​​in the data and returns the value, and the count function calculates the number of times each record appears in a data set.

[0022] Furthermore, the weighted sum z of the attention mechanism i Expressed as

[0023]

[0024] Considering that there are multiple subsystems in the chemical industry interconnection system, a multi-party attention mechanism is used to ensure the accuracy of system classification. There are M client participants, each participant P i Corresponding to each input X i Through linear transformation, we can obtain the corresponding query, key, and value matrices Q i =X i W iQ ,K i =X i W i K ,V i =X i W i V , where W i V , W i K , W i Q is the learning parameter matrix, the Softmax function is used to normalize the attention scores to attention weights, z i Represents client P i Output.

[0025] Further, the S2 includes:

[0026] S2-1. Clients using the federated learning intrusion framework need to upload the average prediction of each label instead of uploading all parameters. Each client m has a private dataset D m (D m =D m，0 ∪D m，1 ∪...∪D m，L-1 ), Divide into L subsets according to the labels, and client m uses the corresponding dataset D m To train a local model, each client calculates the average value of each label. The result is called the local average logit vector. Details are as follows:

[0027]

[0028] represents the local average logit vector N is the number of private samples, n represents the vectorized features of the sample, the single-hot label of the sample, and the dataset D m The number of samples in is a one-hot vector, i.e.

[0029] S2-2, before the first round of communication begins, use the central server's deep learning model to initialize the client m's deep learning model, i.e., α m ←α s , where α s and α m denote the initial parameters of the deep learning model of the server and the initial parameters of the deep learning model of the client m respectively;

[0030] S2-3. Each client uploads its own average logit vector after local training To the central server, the central server broadcasts the global average logit vector to each client m. Client m trains its model using the new local average logit vector and label. The updated local average logit expression is as follows:

[0031]

[0032] Further, the S3 includes:

[0033] S3-1. Each client has a private labeled dataset and an unlabeled dataset in and N0 represent the vectorized features of the sample and the number of samples in the open dataset, respectively. Here we use the matrix X 0 To represent D 0 Each client uses a private labeled dataset to train a knowledge distillation model for classification prediction, makes predictions for each client m, and performs local logit on the unlabeled samples to output the probability distribution of each category, using the matrix To express The knowledge distillation model, the formula is as follows:

[0034]

[0035] S3-2. Initialize the global model w s , the server trains a global model with a shared unlabeled dataset and global logit, namely:

[0036]

[0037] Where γ represents the learning rate, ψ(·,·) represents the cross entropy loss function, Represents the connection matrix Propagate global information to each client;

[0038] S3-3. Each client uploads local data to the central server, which aggregates the local logits and calculates the entropy of the global logits. The simplification is as follows:

[0039]

[0040] Where S is the corresponding p under condition T j The softmax function is expressed as follows:

[0041]

[0042] When T<1, p j The distribution of is more obvious, and the information entropy decreases accordingly; while reducing the global logit, the stability of the federation-knowledge distillation model in the non-IID distribution is accelerated;

[0043] S3-4, refer to multiple mechanisms to jointly improve the classification quality, retrain the discriminator based on S1-2, and create an empty set If the confidence score is less than the bound number θ, then the sample will be added to the set And the sample is classified as "unfamiliar" to client m; the process is described as:

[0044]

[0045] where one-hot labels [1,0] are used T to indicate unfamiliar labels;

[0046] Each sample in the private dataset is “familiar” to client m, so all samples in the private dataset are added to D m,d , and the label is "familiar"; the process is expressed as:

[0047]

[0048] S3-5, the prediction of the unlabeled open samples in S1-2 is filtered by the trained discriminator, and the formula for calculating the discrimination result is:

[0049]

[0050] F(·|w m,d The output of ) is a two-dimensional vector, argmax(·) is the index of the maximum value of the vector, for 0 Each sample in Predicted The filtering method is as follows:

[0051]

[0052] -1 indicates that the sample is unfamiliar. Each client uploads the hard labels of the unlabeled open data to the central server;

[0053] S3-6, based on the data filtered in the previous step, voting and propagation are performed. After all clients input the predicted labels of the samples into the corresponding voting sets, the central server determines the global hard label of the sample according to the number of votes in each voting set, that is, The global hard label of a sample is calculated as follows:

[0054]

[0055] Furthermore, the Paillier encryption algorithm in S4 includes:

[0056] S4-1. Given a safety factor The trust agency generates a public key PA = (N, G) and a corresponding private key SA = (λ, υ) according to the Paillier cryptographic system. The trust agency publishes the public key PA and distributes the private key SA to all local servers, and each of them establishes a corresponding secure communication channel. The trust agency is the central cloud server and each local server A i Generate a symmetric key S respectively i , where G is the generator, N is the product of two prime numbers, υ=(L(G λ modN 2 )) -1 modN, L is defined as L(β) = (β-1) / N;

[0057] S4-2. Define encryption function υ′=f(υ)=10 -8 ·υmodN and given information me, through the calculation of the encryption function me′=f(me), each model parameter uploaded by the local server is converted into a positive integer me′∈Z N , select a random number and encrypt it, where the encryption algorithm of the public key PA model parameters is:

[0058] E PAi (me) = G f(me) ·r N modN 2 =G me′ ·r N modN 2

[0059] where r∈Z N * is a random number;

[0060] S4-3, given the contribution rate of each industrial agent {β1,β2,...,β M}, the cloud server magnifies the contribution rate by a certain multiple and converts it into a positive integer. The model parameters of M cloud servers {E PAi (me1),E PAi (me2),...,E PAi (me M ),} fuses the data of the server, and the cloud server aggregates the data to form the ciphertext c through the following steps:

[0061]

[0062] S4-4. After receiving the ciphertext c from the cloud server, each industrial agent decrypts it by summing up the updated model parameters as follows:

[0063]

[0064] The formula for calculating the sum of the average values ​​of the updated model parameters is: Where 1000 represents a scalar used to convert the contribution rate into a positive integer, and the function υ = f -1 (υ′)=10 -8 υ′modN converts the original model parameters to positive or negative, and then restores the updated model parameters to the original scale by:

[0065]

[0066] Furthermore, the performance evaluation in S5 includes:

[0067] S5-1. Evaluate the local model and verify the performance. Use the threshold to evaluate whether it matches the current model. If it is higher than or equal to the threshold, it means that the local model does not match the current global model. Models smaller than the threshold are left for further global model aggregation. The specific formula is as follows:

[0068]

[0069]

[0070] threhold=0.5-ψ f ×0.01

[0071] Where M is the number of devices, E i is the parameter value of device i, c i is the mean difference between device i and all devices, and c is calculated using the sigmoid function. i The output value e is obtained by conversion i , the interval is (0,1), ψ f Adjust parameters for threshold;

[0072] S5-2. Use the integrated model to test the data set and determine the accuracy of the overall model through accuracy, precision, recall, and false negative rate evaluation indicators.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] The present invention adopts the network attack detection technology that integrates the transformer-federated learning-knowledge distillation framework, which effectively solves the defects of poor encryption performance and low detection accuracy of the current industrial control system. In order to strengthen the privacy protection of the chemical industrial control system, a transformer-based and federated learning intrusion detection network framework is built, which improves the security of the model parameters during the training process. It has been proved in the actual CPS data set that the training results meet the performance requirements and realize the security protection of the chemical industrial control network system. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a schematic diagram of the local client and central server network attack detection principle proposed by the present invention.

[0076] Figure 2 It is a flowchart of the system network attack detection based on Paillier encryption of the present invention. DETAILED DESCRIPTION

[0077] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] A network attack detection method that integrates Transformer-federated learning-knowledge distillation. The detection method collects a network attack data set of an industrial Internet system, uses a Transformer deep learning model to extract and classify the data set, and each local server performs federated learning training and uploads the data to the central server using the Transformer encryption protocol. The central server decrypts the overall data, performs knowledge distillation feedback training on non-IID distribution data, and returns it for further decision evaluation and prediction until the detection meets the performance requirements and the program ends.

[0079] The specific steps of this network attack detection method are described as follows:

[0080] Step 1: An improved Transformer model is designed by introducing the multi-party attention learning mechanism. A deep learning-based intrusion detection method for chemical industry interconnected control systems is established. The chemical industry network attack dataset is used to mark the local industrial data and perform data preprocessing. After data cleaning, feature extraction, serialization and other operations, the data is input into the Transformer for training to realize the classification of the detection system.

[0081] S1-1. Perform data cleaning on the chemical industry interconnection system network attack data set to identify duplicate data in the data set and process missing values:

[0082] Duplicates = {x i ∈X|count(x i )>1}

[0083] X clean =X\Duplicates

[0084] Missing={x i ∈X|is_missing(x i )}

[0085] X clean =X\Missing

[0086] Duplicates refers to duplicate records or duplicate data items in a data set, Missing refers to missing data items in a data set, the is_missing function determines whether there are missing values ​​in the data and returns the value, and the count function calculates the number of times each record appears in the data set.

[0087] S1-2. On the basis of the preprocessed data set, the network data of the chemical interconnection system is organized into sequence data and the event position corresponding to each time step in the sequence is encoded using position coding. The constructed input sequence is input into the Transformer model as training data for training. The structure formed by the attention mechanism and the neural network is used to achieve accurate feature extraction. The weighted sum of the attention mechanism and z i It is expressed as:

[0088]

[0089] Considering that there are multiple subsystems in the chemical industry interconnection system, a multi-party attention mechanism is used to ensure the accuracy of system classification. There are M client participants (local models), and each participant P i Corresponding to each input X i Through linear transformation, we can obtain the corresponding query, key, and value matrices Q i =X i W i Q , K i =X i W i K 、V i =X i W i V , where W i V , W i K , W i Q is the learning parameter matrix. The Softmax function is used to normalize the attention scores to attention weights, z i Represents client P i Output.

[0090] Step 2: According to the results of system classification, the original chemical industry interconnection system data set is sorted and partitioned using known classification labels. The data in each partition is formed into data subsets using the federated learning framework for local model training to obtain weights. The qualified local model weights are screened out according to the designed threshold range and uploaded to the central server for aggregation to obtain the optimal global prediction weight.

[0091] S2-1. Clients using the federated learning intrusion framework need to upload the average prediction of each label instead of uploading all parameters. Each client m has a private dataset D m (D m =D m，0 ∪D m，1 ∪...∪D m，L-1 ), Divide into L subsets according to the labels, and client m uses the corresponding dataset D m To train a local model, each client calculates the average value of each label, and the result is called the local average logit vector. Details are as follows:

[0092]

[0093]

[0094] represents the local average logit vector N is the number of private samples, n represents the vectorized features of the sample, the single-hot label of the sample, and the dataset D m The number of samples in is a one-hot vector, i.e.

[0095] S2-2, before the first round of communication begins, use the central server's deep learning model to initialize the client m's deep learning model parameters, i.e., α m ←α s , where α s and α m They represent the initial parameters of the deep learning model of the central server and the initial parameters of the deep learning model of the client m respectively.

[0096] S2-3. Each client uploads its own average logit vector after local training To the central server, the central server broadcasts the global average logit vector to each client, and client m trains its model using the new local average logit vector and label. The updated local average logit expression is as follows:

[0097]

[0098] Step 3: Considering that the private dataset in the local server is non-IID (non-independent and identical) distributed, an unlabeled open dataset is introduced into each client dataset to achieve efficient class division. In order to avoid generating low-quality logits on the unlabeled open dataset, knowledge distillation technology is integrated on the basis of the federated learning framework, and a large model is used to feed back the small model. Multiple mechanisms are introduced to jointly improve the quality of logits in unlabeled areas, thereby forming a Transformer-federated learning-knowledge distillation framework.

[0099] Step 3-1. According to the proposed network attack detection framework, each client has a labeled private dataset and an unlabeled dataset. in and N0 represent the vectorized features of the sample and the number of samples in the open dataset, respectively. Here we use the matrix X 0 To represent D 0 Each client uses a private labeled dataset to train a knowledge distillation model for classification prediction, makes predictions for each client m, and performs local logit on the unlabeled samples to output the probability distribution of each category, using the matrix To express The knowledge distillation model, the formula is as follows:

[0100]

[0101] S3-2. Initialize the global model w s , the server trains a global model with a shared unlabeled dataset and global logit, namely:

[0102]

[0103] where γ represents the learning rate, ψ represents the cross entropy loss function, Represents the connection matrix Propagate global information to each client.

[0104] S3-3. Each client uploads local data to the central server, which aggregates the local logits and calculates the entropy of the global logits. The simplification is as follows:

[0105]

[0106] in Represents the connection matrix The information is transmitted from the central server to each client through this connection matrix. S is the corresponding p under condition T. j The softmax function is:

[0107]

[0108] When T<1, p j The distribution of is more obvious, and the information entropy decreases accordingly; while reducing the global logit, it speeds up the stabilization of the federated-knowledge distillation model in the non-IID distribution.

[0109] S3-4. Since it is difficult to generate high-quality logits from unlabeled open datasets, multiple mechanisms need to be used to jointly improve the classification quality. Therefore, the discriminator is retrained based on S1-2 to create an empty set. If the confidence score is less than the bound number θ, then the sample will be added to the set And the sample is classified as "unfamiliar" to customer m. The process is described as:

[0110]

[0111] where we use one-hot labels [1,0] T To represent unfamiliar labels. More importantly, we consider every sample in the private dataset to be “familiar” to client m, so all samples in the private dataset are added to D m,d and the label is "familiar". The process is expressed as:

[0112]

[0113] S3-5, the prediction of the unlabeled open samples in S1-2 is filtered by the trained discriminator, and the formula for calculating the discrimination result is:

[0114]

[0115] F(·|w m,d ) is a two-dimensional vector; argmax(·) is the index of the maximum value of the vector; as in S3-4, Then the client is not familiar with the sample. 0 Each sample in Predicted The filtering method is as follows:

[0116]

[0117] "-1" indicates that the sample is unfamiliar; each client will unlabel the hard label of the open data (i.e. ) to the central server.

[0118] S3-6. Voting and dissemination are required based on the data filtered in the previous step. Sample There are L voting sets, namely {|V j,0 |,|V j,1 |,...,|V j,L-1 |}. Class-1 is used to identify unfamiliar samples, so class-1 is not included in the L voting sets, and the predicted label of the sample like The predicted label is added to the corresponding voting set. After all clients input the predicted labels of samples (excluding unfamiliar samples) into the corresponding voting set, the central server determines the global hard label of the sample (i.e. ). The global hard label of the sample is calculated as follows:

[0119]

[0120] Step 4: Design a communication protocol for the Paillier cryptographic system between the local client and the central server. The local prediction weights uploaded by the participants (each node in the chemical interconnected control system) will be encrypted using the Paillier encryption algorithm, and then transmitted to the central server using a secure channel. After a series of Paillier decryptions, the system decryption is achieved and the system parameter model is updated. This protocol protects the security and privacy of the model parameters during the training process.

[0121] S4-1. Given a safety factor The trust agency generates a public key PA = (N, G) and a corresponding private key SA = (λ, υ) according to the Paillier cryptographic system. The trust agency publishes the public key PA and distributes the private key SA to all local servers, and each of them establishes a corresponding secure communication channel. The trust agency is the central cloud server and each local server A i Generate a symmetric key S respectively i .

[0122] Where G is the generator, N is the product of two prime numbers, υ=(L(G λ modN 2 )) -1 modN,L is defined as L(β)=(β-1) / N.

[0123] S4-2. Define encryption function υ′=f(υ)=10 -8 ·υmodN and given information me, through the calculation of the encryption function me′=f(me), each model parameter uploaded by the local server is converted into a positive integer me′∈ZN , select a random number and encrypt it, where the encryption algorithm of the public key PA model parameters is:

[0124] E PAi (me) = G f(me) ·r N modN 2 =G me′ ·r N modN 2

[0125] where r∈Z N * Is a random number.

[0126] S4-3, given the contribution rate of each industrial agent {β1,β2,...,β M}, the cloud server magnifies the contribution rate by 1000 times and converts it into a positive integer. There are M cloud servers with model parameters {E PAi (me1),E PAi (me2),...,E PAi (me M ),} can merge the data of the server, and the cloud server aggregates the data to form the ciphertext c through the following steps:

[0127]

[0128] S4-4. After receiving the ciphertext c from the cloud server, each industrial agent will decrypt it by summing up the updated model parameters as follows:

[0129]

[0130] The formula for calculating the sum of the average values ​​of the updated model parameters is: Where 1000 represents a scalar used to convert the contribution rate into a positive integer, and the function υ = f -1 (υ′)=10 -8 υ′modN can convert the original model parameters to positive (υ=f(υ)) less than N / 2 after conversion) or negative (greater than N / 2 after conversion), and then restore the updated model parameters to the original scale by:

[0131]

[0132] Step 5: Based on the proposed Transformer-federated learning-knowledge distillation model, a model fusion strategy is introduced. The knowledge distillation technology is used to transfer the trained global prediction model knowledge to the local sub-server. The performance of the integrated model is evaluated through the model integration method. The effectiveness of the model is proved through the calculation and performance comparison of various indicators.

[0133] S5-1. Evaluate the local model and verify the performance. Use the threshold to evaluate whether it matches the current model. If it is higher than or equal to the threshold, it means that the local model does not match the current global model. If and only if the model is less than the threshold, the global model is retained for further aggregation. The specific formula is as follows:

[0134]

[0135] threhold=0.5-ψ f ×0.01

[0136] Where M is the number of devices, E i is the parameter value of device i, c i is the mean difference between device i and all devices, and c is calculated using the sigmoid function. i The output value e is obtained by conversion i , the interval is (0,1), ψ f Adjust the parameters for the threshold.

[0137] S5-2: Use the integrated model to test the data set and determine the accuracy of the overall model through evaluation indicators such as accuracy, precision, recall, and false negative rate:

[0138]

[0139] Accuracy refers to the correct proportion of the model's prediction results, Precision refers to the number of cases that are actually cyber attacks among the examples that are identified as cyber attacks, and Recall refers to the proportion of the total number of samples that are actually cyber attacks among the total number of actual positive samples predicted by the model. FPR refers to the proportion of the number of samples that are incorrectly classified as positive among the examples that are actually identified as not cyber attacks.

[0140] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention. The parts not involved in the present invention are the same as the prior art or can be implemented by the prior art.

Claims

1. A network attack detection method integrating transformer-federated learning-knowledge distillation, characterized in that The steps include: S1: Introduce multi-party attention learning mechanism to design Transformer model, and establish intrusion detection model of chemical interconnected control system based on deep learning; use chemical industry network attack dataset to mark local industrial data and perform data preprocessing, and then input it into Transformer model for training to realize classification of detection system; S2: Based on the results of system classification, the original chemical industry interconnection system data set is sorted and partitioned using known classification labels. The data in each partition is used to form a data subset using the federated learning framework for local model training to obtain weights. The qualified local model weights are screened out according to the designed threshold range and uploaded to the central server for aggregation to obtain the optimal global prediction weight. S3: Introduce an unlabeled open dataset into each client dataset to achieve efficient classification of classes. To avoid generating low-quality logits on unlabeled open datasets, integrate knowledge distillation technology on the basis of federated learning framework, use large models to feed back small models, introduce multiple mechanisms to jointly improve the quality of logits in unlabeled regions, and thus form a Transformer-federated learning-knowledge distillation framework; S4: Design a communication protocol of the Paillier cryptographic system between the local client and the central server. The local prediction weights uploaded by the participants will be encrypted using the Paillier encryption algorithm, and then transmitted to the central server using a secure channel. The system will then be decrypted using the Paillier decryption algorithm to complete the update of the system parameter model. This protocol protects the security and privacy of the model parameters during the training process. S5: Based on the Transformer-federated learning-knowledge distillation framework model, a model fusion strategy is introduced. The knowledge distillation technology is used to transfer the trained global prediction model knowledge to the local sub-server. The performance of the integrated model is evaluated through the model integration method. The effectiveness of the model is proved through the calculation and performance comparison of various indicators. The step S2 comprises: S2-1. Clients using the federated learning intrusion framework need to upload the average prediction of each label instead of uploading all parameters. Each client m has a private dataset D m (D m =D m，0 ∪D m，1 ∪...∪D m，L-1 ), Divide into L subsets according to the labels, and client m uses the corresponding dataset D m To train a local model, each client calculates the average value of each label. The result is called the local average logit vector. Details are as follows: represents the local average logit vector N is the number of private samples, n represents the vectorized features of the sample, the single-hot label of the sample, and the dataset D m The number of samples in is a one-hot vector, i.e. S2-2, before the first round of communication begins, use the central server's deep learning model to initialize the client m's deep learning model, i.e., α m ←α s , where α s and α m denote the initial parameters of the deep learning model of the server and the initial parameters of the deep learning model of the client m respectively; S2-3. Each client uploads its own average logit vector after local training To the central server, the central server broadcasts the global average logit vector to each client m, and client m uses the new local average logit vector and label to train its model. The updated local average logit expression is as follows:

2. According to claim 1, a network attack detection method integrating transformer-federated learning-knowledge distillation is characterized in that: The intrusion detection model of the chemical industry interconnected control system based on deep learning constructed in S1 includes: S1-1. Clean the data set of cyber attacks on chemical interconnected control systems, identify duplicate data in the data set and process missing values; S1-2. Based on the preprocessed data set, the network data of the chemical interconnected control system is organized into sequence data and the event position corresponding to each time step in the sequence is encoded using position coding. The constructed input sequence is input as training data into the transformer model for training. The structure formed by the attention mechanism and the neural network is used to achieve accurate feature extraction.

3. A network attack detection method integrating transformer-federated learning-knowledge distillation according to claim 2, characterized in that: The formula used to identify duplicate data in the data set and handle missing values ​​is as follows: Duplicates={x i ∈X|count(x i )>1} X clean =X\Duplicates Missing={x i ∈X|is_missing(x i )} X clean =X\Missing Duplicates represents duplicate records or duplicate data items in a data set, Missing represents missing data items in a data set, the is_missing function determines whether there are missing values ​​in the data and returns the value, and the count function calculates the number of times each record appears in a data set.

4. A network attack detection method integrating transformer-federated learning-knowledge distillation according to claim 2, characterized in that: The weighted sum z of the attention mechanism i Expressed as Considering that there are multiple subsystems in the chemical industry interconnection system, a multi-party attention mechanism is used to ensure the accuracy of system classification. There are M client participants, each participant P i Corresponding to each input X i Through linear transformation, we can obtain the corresponding query, key, and value matrices Q i =X i W i Q ,K i =X i W i K ,V i =X i W i V , where W i V , W i K , W i Q is the learning parameter matrix, the Softmax function is used to normalize the attention scores to attention weights, z i Represents client P i Output.

5. A network attack detection method integrating transformer-federated learning-knowledge distillation according to claim 2, characterized in that: The S3 includes: S3-1. Each client has a private labeled dataset and an unlabeled dataset. in and N0 represent the vectorized features of the sample and the number of samples in the open dataset, respectively. Here we use the matrix X 0 To represent D 0 Each client uses a private labeled dataset to train a knowledge distillation model for classification prediction, makes predictions for each client m, and performs local logit on the unlabeled samples to output the probability distribution of each category, using the matrix To express The knowledge distillation model, the formula is as follows: S3-2. Initialize the global model w s , the server trains a global model with a shared unlabeled dataset and global logit, namely: Where γ represents the learning rate, ψ(·,·) represents the cross entropy loss function, Represents the connection matrix Propagate global information to each client; S3-3. Each client uploads local data to the central server, which aggregates the local logits and calculates the entropy of the global logits. The simplification is as follows: Where S is the corresponding p under condition T j The softmax function is expressed as follows: When T<1, p j The distribution of is more obvious, and the information entropy decreases accordingly; while reducing the global logit, the stability of the federation-knowledge distillation model in the non-IID distribution is accelerated; S3-4, refer to multiple mechanisms to jointly improve the classification quality, retrain the discriminator based on S1-2, and create an empty set If the confidence score is less than the bound number θ, then the sample will be added to the set And the sample is classified as "unfamiliar" to client m; the process is described as: where one-hot labels [1,0] are used T to indicate unfamiliar labels; Each sample in the private dataset is "familiar" to client m, so all samples in the private dataset are added to D m,d , and the label is "familiar"; the process is expressed as: S3-5, the prediction of the unlabeled open samples in S1-2 is filtered by the trained discriminator, and the formula for calculating the discrimination result is: F(·|w m,d The output of ) is a two-dimensional vector, argmax(·) is the index of the maximum value of the vector, for 0 Each sample in Predicted The filtering method is as follows: -1 indicates that the sample is unfamiliar. Each client uploads the hard labels of the unlabeled open data to the central server; S3-6, based on the data filtered in the previous step, voting and propagation are performed. After all clients input the predicted labels of the samples into the corresponding voting sets, the central server determines the global hard label of the sample according to the number of votes in each voting set, that is, The global hard label of a sample is calculated as follows:

6. A network attack detection method integrating transformer-federated learning-knowledge distillation according to claim 1, characterized in that: The Paillier encryption algorithm in S4 includes: S4-1. Given a safety factor The trust agency generates a public key PA = (N, G) and a corresponding private key SA = (λ, υ) according to the Paillier cryptographic system. The trust agency publishes the public key PA and distributes the private key SA to all local servers, and each of them establishes a corresponding secure communication channel. The trust agency is the central cloud server and each local server A i Generate a symmetric key S respectively i , where G is the generator, N is the product of two prime numbers, υ=(L(G λ modN 2 )) -1 modN, L is defined as L(β) = (β-1) / N; S4-2. Define encryption function υ′=f(υ)=10 -8 ·υmodN and given information me, through the calculation of the encryption function me′=f(me), each model parameter uploaded by the local server is converted into a positive integer me′∈Z N , select a random number and encrypt it, where the encryption algorithm of the public key PA model parameters is: And PAi (May)=G f(me) ·r N modN 2 =G me′ ·r N modN 2 where r∈Z N * is a random number; S4-3, given the contribution rate of each industrial agent {β1,β2,...,β M }, the cloud server magnifies the contribution rate by a certain multiple and converts it into a positive integer. The model parameters of M cloud servers {E PAi (me1),E PAi (me2),...,E PAi (me M ),} fuses the data of the server, and the cloud server aggregates the data to form the ciphertext c through the following steps: S4-4. After receiving the ciphertext c from the cloud server, each industrial agent decrypts it by summing up the updated model parameters as follows: The formula for calculating the sum of the average values ​​of the updated model parameters is: Where 1000 represents a scalar used to convert the contribution rate into a positive integer, and the function υ = f -1 (υ′)=10 -8 υ′modN converts the original model parameters to positive or negative, and then restores the updated model parameters to the original scale by:

7. A network attack detection method integrating transformer-federated learning-knowledge distillation according to claim 1, characterized in that: The performance evaluation in S5 includes: S5-1. Evaluate the local model and verify the performance. Use the threshold to evaluate whether it matches the current model. If it is higher than or equal to the threshold, it means that the local model does not match the current global model. Models smaller than the threshold are left for further global model aggregation. The specific formula is as follows: threhold=0.5-ψ f ×0.01 Where M is the number of devices, E i is the parameter value of device i, c i is the mean difference between device i and all devices, and c is calculated using the sigmoid function. i The output value e is obtained by conversion i , the interval is (0,1), ψ f Adjust parameters for threshold; S5-2. Use the integrated model to test the data set and determine the accuracy of the overall model through accuracy, precision, recall, and false negative rate evaluation indicators.

Images