Distributed intrusion detection system based on parallel chain and federated learning framework

Abstract

The invention relates to a distributed intrusion detection system based on parallel chains and a federated learning framework, and belongs to the technical field of block chains. The intrusion detection system comprises a relay chain, a data chain, a contribution chain, a client and an interstellar file system. The data chain and the contribution chain are two parallel chains; the relay chain is respectively in communication connection with the data chain and the contribution chain, the client is in communication connection with the data chain, and the interstellar file system is respectively in communication connection with the client, the data chain and the contribution chain. The relay chain is mainly composed of verifier nodes, the data chain is mainly composed of data chain collector nodes, and the contribution chain is mainly composed of contribution chain collector nodes. The method has the advantages that reward and punishment are fair, upgrading or connection with other block chain systems in the future is more convenient, and data tampering and hostile attacks can be effectively prevented.

Description

Technical Field

[0001] The present invention belongs to the field of blockchain technology, and specifically relates to a distributed intrusion detection system based on parallel chains and a federated learning framework. Background Art

[0002] With the rapid development of the Internet and the continuous change of network attack technology, network security issues have received more and more attention. Intrusion detection system can monitor network attack behaviors and potential security risks by analyzing network traffic. It is a commonly used network security tool. Intrusion detection systems can be roughly divided into two categories: centralized intrusion detection systems and distributed intrusion detection systems. Compared with centralized intrusion detection systems, distributed intrusion detection systems can achieve efficient information sharing and collaborative detection, and have stronger disaster tolerance and disaster recovery capabilities. Distributed intrusion detection systems generally require multiple parties to share data and build detection models based on machine learning, but data sharing may lead to the leakage of individual user privacy. Federated learning can achieve the purpose of collaborative training models while protecting user private data, which can effectively solve the aforementioned privacy issues.

[0003] There are already some intrusion detection systems based on federated learning. These systems usually require users to train local models and upload the trained local model gradients to a central server, which then aggregates the global model. However, this centralized federated learning approach has the following problems:

[0004] 1) Imperfect reward and punishment mechanism: The reward and punishment rules of centralized federated learning are generally formulated and implemented by the central server, which means that users need to fully trust the central server, and the central server needs to be absolutely fair when rewarding or punishing, otherwise it will infringe on the rights and interests of users.

[0005] 2) Single point of failure: The central server is the key node of the entire system. Once the server fails or is attacked, the operation of the entire system will be affected.

[0006] 3) Historical data can be tampered with: If the central server uses a traditional database to store data, the historical data stored on the server can be controlled and modified by roles with corresponding permissions.

[0007] Blockchain is an integration of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other technologies. In a narrow sense, blockchain can be regarded as a decentralized distributed ledger, and each node in the network saves a complete copy of the data. Each block in the blockchain ledger contains the hash value of the previous block. Any tampering with the data will destroy the entire chain, so the blockchain has the property of being tamper-proof. Using blockchain to store data in the intrusion detection system can effectively prevent the data from being maliciously tampered with. Combining blockchain with federated learning to build a decentralized federated learning architecture can also help solve the aforementioned single point failure problem. In addition, smart contracts can be run on the blockchain. Once the contract is deployed on the blockchain, it is difficult to tamper with it, and the contract code is open and transparent. Using smart contract technology to implement a reward and punishment mechanism can effectively ensure fairness. Summary of the invention

[0008] In view of the problems of imperfect reward and punishment mechanism, single point failure, and tamperable historical data in the prior art, the present invention provides a distributed intrusion detection system based on the Polkadot blockchain framework and federated learning. The system consists of a relay chain responsible for consensus verification and cross-chain communication of the entire blockchain network and two parallel chains responsible for data storage and contribution evaluation respectively. The parallel chains are independent of each other and connected to the relay chain respectively. The relay chain is responsible for completing the interactive communication between different parallel chains. The two parallel chains are named data chain and contribution chain respectively. The data chain can interact with the client, is responsible for storing parameters such as model gradients of federated learning, and is responsible for aggregating the global model; the contribution chain is responsible for evaluating the contribution of the local models submitted by each client to the global model, and storing the contribution evaluation results.

[0009] The characteristics of the present invention are that the blockchain system is divided into two independent parallel chains and a relay chain according to the different execution functions and storage contents. The parallel chains perform different tasks respectively and are managed uniformly by the relay chain; the contribution evaluation of federated learning is realized by using smart contract technology to ensure the openness and transparency of the reward and punishment mechanism; the distributed architecture of the blockchain is used to avoid single point failure problems; and the tamper-proof nature of the blockchain is used to ensure the integrity of the model parameters.

[0010] The technical solution of the present invention is as follows.

[0011] A distributed intrusion detection system based on a parallel chain and a federated learning framework, the intrusion detection system includes a relay chain, a data chain, a contribution chain, a client and an InterPlanetary File System (IPFS), the data chain and the contribution chain are two parallel chains, the relay chain is communicatively connected to the data chain and the contribution chain respectively, the data chain is communicatively connected to the contribution chain, the client is communicatively connected to the data chain, and the InterPlanetary File System is communicatively connected to the client, the data chain and the contribution chain respectively, as described in detail as follows.

[0012] The relay chain is the central blockchain responsible for handling the consensus mechanism and cross-chain communication of the entire blockchain network. It is mainly composed of validator nodes. Validators are nodes in the relay chain that serve as validators. Validator nodes are responsible for block generation. They will verify the candidate blocks submitted by the collector, determine whether the block header signature of the candidate block is valid, and place the block header with a valid signature on the relay chain for consensus. Validators exist in the form of groups and are uniformly managed by the relay chain. When generating new blocks, a hybrid consensus protocol is used. First, the BABE (Blind Assignment for Blockchain Extension, BABE) protocol is used to determine the new block producer in the validator group, and the generated new blocks need to use GRANDPA (GHOST-based Recursive ANcestor Deriving Prefix Agreement) to achieve final confirmation to avoid chain bifurcation and rollback.

[0013] The data chain is a parallel chain responsible for storing the model parameter data and global models uploaded by users and performing global model aggregation. It is mainly composed of Data Chain Collator nodes. The Data Chain Collector is a node that plays the role of a collector on the data chain. It is mainly responsible for collecting transactions containing local model gradients initiated by the client, and providing the transaction and the validity proof of the transaction to one or more validators responsible for the data chain on the relay chain for verification, making it a candidate block, and then confirmed by the validator on the relay chain to become a formal block. At the same time, the data chain node uses the WebAssembly runtime to execute smart contracts in Wasm format. The smart contract uses the uploaded local model gradients to aggregate the global model, and packages the aggregated global model gradients into transactions and uploads them to the data chain for storage.

[0014] The Contribution Chain is a blockchain that uses Shapley Value to measure the contribution of each user. It is mainly composed of Contribution Chain Collator nodes. Contribution Chain Collator refers to the node that plays the role of collector on the Contribution Chain. The smart contract running on the Contribution Chain will retrieve the local model gradient and global model gradient of each round from the data chain across the chain, use Shapley Value to measure the contribution of each local model to the global model, and then package the contribution value into transactions. The collector will be responsible for collecting these transactions and providing the transaction and the validity proof of the transaction to one or more validators responsible for the Contribution Chain on the Relay Chain for verification, making it a candidate block, and finally verified by the validator to become a new block.

[0015] The client is responsible for providing an operational interface for users and packaging uploaded data into blockchain transactions. The client connects to the data chain collector, uploads the local model gradient to the data chain storage and participates in subsequent calculations.

[0016] IPFS is responsible for storing model gradient data larger than 6kb, and is connected to the client, data chain, and contribution chain respectively. The data flow between IPFS and the parachain system can be achieved by direct upload and download from the client, or by sending and downloading data to IPFS through the parachain smart contract.

[0017] The workflow of a distributed intrusion detection system based on a parallel chain and a federated learning framework of the present invention includes the steps of initialization, gradient parameter uploading, aggregating a global model, calculating contribution values, etc., which are further described as follows:

[0018] A. After setting the relay chain network parameters, create a genesis block, which includes the initial state and configuration information, configure and start the relay chain node that plays the role of a validator; load the genesis block and ensure that it is connected to other relay chain nodes. When starting a parachain, you need to create a relay chain full node that plays the role of a collector, which is responsible for the interaction between the parachain and the relay chain. This node is connected to another parachain collector node. Then the collector can hand over the collected candidate blocks to the relay chain, which will be verified and confirmed by the relay chain to complete the initialization of the parachain and relay chain.

[0019] B. Users use their own data to train local models and upload local model gradients to the client. The client uploads the data to IPFS and obtains the file content identifier (CID) from IPFS. The client initiates a transaction containing the CID of the local model data and the size of the data set. The transaction is broadcast within the data chain node and reaches consensus, making it a new block on the data chain. At the same time, the data chain collector collects these transactions and provides the transaction and the validity proof of the transaction to one or more validators responsible for the data chain on the relay chain for verification. The validators use a hybrid consensus protocol to verify and confirm, ultimately making the blocks on the data chain and the new blocks containing transaction information generated on the relay chain have the same authority.

[0020] C. A smart contract is deployed on the data chain. The smart contract uses the CID of the local model uploaded in this round to request IPFS to obtain the local model gradient, and uses the local model gradient to evaluate the quality of each model. Then the smart contract initiates a cross-chain transaction. The transaction content includes a request message to call the previous round of contribution evaluation results stored in the contribution chain. The transaction will be collected by the data chain collector and sent to the relay chain. The relay chain verifier will verify the legitimacy of the transaction, and then send the cross-chain transaction to the contribution chain. The transaction is processed by the smart contract deployed on the contribution chain. The smart contract on the contribution chain accesses its own stored data according to the request and returns the result to the smart contract that initiated the cross-chain call. If the current training is the first round, the contribution value returned by the contribution chain is 1. Otherwise, the contribution value returned by the contribution chain in the previous round is used. The contribution value and the model quality value are aggregated as weights, and then the weights and local model gradients are aggregated to obtain the global model. The global model is uploaded to IPFS, and the CID of the global model is packaged into a transaction and uploaded to the data chain.

[0021] D. The smart contract deployed on the contribution chain node will initiate a cross-chain transaction to obtain the CID of the global model gradient stored in the data chain and the CID of each local model gradient. The smart contract uses the CID to request the model gradient from the IPFS system, uses the Shapley value to measure the contribution of each local model to this round of training, and packages the contribution value into a transaction and stores it in the contribution chain for use in the next round of data chain aggregation of the global model.

[0022] After completing step D, step B will be repeated to record the local model gradient, global model gradient, and contribution value of each local model generated in each round of training.

[0023] Beneficial Effects

[0024] Compared with the existing distributed intrusion detection system, the distributed intrusion detection system proposed in the present invention has the following advantages:

[0025] (1) Fair rewards and punishments: The contribution evaluation of federated learning is completed by smart contracts. The evaluation process is open and transparent. Participants can trust the execution results of the contract, avoiding unfair situations caused by human factors.

[0026] (2) High degree of modularity: The present invention is based on blockchain framework technology. The design of the relay chain can ensure that the parallel chains do not affect each other. Each part has a high degree of modularity and strong independence, which makes it more convenient to upgrade or connect with other blockchain systems in the future.

[0027] (3) Immutability: Blockchain provides a decentralized data storage and transaction verification mechanism, which can effectively prevent data tampering and malicious attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 .System organizational structure diagram,

[0029] Figure 2 .Initialization process and data upload flow chart,

[0030] Figure 3 .Data flow diagram,

[0031] Figure 4 .Contribution value calculation diagram,

[0032] Figure 5 .Timing diagram. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings and examples. It is worth noting that the examples below are only used to explain the present invention, but do not constitute any limitation to the present invention.

[0034] See the instruction manual Figure 1 The present invention provides a distributed intrusion detection system based on a parallel chain and a federated learning framework, including a relay chain, a data chain, a contribution chain, a client and an interplanetary file system IPFS, wherein the data chain and the contribution chain are two parallel chains, the relay chain is communicatively connected to the data chain and the contribution chain respectively, the data chain is communicatively connected to the contribution chain, the client is communicatively connected to the data chain, and the interplanetary file system is communicatively connected to the client, the data chain and the contribution chain respectively.

[0035] See the instruction manual Figure 2 , 3 , 4 and 5, the establishment and operation of a distributed intrusion detection system based on a parallel chain and a federated learning framework of the present invention are described as follows:

[0036] S1. See the instruction manual Figure 2, start the parachain system: first build the relay chain and write a JSON formatted relay chain specification file F that stores information such as the validator key, boot node address, and parachain identifier. spec , which can be expressed as:

[0037] F spec ={K val ,Addr boot ,ID para}

[0038] Among them, K val Indicates the validator key, Addr boot Indicates the boot node address, ID para A unique identifier that represents a parachain in the relay chain.

[0039] This chain specification file is then converted into the SCALE encoding format and used to start three validator nodes VAL to connect the parachain to the relay chain. To ensure that each parachain retains its own unique identifier ID in the relay chain para , you need to connect to the relay chain node and sign a registration transaction containing an identifier. Then start the parachain collector node COL and export the parachain Wasm runtime and genesis state block header Head Genesis In order to register a parachain on the relay chain, the parachain node needs to be connected to the relay chain node again and submit a message containing the identifier ID to the relay chain. para , Parallel Chain Runtime Code val and Head Genesis Parachain registration transaction Tx reg In the following process, the relay chain will continue to track the latest block header of each parachain to ensure smooth data synchronization and interaction between the parachain and the relay chain.

[0040] S2. See the instruction manual Figure 3 , S21 trains the local model: Use Convolutional Neural Networks (CNNs) to train the local model. Input the intrusion detection log data to the input layer, use three convolutional layers to extract features from the input data, and then flatten the multi-dimensional feature map output by the convolutional layer into a one-dimensional vector for connection to the fully connected layer. Finally, use the Sigmoid function to map the output of the fully connected layer to between 0 and 1 so that the output layer can make classification decisions. Use binary cross entropy as the loss function when training the model, use the Adam optimizer for optimization, and monitor the accuracy index.

[0041] S22, upload model: After the client uploads the trained local model gradient to IPFS, it can obtain a string of hash values, that is, the unique CID value of the file, and call the uploadFile() function in the smart contract deployed on the data chain to upload the CID value, the number of training rounds Round and the data set size Datasize. By calling the getFileDetails() function, the specific data of the above three can be queried.

[0042] S3. See the instruction manual Figure 4 , Cross-chain communication: When cross-chain communication is carried out between parallel chains, the cross-chain message passing (XCMP) mechanism is required. When the smart contract on the data chain sends a cross-chain transaction to the contribution chain, the data chain will put the cross-chain transaction into its own message output queue (Egress). The data chain collector collects the transaction and obtains the cross-chain transaction, and submits the transaction and its validity proof to the validator responsible for the data chain on the relay chain. After the validator successfully verifies, it will submit the block header information of the data chain and the information in the Egress queue in the data chain to the relay chain. The relay chain will confirm the block and route the cross-chain transaction by running the hybrid consensus algorithm. The validator on the relay chain will move the corresponding transaction of the contribution chain from the Egress queue of the data chain to the message input queue (Ingress) of the contribution chain. At this time, the smart contract on the contribution chain can obtain the cross-chain transaction issued by the data chain smart contract. According to the transaction information, the contribution chain smart contract will provide the corresponding contribution value stored on the chain, and then package the contribution value into a new transaction and send it back to the data chain in the same way as above.

[0043] S4. Aggregate global model: Aggregate the global model of round t using smart contracts deployed on the data chain There are two cases. First, the smart contract calls the global_gradient() function to judge the round value t. When t=0, the FedAvg() function is called to use the FedAvg algorithm to aggregate the local gradients uploaded to the data chain in the first round. and the amount of local datasets |D i |, and then obtain The global gradient

[0044]

[0045] When t>0, in the calculation Before that, you need to evaluate the quality of each local gradient and call the evaluateModel() function to obtain the quality evaluation value. The specific steps are as follows:

[0046]

[0047]

[0048] Where N represents the set of all clients. represents the global model obtained by aggregating all local models in round t; represents the global model obtained by aggregating all local models except the local model uploaded by client i; and l (t) Respectively represent and The loss value on the validation set; represents the Shapley value of the local model uploaded by client i in the global model in round t-1; represents the marginal loss of the local model uploaded by client i; represents the quality of the local model uploaded by client i in round t, and γ represents the control parameter.

[0049] In obtaining data quality Then, call the aggregation_weight() function to pass the quality Calculate the aggregate weight of client i

[0050]

[0051] Finally, the weighted_global_weighted() function is called to obtain the weighted global gradient

[0052]

[0053] The smart contract will call the data_upload() function to upload the calculated weighted global gradient and the aggregate weight of each user model The data is packaged and uploaded to IPFS, and the returned CID is uploaded to the data chain storage. A consensus is reached between the nodes within the data chain, and the relay chain performs verification.

[0054] S5. Calculate user contribution value: When the smart contract deployed on the contribution chain detects that the global model of this round has been uploaded, it can call the sharpe_ratio() function in the smart contract deployed on the contribution chain to calculate the contribution value. The present invention uses Shapley value to measure the contribution of different local models to this round of training. The Shapley value of the local model uploaded by client i is denoted as φ i , the valuation function is V(i), and the calculation formula of Shapley value is

[0055]

[0056] When calculating the Shapley value, we need to use the valuation function V(·) to calculate the marginal benefit of each local model. The marginal benefit refers to the contribution of the local model to the accuracy of the global model. We can measure the marginal benefit of each local model by comparing the accuracy of the global model before adding it and the accuracy after adding it. Specifically, we use V(S∪i)-V(S) to get the marginal benefit of client i. It represents the number of combinations, that is, the number of ways to combine |S| elements from |N| different elements.

[0057] The cosine similarity between local and global gradients is used to approximate their evaluation functions:

[0058]

[0059] Substituting the valuation function into the Shapley value calculation function, we can get the method for calculating the Shapley value:

[0060]

[0061] The sharpe_ratio() function uses the above formula to calculate the Shapley value of the local model uploaded by client i in the tth round of training The sharpe_ratio() function is called to calculate the contribution value of each local model. Similarly, the data_upload() function is called to package the contribution value into a transaction and upload it to the contribution chain. Finally, a consensus is reached through the contribution chain nodes, which becomes a new block of the contribution chain, verified by the relay chain, and then broadcast to the entire blockchain network.

[0062] Example:

[0063] The UNSW-NB15 dataset is a real-world traffic dataset for intrusion detection problems and is publicly available at: https: / / research.unsw.edu.au / projects / unsw-nb15-dataset

[0064] cpu 12thGenIntel(R)Core(TM)i7-12700H Dataset UNSW-NB15 Ubuntu 20.04.6LTS Python v3.6.15 solidity v0.4.25 fisco v2.9.1 IPFS v0.19.1 substrate V3.0.0

[0065] Global model results using different aggregation algorithms:

[0066] Model Accuracy Loss value Solution of the present invention 0.91656 0.19957 Solution 1 0.87730 0.24019 Solution 2 0.87748 0.24091 Solution 3 0.89897 0.21383

[0067] Solution 1 is to use only Shapley value as aggregation weight to aggregate the global model, Solution 2 is to directly average local model gradients to aggregate the FairAvg of the global model, and Solution 3 is the FedAvg algorithm.

[0068] It can be seen from the above table that by implementing the technical solution of the present invention, the problems of imperfect reward and punishment mechanism, single point failure, and tamperable historical data existing in the prior art are solved, thereby significantly improving the accuracy and significantly reducing the loss value.

[0069] The present invention has been described in detail above through embodiments. It should be noted that the above embodiments are only preferred embodiments of the present invention, not all embodiments of the present invention, and do not constitute any limitation on the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims of this application document. Under the premise of not departing from the spirit of the present invention, the technical solutions or embodiments obtained by equivalent replacement or other various transformations of the technical features of the present invention by ordinary technicians in this field all fall within the scope of protection of the present invention.

Claims

1. A distributed intrusion detection system based on parallel chain and federated learning framework, characterized in that: The intrusion detection system includes a relay chain, a data chain, a contribution chain, a client and an interplanetary file system. The data chain and the contribution chain are two parallel chains. The relay chain is communicatively connected to the data chain and the contribution chain respectively. The data chain is communicatively connected to the contribution chain. The client is communicatively connected to the data chain. The interplanetary file system is communicatively connected to the client, the data chain and the contribution chain respectively.

2. A distributed intrusion detection system based on parallel chain and federated learning framework as claimed in claim 1, characterized in that: The relay chain is mainly composed of validator nodes, which is the central blockchain responsible for handling the consensus mechanism and cross-chain communication of the entire blockchain network; the data chain is mainly composed of data chain collector nodes, which is a parallel chain responsible for storing model parameter data and global models uploaded by users and performing global model aggregation; the contribution chain is mainly composed of contribution chain collector nodes, which is a blockchain responsible for using Shapley value to measure the contribution of each user.

3. A distributed intrusion detection system based on parallel chain and federated learning framework as described in claim 2, characterized in that: When the aggregation round t=0, the global aggregation formula of the data chain is as follows: in, is each local gradient, |D i | is the amount of local data set; when the aggregation round t>0, the global aggregation formula of the data chain is as follows: in, is the aggregate weight of each user model, are local gradients.

4. A distributed intrusion detection system based on parallel chain and federated learning framework as claimed in claim 3, characterized in that: The aggregate weight of each client model The calculation formula is as follows: in, is the data quality of client i.

5. A distributed intrusion detection system based on parallel chain and federated learning framework as described in claim 4, characterized in that: The data quality The calculation formula is as follows: in, represents the marginal loss of the local model uploaded by client i, and γ represents the control parameter.

6. A distributed intrusion detection system based on parallel chain and federated learning framework as claimed in claim 5, characterized in that: The marginal loss The calculation formula is as follows: Where N represents the set of all clients. represents the global model obtained by aggregating all local models in round t; represents the global model obtained by aggregating all local models except the local model uploaded by client i; and l (t) Respectively represent and The loss value on the validation set; represents the Shapley value of the local model uploaded by client i in the global model in round t-1; represents the marginal loss of the local model uploaded by client i; represents the quality of the local model uploaded by client i in round t, and γ represents the control parameter.

7. A distributed intrusion detection system based on parallel chain and federated learning framework as claimed in claim 6, characterized in that: The Shapley value calculation formula is as follows: Where V(·) represents the valuation function, It represents the number of combinations, that is, the number of combinations obtained by combining |S| elements from |N| different elements.

Images