Trusted sharing method of data elements based on deep reinforcement learning and blockchain verification

Abstract

The present invention discloses a trusted sharing method for data elements based on deep reinforcement learning and blockchain verification, comprising: a training requester initiates a training request; an intelligent agent selects a terminal device according to the training request and sends a request message to the selected terminal device; the selected terminal device downloads relevant data from a node of an associated edge server to complete local model training, and after the training is completed, a transaction is sent to the associated edge server according to the local model parameters; the associated edge server verifies the transaction, and reaches a consensus through a federated proof-of-work consensus algorithm after the transaction pool reaches a threshold, thereby completing this training; the intelligent agent updates the state space and reward, and evaluates whether the currently trained global model is qualified. If it is unqualified, a terminal device is reselected to continue training; the present invention uses blockchain to replace the central server of traditional federated learning, thereby solving the trust problem faced by privacy-protected data sharing through traditional federated learning.

Description

Technical Field

[0001] The present invention belongs to the field of data security technology, and specifically relates to a data element trusted sharing method based on deep reinforcement learning and blockchain verification. Background Art

[0002] The convergence of machine learning and the Internet of Things (IoT) is accelerating the advancement of industrial intelligence. Traditional data sharing methods based on machine learning require the collection and storage of large amounts of data in data centers for centralized training. Given the need for privacy protection, it is crucial to leverage data from multiple sources to train models without compromising privacy.

[0003] Federated learning, as a privacy-preserving computing method, has been widely used in recent years in fields such as smart healthcare, the Internet of Vehicles, and the Industrial Internet of Things. Federated learning transforms the data sharing problem into a machine learning problem by sharing models rather than raw data, enabling privacy-preserving data sharing. Traditional federated learning relies on a relatively ideal environment, which can be summarized as follows: 1) The training environment is stable and the clients and central server are trustworthy. 2) The clients participating in the training have a relatively balanced distribution of computing resources, communication resources, and data resources.

[0004] However, due to the competitive relationships among factories in the Industrial IoT environment, there is a lack of trust between them. This lack of trust can lead to malicious clients disrupting the training process, such as by launching poisoning attacks. Furthermore, the high heterogeneity of IoT devices and the synchronous aggregation process of traditional federated learning lead to long training times and low training efficiency.

[0005] Therefore, there is an urgent need for a trusted sharing method of data elements based on deep reinforcement learning and blockchain verification, using blockchain to replace the central server of traditional federated learning, building a trust relationship among all participants, establishing a trusted federated learning system, and solving the trust issues faced by privacy-protected data sharing through traditional federated learning. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a trusted sharing method for data elements based on deep reinforcement learning and blockchain verification, using blockchain to replace the central server of traditional federated learning, build a trust relationship between all participants, establish a trusted federated learning system, and solve the trust issues faced by privacy-protected data sharing through traditional federated learning.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A trusted sharing method for data elements based on deep reinforcement learning and blockchain verification, including:

[0009] Provides a model training framework, which includes the IoT device layer, edge server layer, and blockchain layer from bottom to top. When training based on the model training framework, the entities involved in the training include the training requester, edge server, terminal device, intelligent agent, and authoritative data certification agency.

[0010] During model training, the training requester initiates a training request; the intelligent agent selects a terminal device based on the training request and sends a request message to the selected terminal device; the selected terminal device downloads relevant data from the node of the associated edge server to complete local model training, and after the training is completed, a transaction is sent to the associated edge server based on the local model parameters; the associated edge server verifies the transaction and reaches a consensus through the federated proof-of-work consensus algorithm after the transaction pool reaches the threshold to complete this training; the intelligent agent updates the state space and reward, and evaluates whether the currently trained global model is qualified. If not, the terminal device is reselected to continue training; if qualified, the trusted sharing of data elements is completed based on the qualified global model.

[0011] Furthermore, the specific training process of the model training includes:

[0012] S1. The training requester sends a training request to the agent; the training request contains a request message, the structure of which includes the maximum number of communication rounds. , target accuracy , learning rate , local iteration number , initial model parameters and convergence control parameters ;

[0013] S2, Federated Learning Client Selection Algorithm Based on Deep Reinforcement Learning, the agent selects a group of IoT terminal devices Participate in Round training, including It is the collection of all terminal devices participating in the training. ;

[0014] S3, the agent sends the selected terminal device Send a request message;

[0015] S4. Selected terminal device From the associated edge server The node downloads the blocks of the previous round of training , parsing block Get the latest global model parameters , initialize the training parameters to , and use local datasets Train the model and get Local model parameters for round training ;

[0016] S5. Terminal equipment After training is completed, the local model parameters Packaged into transactions , using the private key Signed and sent to the associated edge server , the formal expression of the transaction is:

[0017]

[0018] in, is the hash value of the transaction, is the signature of the transaction, is the timestamp of the transaction, It is the client's identity identifier, which is a unique identifier generated by an authoritative data authentication agency for each terminal device after the terminal device is registered. It is the local model obtained by the client in this round of training. is the current training communication round number;

[0019] S6. Associate edge server Use a verifiable random function to determine whether you can participate in this round of consensus and become a consensus node; if you can, collect transactions and put them into the transaction pool When the transaction pool size reaches the threshold Afterwards, consensus is reached through the federated proof-of-work consensus algorithm, completing this training;

[0020] S7: The agent updates the state space and reward, and evaluates whether the currently trained global model is qualified. If not, it returns to S2 to continue training.

[0021] Furthermore, the federated proof-of-work consensus algorithm includes consensus node selection based on BLS-VRF and reaching consensus on the new round of training blocks through an interactive protocol between consensus nodes.

[0022] Furthermore, the consensus node selection based on BLS-VRF includes:

[0023] S11. Key generation: Each consensus node generates its own BLS public and private key pair from the prime field Zq when the system is initialized or joins the blockchain network. , , is a large prime number field, is a large prime number;

[0024]

[0025]

[0026] in, represents uniform random selection, and Represents the private key and public key respectively, It's a group The generator of all nodes, the public keys of all nodes need to be made public and stored in the blockchain system so that other nodes and coordinators can access them;

[0027] S12, VRF output generation: the hash value of the previous block As input message , and then the message Hash to group The point on And calculate the signature:

[0028]

[0029] This message will be sent to other consensus nodes for verification in the subsequent process. The message format is:

[0030]

[0031] S13. Verify VRF output: Node Received node After the VRF message, the node Public key ,information , VRF output and VRF value As the Verify_VRF function Verify that the following equation holds true:

[0032]

[0033] If the equation holds, prove It is a valid signature of H(m), and the verification is successful;

[0034] S14. Sorting and selecting consensus nodes: After all consensus nodes have completed verification, they will sort all legal VRF outputs and select the top K nodes to form the consensus committee for this round:

[0035]

[0036] in, Indicates the consensus committee for this round.

[0037] Furthermore, consensus is reached on the new round of training blocks through an interactive protocol between consensus nodes, including:

[0038] S21. Parallel verification: The edge server uses the verification dataset provided by the training requester Verify local model updates included in transactions;

[0039] S22, Packing: The number of transactions in the transaction pool reaches the specified threshold When the miners package the verified transactions, they use the aggregation algorithm to calculate the global model and sign them together to build the Merkle tree; after testing the accuracy of the global model, the accuracy value Accuracy is written into the block header. At the same time, the block node number, the consensus node committee selected by VRF for this round of training, and the current training round number are also written into the block header;

[0040] S23, Proposal: The first node to complete the above work will initiate a consensus proposal ; Broadcast a message and a new block to other consensus nodes, where The structure is:

[0041]

[0042] in, Indicates that the consensus phase is the proposal phase, Indicates the node number that initiated the proposal, Indicates the signature of the current node, Indicates the block that requires consensus;

[0043] S24. Preliminary Verification: After receiving a new block, other nodes will suspend the block generation process, parse the new block and extract the accuracy field in the block header, the consensus node committee set and the global model leaf node of the Merkle tree in the block body. The accuracy of the global model is verified through a standard verification data set and compared with the accuracy in the block header. If the difference between the two accuracy rates is within a certain range, the proposal is accepted and sent. Message to all other consensus committee nodes; The structure of the message is:

[0044]

[0045] in, Indicates that you are currently in stage, Indicates the current sending The node number, Indicates the node number that initiated the proposal, Indicates the signature of the current node, Indicates the current training round number;

[0046] S25, re-verification: Each consensus node counts the received Number of messages, if the node receives The number of messages exceeds , the consensus node sends Message to all other nodes; where, is the maximum number of Byzantine nodes, The structure of the message is:

[0047]

[0048] in, Indicates that you are currently in stage, Indicates the current sending The node number, Indicates the node number that initiated the proposal, Indicates the signature of the current node, Indicates the current training round number;

[0049] S26, Block Addition: If the node receives The number of messages exceeds , it is considered that the new block has reached consensus, the node adds the new block to the blockchain, and the Tth round of training ends.

[0050] Furthermore, a method for trusted sharing of data elements based on deep reinforcement learning and blockchain verification also includes: a federated learning client selection algorithm based on deep reinforcement learning models each round of the federated learning process as a Markov decision process during the selection process to obtain an optimization problem; and uses the DDQN framework to solve the optimization problem to complete the selection of the terminal device.

[0051] Furthermore, the formal expression of the Markov decision process is:

[0052]

[0053] in, represents the state space, represents the action space, represents the state transition probability caused by action A in state S, Represents the reward after executing action A;

[0054] Specifically expressed as:

[0055]

[0056] The dimension of the state space is , Indicates the number of IoT devices participating in the training, express The number of dimensions after dimensionality reduction, Indicates the acquired delay depth;

[0057] Specifically expressed as:

[0058]

[0059] in, is the balance factor, is a constant parameter.

[0060] The beneficial effects of the present invention are:

[0061] 1. Using blockchain to replace the central server of traditional federated learning, build trust relationships among all participants, establish a trusted federated learning system, and solve the trust issues faced by privacy-preserving data sharing through traditional federated learning;

[0062] 2. A federated proof-of-work consensus algorithm adapted for federated learning was proposed, transforming the workload required to reach consensus into work that is beneficial to federated learning. A specific block structure and transaction structure were designed to support the proposed consensus algorithm, addressing the issue that current blockchain federated learning solutions lack a deep integration of the two technologies.

[0063] 3. Taking into account the data heterogeneity, computing heterogeneity, and communication heterogeneity that exist in real IoT scenarios, and based on the principle of maximizing client privacy, a deep reinforcement learning-based method is proposed to select the set of client devices for each training round, thereby reducing the time required to achieve the target accuracy.

[0064] 4. In order to cater to different heterogeneous scenarios, a reward function for different heterogeneous scenarios is designed. According to the different degrees of resource heterogeneity, the weight is dynamically configured through the balance factor to flexibly adapt to different heterogeneous scenarios.

[0065] Other advantages, objectives, and features of the present invention will be described in the following description and will be apparent to those skilled in the art to some extent, or may be taught by those skilled in the art from the practice of the present invention. The purposes and other advantages of the present invention may be realized and obtained through the structures particularly pointed out in the written description and the accompanying drawings.

[0066] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0068] Figure 1 Schematic diagram of a model training process for a data element trusted sharing method based on deep reinforcement learning and blockchain verification in an embodiment of the present invention;

[0069] Figure 2 This is a diagram showing the overall framework of model training in a method for trusted sharing of data elements based on deep reinforcement learning and blockchain verification in an embodiment of the present invention;

[0070] Figure 3 Schematic diagram of an algorithm for a model training process in a method for trusted sharing of data elements based on deep reinforcement learning and blockchain verification in an embodiment of the present invention;

[0071] Figure 4 This is a block structure diagram of a data element trusted sharing method based on deep reinforcement learning and blockchain verification in an embodiment of the present invention;

[0072] Figure 5 A state space structure diagram of a data element trusted sharing method based on deep reinforcement learning and blockchain verification in an embodiment of the present invention;

[0073] Figure 6 This is a schematic diagram of DDQN training in a data element trusted sharing method based on deep reinforcement learning and blockchain verification in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0075] like Figure 1 As shown, the present invention proposes a data element trusted sharing method based on deep reinforcement learning and blockchain verification, including:

[0076] Provides a model training framework, which includes the IoT device layer, edge server layer, and blockchain layer from bottom to top. When training based on the model training framework, the entities involved in the training include the training requester, edge server, terminal device, intelligent agent, and authoritative data certification agency.

[0077] During model training, the training requester initiates a training request; the agent selects a terminal device based on the training request and sends a request message to the selected terminal device; the selected terminal device downloads relevant data from the node of the associated edge server to complete local model training, and after training is completed, sends a transaction to the associated edge server based on the local model parameters; the associated edge server verifies the transaction and reaches a consensus through the federated proof-of-work consensus algorithm after the transaction pool reaches the threshold, completing this training; the agent updates the state space and reward, and evaluates whether the currently trained global model is qualified. If not, the agent reselects a terminal device to continue training; if qualified, the trusted sharing of data elements is completed based on the qualified global model;

[0078] The working principle of the above technical solution is as follows: Blockchain is widely used in federated learning as a bridge for building trust. However, the common point of current blockchain-based federated learning work is that blockchain and federated learning are relatively independent, and the two technologies have not been deeply integrated from the bottom up. On the other hand, in the industrial Internet of Things environment, due to the existence of data heterogeneity, computing heterogeneity, and communication heterogeneity, federated learning often faces the problem of inefficiency (i.e., it takes a long time to achieve the target accuracy). To address this problem, the existing technology discloses CN-118734990A, a decentralized learning method for federated learning models based on IPFS and blockchain. It uses the global parameter hash link on the blockchain to obtain the global model parameters of the federated learning model from the InterPlanetary File System; initializes the local model according to the global model parameters, and trains the local model. Update the client's local parameters and the average round reward of the local model; upload the client's local parameters and the average round reward of the local model to the InterPlanetary File System for storage and store the corresponding local parameter hash link in the blockchain; any blockchain node obtains the global model parameters from the InterPlanetary File System based on several client local parameters and the average round reward of the local model; determine whether to end training or proceed to the next round of training based on the updated global model parameters. However, it does not utilize the close integration of blockchain and federated learning, but only combines the InterPlanetary File System to use the blockchain as a data exchange and storage platform. In contrast, this application takes into account the heterogeneity of the three resources, improves training efficiency from the perspective of client selection, and uses deep reinforcement learning methods to optimize the clients participating in each round of training, reducing the time spent on reaching the target accuracy, and improving training efficiency overall.

[0079] Specifically, the overall architecture of the system model of this application is as follows Figure 2 As shown, the overall architecture is divided into IoT device layer, edge server layer, and blockchain layer;

[0080] The IoT device layer is all entities participating in federated learning training, including various sensors and smart devices used for industrial production in the Industrial Internet of Things;

[0081] The edge server layer consists of edge servers and communication base stations in each factory, responsible for maintaining the blockchain network. The edge servers have strong communication and computing capabilities.

[0082] The blockchain layer replaces the central server of traditional federated learning. Its role is to coordinate the training process and complete the aggregation of the global model through the consensus algorithm.

[0083] In our framework, different from the previous block structure, we propose a new blockchain architecture in combination with federated learning;

[0084] The entities involved in the training are the training requester, edge server, terminal device, intelligent agent, and authoritative data certification agency;

[0085] The training requester can be any factory or entity with data sharing needs. It is responsible for initiating the data sharing training request and needs to provide a standardized verification data set.

[0086] As nodes in the blockchain network, edge servers are primarily responsible for verifying the legitimacy of local model updates, aggregating new global models, packaging training blocks, and maintaining training data for the consortium blockchain network. Each factory runs an edge server, corresponding to a node in the blockchain.

[0087] Terminal devices are clients in federated learning, but they have weak computing and communication capabilities. Each terminal device is associated with an edge server in its own factory. The terminal device collects and stores industrial production data locally, trains models using local data, and uploads the trained local models to the associated edge server.

[0088] The agent is responsible for scheduling the clients participating in each round of federated learning. It is essentially a reinforcement learning network with intelligent decision-making capabilities. It uses the state space and reinforcement learning network to intelligently select the clients participating in each round of training.

[0089] Authoritative certification bodies are responsible for registering and authenticating edge servers and IoT devices in the industrial IoT. Edge servers must be certified by an authoritative data certification body and obtain a certificate issued by the body before they can become blockchain nodes to maintain training data. At the same time, IoT devices must also obtain an authorization certificate before they can package the trained local model into a transaction and send it to the blockchain node.

[0090] The workflow of the method proposed in this application is as follows Figure 1 As shown, the algorithm process is as follows Figure 3 As shown in the figure, the specific training process of the model training includes:

[0091] S1. Training Requester Towards the Agent Send a training request; the training request contains a request message, the structure of the request message includes the maximum number of communication rounds , target accuracy , learning rate , local iteration number , initial model parameters and convergence control parameters etc.; the formal representation of the training request is:

[0092]

[0093] S2, Federated Learning Client Selection Algorithm Based on Deep Reinforcement Learning, Agent Select a set of quantities IoT terminal devices Participate in Round training, including It is the collection of all terminal devices participating in the training. ;

[0094]

[0095] S3, Agent To the selected terminal device Send a request message, including the number of local iterations , initial model parameters , learning rate And other related training parameters;

[0096] S4. Selected terminal device From the associated edge server The node downloads the blocks of the previous round of training , parsing block Get the latest global model parameters , initialize the training parameters to , and use local datasets Train the model and get Local model parameters for round training ;

[0097] S5. Terminal equipment After training is completed, the local model parameters Packaged into transactions , using the private key Signed and sent to the associated edge server , the formal expression of the transaction is:

[0098]

[0099] in, is the hash value of the transaction, is the signature of the transaction, is the timestamp of the transaction, It is the client's identity identifier, which is a unique identifier generated by an authoritative data authentication agency for each terminal device after the terminal device is registered. It is the local model obtained by the client in this round of training. is the current training communication round number;

[0100] S6. Associate edge server Use the Verifiable Random Function (VRF) to determine whether you can participate in this round of consensus and become a consensus node; if you can, collect transactions and put them into the transaction pool When the transaction pool size reaches the threshold Afterwards, consensus is reached through the federated proof-of-work consensus algorithm, completing this training;

[0101] S7: The agent updates the state space and reward, and evaluates whether the currently trained global model is qualified. If not, it returns to S2 to continue training.

[0102] The beneficial effects of the above technical solution are as follows: this application proposes a trusted federated learning architecture that integrates blockchain and federated learning, promotes trusted privacy-protected data sharing between mutually distrusting entities, and makes the training process decentralized, traceable, and controllable; deeply integrates the two technologies of blockchain and federated learning, and proposes a federated proof-of-work consensus algorithm, which converts the computing power consumed by the traditional proof-of-work of blockchain into work that is beneficial to federated learning, and realizes energy recovery; in view of the heterogeneity of data, computing power, and communication of the devices participating in the training, from the perspective of client selection, the deep reinforcement learning method is used to intelligently select the devices participating in each round of training, reducing the time consumed to achieve the predetermined accuracy in heterogeneous scenarios; at the same time, the state space we designed does not need to know other information of the client, thereby protecting the privacy of the client.

[0103] In one embodiment, the federated proof-of-work consensus algorithm includes BLS-VRF-based consensus node selection and reaching consensus on the next round of training blocks through an interactive protocol between consensus nodes, including:

[0104] The working principle and beneficial effects of the above technical solution are as follows: To address the trust issues existing in federated learning for the Industrial Internet of Things, this paper proposes a blockchain-based federated learning architecture. Furthermore, we use blockchain consensus as a starting point to propose our federated proof-of-work consensus algorithm. Our federated proof-of-work consensus algorithm is described below. The first step is the selection of consensus nodes based on BLS-VRF, and the second step is reaching consensus on the next round of training blocks through an interactive protocol between consensus nodes. These two steps will be described in turn below.

[0105] 1. In a blockchain network, if all nodes participate in the consensus process, communication overhead will be very high. Selecting a small number of nodes as consensus nodes can significantly reduce communication overhead, but this can also easily lead to centralization and security issues. Therefore, when selecting a group of nodes to form a consensus group, security and fairness must be considered. The main feature of VRF is its ability to generate unpredictable and verifiable random numbers, which can be publicly verified after generation. This feature is very suitable for scenarios such as blockchains that require random node selection or random number generation. This paper proposes to address the security and fairness issues faced by consensus node selection by adopting a BLS-Based VRF method to select the consensus node group. By combining the efficiency and security of BLS signatures, BLS-Based VRF provides a powerful tool for random number generation and verification. It not only improves the efficiency of the verification process and simplifies key management, but also ensures the decentralization and unbiasedness of the generation process. The main workflow is as follows:

[0106] S11. Key generation: Each consensus node generates its own BLS public and private key pair from the prime field Zq when the system is initialized or joins the blockchain network. , , is a large prime number field, is a large prime number;

[0107]

[0108]

[0109] in, represents uniform random selection, and Represents the private key and public key respectively, It's a group The generator of all nodes, the public keys of all nodes need to be made public and stored in the blockchain system so that other nodes and coordinators can access them;

[0110] S12, VRF output generation: the hash value of the previous block As input message , and then the message Hash to group The point on And calculate the signature:

[0111]

[0112] This message will be sent to other consensus nodes for verification in the subsequent process. The message format is:

[0113]

[0114] S13. Verify VRF output: Node Received Node After the VRF message, the node Public key ,information , VRF output and VRF value As the Verify_VRF function Verify that the following equation holds true:

[0115]

[0116] If the equation holds, prove It is a valid signature of H(m), and the verification is successful;

[0117] S14. Sorting and selecting consensus nodes: After all consensus nodes have completed verification, they will sort all legal VRF outputs and select the top K nodes to form the consensus committee for this round:

[0118]

[0119] in, Indicates the consensus committee for this round;

[0120] Second, after selecting the consensus node committee for round t of training using the algorithm above, in order to achieve global model consistency, consensus must be reached on the global model of round t training, that is, consensus on the blocks produced by round t training. We designed a competitive solution. Traditional competitive algorithms solve hash puzzles, but this computation is meaningless for federated learning. We define meaningful work for federated learning as verifying the legitimacy of local model updates and aggregating the global model to achieve energy recovery.

[0121]

[0122] The process of nodes reaching consensus becomes the following steps:

[0123] S21. Parallel verification: The edge server uses the verification dataset provided by the training requester Verify local model updates contained in transactions. Unlike traditional blockchains with digital currency attributes, transactions in our proposed blockchain structure do not involve currency transactions. Therefore, there is no need to consider dependencies between transactions. Local models can be verified in parallel, greatly improving verification speed.

[0124] S22, Packing: The number of transactions in the transaction pool reaches the specified threshold When the miners package the verified transactions, they use the aggregation algorithm to calculate the global model and sign them together to build the Merkle tree; after testing the accuracy of the global model, the accuracy value Accuracy is written into the block header. At the same time, the block node number, the consensus node committee selected by VRF for this round of training, and the current training round number are also written into the block header;

[0125] S23, Proposal: The first node to complete the above work will initiate a consensus proposal ; Broadcast a message and a new block to other consensus nodes, where The structure is:

[0126]

[0127] in, Indicates that the consensus phase is the proposal phase, Indicates the node number that initiated the proposal, Indicates the signature of the current node, Indicates the block that requires consensus;

[0128] S24, Preliminary Verification: After receiving the new block, other nodes will suspend the block generation process, parse the new block and extract the accuracy field in the block header, the consensus node committee set and the global model leaf node of the Merkle tree in the block body (block structure as shown in the following example). Figure 4 As shown in the figure), the accuracy of the global model is verified through a standard verification data set and compared with the accuracy in the block header. If the difference between the two accuracy rates is within a certain range, the proposal is accepted and sent. Message to all other consensus committee nodes; The structure of the message is:

[0129]

[0130] in, Indicates that you are currently in stage, Indicates the current sending The node number, Indicates the node number that initiated the proposal, Indicates the signature of the current node, Indicates the current training round number;

[0131] S25, re-verification: Each consensus node counts the received Number of messages, if the node receives The number of messages exceeds , the consensus node sends Message to all other nodes; where, is the maximum number of Byzantine nodes, The structure of the message is:

[0132]

[0133] in, Indicates that you are currently in stage, Indicates the current sending The node number, Indicates the node number that initiated the proposal, Indicates the signature of the current node, Indicates the current training round number;

[0134] S26, Block Addition: If the node receives The number of messages exceeds , it is considered that the new block has reached consensus, the node adds the new block to the blockchain, and the Tth round of training ends;

[0135] After completing the above steps, the global model of the tth round of training reaches consensus among the consensus nodes. All information contained in this round of training, including local models, global models, current training round number, current model version, etc., will be included in the new block. Based on the characteristics of blockchain, the training process can be traced throughout its life cycle.

[0136] In one embodiment, a method for trusted sharing of data elements based on deep reinforcement learning and blockchain verification further includes: during the selection process of a federated learning client selection algorithm based on deep reinforcement learning, modeling each round of the federated learning process as a Markov decision process to obtain an optimization problem; and using a DDQN framework to solve the optimization problem to complete the selection of a terminal device;

[0137] The working principle of the above technical solution is as follows: To address different heterogeneous scenarios, this solution designs reward functions for different heterogeneous scenarios. According to the different degrees of resource heterogeneity, the solution dynamically configures weights through balancing factors to flexibly adapt to different heterogeneous scenarios. In order to maximize the efficiency of federated learning in the context of the Internet of Things and address the inefficiency of privacy-preserving data sharing through federated learning in the open and heterogeneous industrial Internet of Things environment, the node selection problem is modeled. In essence, this is a combinatorial optimization problem. Each round of federated learning can be modeled as a Markov decision process (MDP). The formal expression is:

[0138]

[0139] in, represents the state space, represents the action space, represents the state transition probability caused by action A in state S, represents the reward after executing action A. Our goal is to maximize the efficiency of federated learning in the IoT environment by simultaneously considering the heterogeneous computing, data, and communication resources of devices, that is, to reduce the time required to achieve a fixed accuracy. The state space, action space, and reward function of the problem are modeled below.

[0140] A. State Space

[0141] The state space we designed needs to reflect the characteristics of the client's data resources, computing resources, and communication resources. The state space design is based on the principle of maximizing user privacy. We cannot directly obtain the client's data distribution and device resources. We use the local model parameters and global model parameters after PCA dimensionality reduction as the characteristics of the data distribution. This is because the data distribution after dimensionality reduction can still reflect the original data distribution. At the same time, the benefit of dimensionality reduction is that it can greatly reduce the dimensionality of the state space and reduce computational complexity.

[0142] At the same time, we use computing time and communication time to measure the computing power and communication power of the device respectively; our state space is represented as:

[0143]

[0144] The dimension of the state space is , Indicates the number of IoT devices participating in the training, express The number of dimensions after dimensionality reduction, Indicates the acquired delay depth; for example When , the three most recently collected delay data will be retained as the input of the model, where the specific structure of the state space is as follows Figure 5 As shown;

[0145] B. Action Space

[0146] Given the above state space, the agent needs to decide whether each device participates in the tth round of training, where represents no selection, 1 represents selection, and we define the action space as:

[0147]

[0148] C. Reward Function

[0149] The reward function in reinforcement learning is a key component that guides the agent to learn and perform tasks in an environment. Incorrect or inappropriate reward functions can lead to many problems, such as over-exploration, learning instability, and non-convergence. Therefore, formulating a reasonable reward function is challenging. We try to define the reward function for the client selection problem as follows:

[0150]

[0151] The reward function is explained as follows: the reward function needs to reflect our optimization goal, which is to minimize the time taken to reach a given accuracy. First, our goal is to make the client set selected by the agent quickly contribute to the improvement of accuracy, so as to reach the target accuracy in fewer communication rounds. If the accuracy of the selected client is lower than that of the previous round, we will provide a negative reward. The second term takes into account the problem of alleviating computation and communication delays. We seek a balance between accuracy and delay to achieve our optimization goal. Here is a balancing factor. Furthermore, given that accuracy gains tend to slow during training, we need to amplify the impact of data utility and minimize the impact of latency. Therefore, we utilize exponential and logarithmic functions to achieve our goal. More specifically, as the number of training rounds increases, in the later stages of training, to maximize the cumulative reward, we amplify one factor and slow down the growth of the second factor, thereby further improving our cumulative reward.

[0152] Furthermore, our reward function can be balanced by the factor and λ parameters dynamically adapt to different heterogeneous scenarios. For example, if the training scenario only considers data heterogeneity, then we will If λ is set to 0, the agent will focus on quickly improving the accuracy and reducing the number of communication rounds required to achieve the target accuracy, thereby improving training efficiency. If the training scenario only considers computing heterogeneity and communication heterogeneity, then we can set λ to 0. In this case, the agent will focus on selecting devices with smaller computing and communication delays to participate in the training in order to maximize the cumulative reward, thereby improving training efficiency. In particular, for the case where the three heterogeneities coexist, by adjusting Parameters allow the agent to achieve a balance between quickly improving accuracy to reduce the number of training rounds and selecting devices with smaller computing and communication delays to shorten the training time per round, thereby comprehensively improving training efficiency.

[0153] D. Optimization goals

[0154] Our optimization goal is to maximize the efficiency of federated learning by considering the heterogeneity of data resources, computing resources, and communication resources among smart devices in a heterogeneous industrial IoT environment. We define efficiency as the time it takes to achieve a predetermined accuracy. The optimization goal can be formally expressed as:

[0155]

[0156]

[0157]

[0158]

[0159] in, is the set of clients selected in round t, M is the number of smart devices selected for training in each round, acct is the accuracy of the global model after round t of training, and acctar is the target accuracy. R is the number of communication rounds required to achieve the target accuracy, and r is the maximum number of communication rounds, which must satisfy the constraint R ≤ r. Based on the above modeling of the state space, action space, and reward function, as well as the optimization objective, we will now introduce how to use the DDQN framework to optimize and solve the optimization objective after problem modeling.

[0160] E.DDQN Framework

[0161] We use the DDQN framework to train the agent. The training process is as follows Figure 6 As shown, the interaction process is as follows:

[0162] DDQN (Double Deep Q-Networks) is an enhanced Deep Q-Network (DQN) algorithm for reinforcement learning. It mainly solves the overestimation problem in DQN. DDQN reduces this overestimation phenomenon by separating action selection and Q-value updates, thereby improving learning stability and performance. The agent is trained to maximize the expectation of the cumulative discounted reward given by the following formula:

[0163]

[0164] The following is the framework and explanation of DDQN; the training process of DDQN is:

[0165] 1) Initialize the network: Initialize two neural networks with the same structure: the online network (Q) and the target network (Q'); initialize the experience replay buffer to store experience samples (state, action, reward, next state); set the maximum capacity of the experience replay buffer; set the learning rate and discount factor ( ), target network update frequency and other hyperparameters;

[0166] 2) Select action: From current state Start using Greedy strategy selects actions The specific method is based on the probability Choose a random action with probability Choose such that Q( ) the biggest movement;

[0167] 3) Execute action: Execute action That is, conduct a round of federated learning training and observe the rewards and the next state ;

[0168] 4)Store experience: store experience samples ( ) is stored in the experience replay buffer D;

[0169] 5) Experience replay: Randomly sample a small batch of experience samples from the experience replay buffer ;

[0170] 6) Calculate the target Q value: For each sample ( ) Calculate the target Q value:

[0171]

[0172] in, are the parameters of the online network, are the parameters of the target network;

[0173] 7) Update the online network: Use the mean square error loss function to update the parameters of the online network:

[0174]

[0175] 8) Update the target network: every fixed time step , the parameters of the online network Parameters copied to the target network middle;

[0176] The beneficial effects of the above technical solution are: according to the above technical solution, the state space designed in this application does not need to know other information of the client, thereby protecting the privacy of the client. At the same time, the reward function designed in this application can flexibly adapt to the client selection problem in different heterogeneous scenarios by adjusting the balance factor.

[0177] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A data element trusted sharing method based on deep reinforcement learning and blockchain verification, characterized by: include: Provides a model training framework, which includes the IoT device layer, edge server layer, and blockchain layer from bottom to top. When training based on the model training framework, the entities involved in the training include the training requester, edge server, terminal device, intelligent agent, and authoritative data certification agency. During model training, the training requester initiates a training request; the agent selects a terminal device based on the training request and sends a request message to the selected terminal device; the selected terminal device downloads relevant data from the node of the associated edge server to complete local model training, and after training is completed, sends a transaction to the associated edge server based on the local model parameters; the associated edge server verifies the transaction and reaches a consensus through the federated proof-of-work consensus algorithm after the transaction pool reaches the threshold, completing this training; the agent updates the state space and reward, and evaluates whether the currently trained global model is qualified. If not, the agent reselects a terminal device to continue training; if qualified, the trusted sharing of data elements is completed based on the qualified global model; The specific training process of the model training includes: S1. The training requester sends a training request to the agent; the training request contains a request message, the structure of which includes the maximum number of communication rounds. , target accuracy , learning rate , local iteration number , initial model parameters and convergence control parameters ; S2, Federated Learning Client Selection Algorithm Based on Deep Reinforcement Learning, the agent selects a group of IoT terminal devices Participate in Round training, including It is the collection of all terminal devices participating in the training. ; S3, the agent sends the selected terminal device Send a request message; S4. Selected terminal device From the associated edge server The node downloads the blocks of the previous round of training , parsing block Get the latest global model parameters , initialize the training parameters to , and use local datasets Train the model and get Local model parameters for round training ; S5. Terminal equipment After training is completed, the local model parameters Packaged into transactions , using the private key Signed and sent to the associated edge server , the formal expression of the transaction is: in, is the hash value of the transaction, is the signature of the transaction, is the timestamp of the transaction, It is the client's identity identifier, which is a unique identifier generated by an authoritative data authentication agency for each terminal device after the terminal device is registered. It is the local model obtained by the client in this round of training. is the current training communication round number; S6. Associate edge server Use a verifiable random function to determine whether you can participate in this round of consensus and become a consensus node; if you can, collect transactions and put them into the transaction pool When the transaction pool size reaches the threshold Afterwards, consensus is reached through the federated proof-of-work consensus algorithm, completing this training; S7: The agent updates the state space and reward, and evaluates whether the currently trained global model is qualified. If not, it returns to S2 to continue training.

2. A data element trusted sharing method based on deep reinforcement learning and blockchain verification according to claim 1, characterized in that: The federated proof-of-work consensus algorithm includes consensus node selection based on BLS-VRF and reaching consensus on the new round of training blocks through an interactive protocol between consensus nodes.

3. A data element trusted sharing method based on deep reinforcement learning and blockchain verification according to claim 2, characterized in that: Consensus node selection based on BLS-VRF, including: S11. Key generation: Each consensus node generates its own BLS public and private key pair from the prime field Zq when the system is initialized or joins the blockchain network. , , is a large prime number field, is a large prime number; in, represents uniform random selection, and Represents the private key and public key respectively, It's a group The generator of all nodes, the public keys of all nodes need to be made public and stored in the blockchain system so that other nodes and coordinators can access them; S12, VRF output generation: the hash value of the previous block As input message , and then the message Hash to group The point on And calculate the signature: This message will be sent to other consensus nodes for verification in the subsequent process. The message format is: S13. Verify VRF output: Node Received node After the VRF message, the node Public key ,information , VRF output and VRF value As the Verify_VRF function Verify that the following equation holds true: If the equation holds, prove It is a valid signature of H(m), and the verification is successful; S14. Sorting and selecting consensus nodes: After all consensus nodes have completed verification, they will sort all legal VRF outputs and select the top K nodes to form the consensus committee for this round: in, Indicates the consensus committee for this round.

4. The method for trusted sharing of data elements based on deep reinforcement learning and blockchain verification according to claim 2 is characterized in that: A consensus is reached on the new round of training blocks through an interactive protocol between consensus nodes, including: S21. Parallel verification: The edge server uses the verification dataset provided by the training requester Verify local model updates included in transactions; S22, Packing: The number of transactions in the transaction pool reaches the specified threshold When the miners package the verified transactions, they use the aggregation algorithm to calculate the global model and sign them together to build the Merkle tree; after testing the accuracy of the global model, the accuracy value Accuracy is written into the block header. At the same time, the block node number, the consensus node committee selected by VRF for this round of training, and the current training round number are also written into the block header; S23, Proposal: The first node to complete the above work will initiate a consensus proposal ; Broadcast a message and a new block to other consensus nodes, where The structure is: in, Indicates that the consensus phase is the proposal phase, Indicates the node number that initiated the proposal, Indicates the signature of the current node, Indicates the block that requires consensus; S24. Preliminary Verification: After receiving a new block, other nodes will suspend the block generation process, parse the new block and extract the accuracy field in the block header, the consensus node committee set and the global model leaf node of the Merkle tree in the block body. The accuracy of the global model is verified through a standard verification data set and compared with the accuracy in the block header. If the difference between the two accuracy rates is within a certain range, the proposal is accepted and sent. Message to all other consensus committee nodes; The structure of the message is: in, Indicates that you are currently in stage, Indicates the current sending The node number, Indicates the node number that initiated the proposal, Indicates the signature of the current node, Indicates the current training round number; S25, re-verification: Each consensus node counts the received Number of messages, if the node receives The number of messages exceeds , the consensus node sends Message to all other nodes; where, is the maximum number of Byzantine nodes, The structure of the message is: in, Indicates that you are currently in stage, Indicates the current sending The node number, Indicates the node number that initiated the proposal, Indicates the signature of the current node, Indicates the current training round number; S26, Block Addition: If the node receives The number of messages exceeds , it is considered that the new block has reached consensus, the node adds the new block to the blockchain, and the Tth round of training ends.

5. The method for trusted sharing of data elements based on deep reinforcement learning and blockchain verification according to claim 1 is characterized in that: During the selection process, the federated learning client selection algorithm based on deep reinforcement learning models each round of federated learning process as a Markov decision process to obtain an optimization problem; and uses the DDQN framework to solve the optimization problem to complete the selection of terminal devices.

6. A data element trusted sharing method based on deep reinforcement learning and blockchain verification according to claim 5, characterized in that: The formal expression of the Markov decision process is: in, represents the state space, represents the action space, represents the state transition probability caused by action A in state S, Represents the reward after executing action A; Specifically expressed as: The dimension of the state space is , Indicates the number of IoT devices participating in the training, express The number of dimensions after dimensionality reduction, Indicates the acquired delay depth; Specifically expressed as: in, is the balance factor, is a constant parameter.

Images