Federal learning system based on fisco group mechanism

Abstract

The application discloses a federal learning system based on a FISCO group mechanism, which comprises a supervision node group and multiple levels of client groups, for any client, obtains and updates a local model according to a global model obtained through initialization or a last learning iteration; for any client group, based on a practical Byzantine fault tolerance consensus mechanism and a malicious node screening mechanism based on cosine similarity, an updated local model is obtained; based on a blockchain or a supervision node, an updated global model is aggregated; until a preset federal learning convergence condition is met, otherwise, the next round of federal learning is entered. The application divides different levels of client groups, guarantees the privacy of the blockchain system while retaining security and decentralization, and the introduction of the supervision node can not only play a penetrating supervision across agencies, but also greatly improve the efficiency of model aggregation.

Description

Technical Field

[0001] This invention relates to a federated learning system based on the FISCO group mechanism, belonging to the field of federated learning technology. Background Technology

[0002] With the rapid development of next-generation information technologies such as big data and artificial intelligence, machine learning has been widely applied. Machine learning requires massive amounts of data for training; however, traditional machine learning typically requires all data to be centralized on a single server for training. Due to data security and privacy concerns, various organizations and individuals are unwilling to share data, resulting in the problem of data silos.

[0003] Federated learning offers a viable solution to the data silo problem. It is a novel machine learning paradigm that enables institutions to collaboratively train a shared global model while ensuring data remains within their local systems, thus effectively addressing the data silo problem while preserving privacy.

[0004] However, traditional federated learning architectures require a central server to collect and aggregate models or updates from all clients. This central server is crucial for traditional federated learning, as it possesses model information from all clients and can determine the direction of global model updates. However, this centralized federated learning architecture has two problems: first, the trustworthiness of the central parameter server is an issue; all operations of the central server are invisible and opaque to the clients, who cannot verify whether their local models have been correctly received by the server; second, all client models or updates must be uploaded to the server for computation, placing extremely high demands on the server's communication and computing capabilities, and if the server crashes, the entire federated learning process will cease. Therefore, the central server is prone to single points of failure or lack of neutrality (leaning towards certain devices), which can significantly affect the willingness of devices to participate in federated learning. Furthermore, due to legal, security, and competitive reasons, it may be impossible to select a trustworthy third party that satisfies every client as the central parameter server. Therefore, designing a decentralized federated learning architecture to improve the trustworthiness of federated learning becomes extremely important.

[0005] Based on the above considerations, some researchers have proposed a decentralized federated learning architecture based on blockchain. Blockchain is a serverless, peer-to-peer federated learning architecture based on a P2P architecture, where all clients can interact directly with each other. However, in this decentralized federated learning framework, using blockchain for model storage and update exchange leads to excessive communication and computational overhead. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a federated learning system based on the FISCO group mechanism. First, by introducing the multi-group architecture of FISCO BCOS, the entire federated learning system is divided into client groups at different levels, allowing for a divide-and-conquer approach that maintains security and decentralization while enabling expansion. Transaction processing, data storage, and block consensus between groups are isolated from each other, ensuring the privacy of the blockchain system. Second, the introduction of regulatory nodes not only enables cross-institutional, penetrating supervision but also greatly improves the efficiency of model aggregation.

[0007] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0008] This invention discloses a federated learning system based on the FISCO group mechanism, including a supervisory node group and multiple levels of client groups. A client group includes multiple clients, and a client group at a certain level has multiple client groups at the next lower level. The supervisory node group includes multiple levels of supervisory nodes, and a supervisory node is used to monitor one or more client groups at the same level.

[0009] For any client, the method is to obtain the initialized or previously learned global model; based on the global model, to perform local training and update of the client's local model using the stochastic gradient descent algorithm, and to obtain the updated local model of the client by calculating the preset τ-inexact solution.

[0010] For any client group, the updated local model of the client group is obtained based on the updated local models of multiple clients in the client group, using a practical Byzantine fault-tolerant consensus mechanism and a malicious node filtering mechanism based on cosine similarity.

[0011] Based on blockchain or a supervisory node, the updated global model is obtained by aggregating the updated local models of all client groups. If the updated global model satisfies the preset federated learning convergence condition, the federated learning of the global model based on the FISCO group mechanism is completed; otherwise, the next round of federated learning begins.

[0012] Furthermore, obtaining the updated local model on the client side includes the following steps:

[0013] Get the initialized or global model obtained from the previous learning iteration;

[0014] Based on the global model, the local loss of the client is calculated using the stochastic gradient descent algorithm, and then the local update of the client is calculated.

[0015] Based on the client's local update, the updated local model of the client is obtained by calculating the preset τ-inexact solution.

[0016] Furthermore, the expression for the local loss is:

[0017]

[0018] In the formula, This represents the local loss of client i; Represents the local model of client i; w t Represents the global model; Representation Model The loss on the data distribution of client i; β represents the strength of the constraints imposed on the local model; ||·|| 2 Represents the 2-norm;

[0019] The expression for the local update is as follows:

[0020]

[0021] In the formula, This indicates a local update for client i; This indicates the gradient update obtained by client i using the stochastic gradient descent algorithm to train the local model;

[0022] The updated local model expression is as follows:

[0023]

[0024] In the formula, Let α represent the local model of client i in round t+1; let α represent the machine learning rate; and let τ represent the local training round.

[0025] Furthermore, obtaining the updated local model of the client group includes the following steps:

[0026] For any client group, calculate the cosine similarity score of the updated local model for each client in the client group;

[0027] The client with the highest cosine similarity score is selected as the central client;

[0028] Based on the local model updated by the central client and a preset number of clients with the closest cosine similarity to the central client, the updated local model of the client group is obtained by averaging.

[0029] The preset quantity is set to Nf-2, where N represents the total number of clients in the client group; f represents the number of malicious clients, and satisfies f = (N-1) / 3.

[0030] Furthermore, the calculation expression for the cosine similarity score is as follows:

[0031]

[0032] In the formula, s(i) represents the cosine similarity score of the local model of client i; i→j indicates that client j belongs to the multiple local models that are closest to the local model of client i; Cos(w i ,w j ) represents the cosine similarity between the local models of client i and client j within the group; w i Represents the local model of client i; w j This represents the local model of client j.

[0033] Furthermore, the client groups include first-level client groups, second-level client groups, and third-level client groups, and the aggregation methods include blockchain subscription aggregation and regulatory node aggregation;

[0034] The blockchain subscription aggregation includes the following steps:

[0035] Based on the second-level client group, the updated local model is obtained by taking into account the updated local models of all subordinate third-level client groups and the updated local model of the second-level client group itself.

[0036] Based on the first-level client group, the updated local model is obtained by taking into account the updated local models of all the subordinate second-level client groups and the updated local model of the first-level client group itself.

[0037] Aggregate the updated local models of all first-level client groups to obtain the updated global model;

[0038] The aggregation of regulatory nodes includes the following steps: sending the local models of all client groups to the regulatory node group based on preset regulatory nodes; aggregating all local models based on the regulatory node group to obtain an updated global model.

[0039] Furthermore, the expression for aggregation via blockchain subscription is as follows:

[0040]

[0041] In the formula, w t+1w represents the global model obtained by aggregation at the (t+1)th iteration. t This represents the global model obtained by aggregation in the t-th iteration; K represents the number of client groups in the first level. Let the local model of the k-th first-level client group after the second update be represented in the (t+1)-th iteration, as shown in the following expression:

[0042]

[0043] In the formula, M represents the local model of the k-th first-level client group in the (t+1)-th iteration; k This represents the number of second-level client groups under the k-th first-level client group; Let m represent the local model after the second update of the m-th second-level client group in the (t+1)-th iteration, expressed as follows:

[0044]

[0045] In the formula, This represents the local model of the m-th second-level client group at the (t+1)-th iteration; This represents the number of third-level client groups belonging to the m-th second-level client group under the k-th first-level client group; This represents the local model of the nth third-level client group at the (t+1)th iteration.

[0046] Furthermore, the expression for aggregation through the regulatory node is as follows:

[0047]

[0048] In the formula, w t+1 w represents the global model obtained by aggregation at the (t+1)th iteration. t This represents the global model obtained by aggregation at the t-th iteration;

[0049] Let K represent the local model of the k-th first-level client group in the (t+1)-th iteration; K represents the number of first-level client groups.

[0050] M represents the local model of the m-th second-level client group in the (t+1)-th iteration; k This represents the number of second-level client groups under the k-th first-level client group;

[0051] This represents the local model of the nth third-level client group at the (t+1)th iteration; This represents the number of third-level client groups belonging to the m-th second-level client group under the k-th first-level client group.

[0052] Furthermore, the regulatory node group is also used to realize revenue distribution, including the following steps:

[0053] After aggregating to obtain the updated global model, obtain the total benefit R(t) of the current iteration of federated learning;

[0054] Based on the total revenue R(t), the revenue of client groups at different levels is calculated, and the revenue is distributed through multiple regulatory nodes based on the regulatory node group.

[0055] Furthermore, the expression for the revenue of the first-tier client group is as follows:

[0056]

[0057] In the formula, R k (t) represents the profit of the k-th first-level client group in the t-th iteration; Cos(w k ,w t ) represents the local model w of the first-level client group k. k With global model w t The cosine similarity; R(t) represents the total revenue at the t-th iteration; K represents the number of clients in the first-level client group;

[0058] The expression for the revenue of the second-tier client group is as follows:

[0059]

[0060] In the formula, R m (t) represents the profit of the m-th second-level client group in the t-th iteration; Cos(w m ,w t ) represents the local model w of the second-level client group m. m With global model w t Cosine similarity; M k This indicates the number of second-level client groups under the first-level client group k;

[0061] The expression for the revenue of the third-tier client group is as follows:

[0062]

[0063] In the formula, R n (t) represents the profit of the nth third-level client group in the t-th iteration; Cos(w n ,w t) represents the local model w of the third-level client group n. n With global model w t Cosine similarity; This represents the number of third-level client groups that belong to the second-level client group m under the first-level client group k.

[0064] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0065] The federated learning system based on the FISCO group mechanism of this invention firstly divides the entire federated learning system into client groups at different levels by introducing a multi-group architecture of FISCO BCOS, thus achieving a divide-and-conquer approach that maintains security and decentralization while enabling expansion. Transaction processing, data storage, and block consensus between groups are isolated, ensuring the privacy of the blockchain system. Secondly, the introduction of supervisory nodes not only enables cross-institutional, penetrating supervision but also significantly improves the efficiency of model aggregation. Furthermore, the malicious node screening mechanism based on cosine similarity offers better robustness, guaranteeing the overall effectiveness of the group. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of a federated learning system based on the FISCO group mechanism. Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0068] Example 1

[0069] This embodiment 1 provides a federated learning system based on the FISCO group mechanism, including a supervisory node group and multiple levels of client groups. A client group includes multiple clients, and a client group at a certain level has multiple client groups at the next lower level. The supervisory node group includes multiple levels of supervisory nodes, and a supervisory node is used to monitor one or more client groups at the same level.

[0070] For any client, the method is to obtain the initial or previous global model; based on the global model, to perform local training and update on the client's local model using the stochastic gradient descent algorithm; and to obtain the updated local model of the client by calculating the preset τ-inexact solution.

[0071] For any client group, the updated local model of the client group is obtained based on the updated local models of multiple clients in the client group, using a practical Byzantine fault-tolerant consensus mechanism and a malicious node filtering mechanism based on cosine similarity.

[0072] Based on blockchain or custodian nodes, it is used to aggregate the updated local models of all client groups to obtain an updated global model; in response to the updated global model satisfying the preset federated learning convergence conditions, the federated learning of the global model based on the FISCO group mechanism is completed, otherwise the next round of federated learning is entered.

[0073] The technical concept of this invention is as follows: First, by introducing the multi-group architecture of FISCO BCOS, the entire federated learning system is divided into client groups at different levels, using a divide-and-conquer approach to expand while maintaining security and decentralization; transaction processing, data storage, and block consensus between groups are isolated from each other, ensuring the privacy of the blockchain system; second, the introduction of regulatory nodes not only enables cross-institutional, penetrating supervision but also greatly improves the efficiency of model aggregation; furthermore, the malicious node screening mechanism based on cosine similarity has better robustness, ensuring the overall benefit of the group.

[0074] like Figure 1 As shown, the federated learning system comprises a group of supervisory nodes and three tiers of client groups: the first-tier client group, the second-tier client group, and the third-tier client group. The blockchain network topology of the federated learning system is as follows:

[0075] There are a total of K client groups in the first level, namely group_1, group_2, ..., group_k, ..., group_K;

[0076] For any first-level client group group_k, there are M members. k There are two second-level client groups, namely group_k_1, group_k_2, ..., group_k_m. k , ..., group_k_M k ;

[0077] For any second-level client group group_k_m k Subordinates have There are three third-level client groups, namely group_k_m k _1, group_k_m k _2, ..., …,

[0078] This federated learning system constructs a group of supervisory nodes, which, like the client groups, are divided into multiple levels. Each level of supervisory node needs to supervise the client groups at its corresponding level. Therefore, each supervisory node is simultaneously joined to multiple groups, including one supervisory node group and others client groups. If the number of clients is too large, to reduce the number of supervisory nodes, supervisory node i can also belong to several client groups simultaneously. For example, a supervisory node can supervise multiple third-level client groups at the same time, and must join its group each time it supervises a third-level client.

[0079] The federated learning method of the FISCO group mechanism-based federated learning system includes the following steps:

[0080] Step 1: Local training on the client side.

[0081] To obtain the updated local model for any given client, the following steps are involved:

[0082] It obtains the initialized or previously learned global model w from the client group of the next higher level to which it belongs. t ;

[0083] Based on the global model, the local loss of the client is calculated using the stochastic gradient descent algorithm. The expression for local loss is:

[0084]

[0085] In the formula, This represents the local loss of client i; Represents the local model of client i; w t Represents the global model; Representation Model The loss on the data distribution of client i; β represents the strength of the constraint imposed on the local model. When β = 0, the local model completely ignores global information. The larger β is, the closer the local model is to the global model; ||·|| 2 This represents the 2-norm, also known as the Euclidean norm, which is primarily used to measure the size of a vector.

[0086] Then calculate the client's local update. The expression for local update is as follows:

[0087]

[0088] In the formula, This indicates that the local model of client i has been updated; This indicates the gradient update obtained by client i using the stochastic gradient descent algorithm to train the local model;

[0089] As can be seen from the expression for local updates, when the client updates the local model, while calculating the gradient according to the local data, it also strives to use the global model to constrain the local model update so that it does not deviate too far.

[0090] During local training, we further defined the τ-inexact solution, which allows the client to determine the number of training rounds based on available local system resources, reducing terminal failures caused by system heterogeneity.

[0091] Specifically, for the local loss function We set It is a τ-inexact solution for client i, where τ is used to measure the local computational cost on the client side, i.e., the number of local training rounds.

[0092] Suppose that client i obtains a local training result after performing local training in the t-th round of global iteration. Solve the problem and send it to its parent client:

[0093]

[0094] In the formula, Let α represent the local model of edge client i in round t+1; α represent the machine learning rate; τ represent the local training rounds, used to measure the local computational cost of the edge terminal. This represents the local computation cost of client i in the t-th global iteration round. It can adapt to different clients and different iteration rounds, mitigating the impact of client system heterogeneity. Furthermore, unlike other studies where excessive local execution rounds lead to client drift, we apply the constraint ||w|| to the local loss function. i -w t || 2 Even if the client performs multiple local updates, it will not deviate too far from the global model, thus mitigating client drift.

[0095] After each client completes its local training, it obtains its own local model. We need to aggregate the local models from the clients, first generating a local model, then generating a global model, and then redistributing it to all clients for the next round of iterative training.

[0096] Step 2: Aggregate local models within the group to generate a local model.

[0097] Each group has its own ledger, which allows for transaction processing, data storage, and block consensus within the group, ensuring the privacy of the blockchain system. At the same time, transactions between different groups can be executed in parallel, improving efficiency.

[0098] Clients within the same group can reach a consensus without a central server based on the Practical Byzantine Fault Tolerance (PBFT) consensus mechanism, aggregating the local models of clients within the group to generate a local model.

[0099] In the t-th round of global iteration, after institutions within the group complete their local training within a certain time, the aggregation smart contract within the group is triggered. The smart contract should calculate the local model within the group based on these local models. However, to prevent some institutions from uploading local models that deviate too far from the local models of other institutions due to dishonesty or attacks, thereby forcing the aggregated local model to stray from the global model and lowering the overall efficiency of the group, we set up a malicious node filtering mechanism based on cosine similarity within the group.

[0100] The Practical Byzantine Fault Tolerance (PBFT) consensus mechanism operates on the assumption that less than one-third of the blockchain nodes are dishonest, which means there should be at least n = 3f + 1 nodes to tolerate f malicious nodes. Therefore, f = (n - 1) / 3.

[0101] To tolerate f erroneous miners, we define a malicious node screening mechanism based on cosine similarity. First, we use cosine similarity to measure the distance between the local models provided by two nodes. This is because cosine similarity calculates the cosine of the angle between the two local models, independent of model size; while Euclidean distance refers to the true distance between two points in a high-dimensional space. Euclidean distance is affected by dimensionality, and cosine similarity maintains the property of having a similarity of 1 even when the two nodes are completely identical in low dimensions. Therefore, cosine similarity is more robust than Euclidean distance.

[0102] Therefore, to obtain the updated local model for any client group, the following steps are involved:

[0103] Calculate the cosine similarity score of the updated local model for each client in the client group;

[0104] The client with the highest cosine similarity score is selected as the central client.

[0105] Based on the local models updated by the central client and a preset number of clients whose cosine similarity is closest to that of the central client, the updated local model of the client group is obtained by averaging.

[0106] The preset quantity is set to Nf-2, where N represents the total number of clients in the client group; f represents the number of malicious clients, and satisfies f = (N-1) / 3.

[0107] The formula for calculating the cosine similarity score is as follows:

[0108]

[0109] In the formula, s(i) represents the cosine similarity score of the local model of client i; i→j indicates that client j belongs to the multiple local models that are closest to the local model of client i; Cos(w i ,w j ) represents the cosine similarity between the local models of client i and client j within the group; w i Represents the local model of client i; w j This represents the local model of client j.

[0110] Step 3: Generate the global model.

[0111] In addition to aggregating models within the group, each client group also needs to collect local models from its subordinate client groups, aggregate the local models of its subordinate clients with the local models of its own group, and then send the updated local models to its parent client group.

[0112] To collect local models from subordinate client groups and send these models to their parent client groups, we need to perform message passing between groups. However, since groups are independent of each other, it is impossible to establish long-lived connections between them, making it difficult to obtain messages in real time. Therefore, this invention provides a method for achieving message passing between groups—a supervisory node.

[0113] Since the regulatory scope of the nodes in the regulatory node group covers every client group, obtaining model updates from all groups within the regulatory node group would significantly improve efficiency. Therefore, we send a cross-group transaction from the regulatory node group to all client groups. If a client group's verification is successful, its respective regulatory node will send the model information to the regulatory node group. Once the regulatory node group receives the model information from all client groups, it can then aggregate the data.

[0114] Specifically, the steps include: sending the local models of all client groups to the supervisory node group based on a preset supervisory node; aggregating all local models based on the supervisory node group to obtain the updated global model. The expression for aggregation via supervisory nodes is as follows:

[0115]

[0116] In the formula, w t+1 w represents the global model obtained by aggregation at the (t+1)th iteration. t This represents the global model obtained by aggregation at the t-th iteration;

[0117] Let K represent the local model of the k-th first-level client group in the (t+1)-th iteration; K represents the number of first-level client groups.

[0118] M represents the local model of the m-th second-level client group in the (t+1)-th iteration; k This represents the number of second-level client groups under the k-th first-level client group;

[0119] This represents the local model of the nth third-level client group at the (t+1)th iteration; This represents the number of third-level client groups belonging to the m-th second-level client group under the k-th first-level client group.

[0120] Step 4: Verification and supervision of regulatory nodes.

[0121] To encourage client participation in the federated learning system, we need to establish a fair incentive mechanism, where nodes that contribute more receive greater rewards. As mentioned earlier, we used cosine similarity to filter malicious nodes; here, we'll use cosine similarity as an incentive, where nodes closer to the global model contribute more and should receive greater rewards.

[0122] Assuming that in the global model at round t, the total profit is R(t), the group of regulatory nodes needs to distribute R(t) fairly.

[0123] First, the regulatory node group needs to calculate the revenue of each first-tier client group.

[0124] Model w for each first-tier client group k The cosine similarity with the global model w is We will only allocate profits to client groups with positive cosine similarity. The profit of a first-level client group k in round t is:

[0125]

[0126] In the formula, R k (t) represents the profit of the k-th first-level client group in the t-th iteration; Cos(w k ,w t ) represents the local model w of the first-level client group k. k With global model w t Cosine similarity, where Cos(w k ,w t R(t) is positive; R(t) represents the total revenue at the t-th iteration; K represents the number of clients in the first-level client group.

[0127] Then, the second-level client group and the third-level client group are incentivized in sequence using the method described above.

[0128] Assume that the first-level client group k has M members. k There is a second-level client group, m, which is the next level client group after the first-level client group k. The profit of the second-level client group m is:

[0129]

[0130] In the formula, R m (t) represents the profit of the m-th second-level client group in the t-th iteration; Cos(w m ,w t ) represents the local model w of the second-level client group m. m With global model w t Cosine similarity, where Cos(w m ,w t ) is positive; M k This indicates the number of second-level client groups under the first-level client group k.

[0131] Assume the second-level client group m has There is a third-level client group, where the third-level client group n is the next level client group after the second-level client group m. The revenue of the third-level client group is:

[0132]

[0133] In the formula, R n (t) represents the profit of the nth third-level client group in the t-th iteration; Cos(w n ,w t ) represents the local model w of the third-level client group n. n With global model w t Cosine similarity, where Cos(w n ,w t ) is positive; This represents the number of third-level client groups that belong to the second-level client group m under the first-level client group k.

[0134] Step 5: If the updated global model meets the preset federated learning convergence conditions, the federated learning of the global model based on the FISCO group mechanism is completed; otherwise, the next round of federated learning begins.

[0135] The preset convergence conditions for federated learning in this invention include the aggregation of the global model to achieve a preset effect or the number of learning iterations to reach the maximum number of iterations. The specific preset effect or the maximum number of iterations can be adjusted according to the actual situation, and this invention does not impose any restrictions.

[0136] Example 2

[0137] Based on Example 1, Example 2 provides another method for achieving inter-group message passing for generating the global model in step three – block event subscription.

[0138] Fisco supports on-chain event subscription and notification. Let's take WeEvent as an example. WeEvent is a blockchain-based event middleware service. Events published to WeEvent are permanently stored, immutable, and support post-event tracking and auditing. Each third-level client group can only exchange information with its subordinate second-level client groups, and each second-level client can only exchange information with its subordinate first-level client groups. We set up a topic for each group that needs to collect messages from its subordinate client groups to collect model updates from these groups; simultaneously, we need to set up a topic for each subordinate client group to receive the global model. These topics are completely isolated; publishing, storage, and notifications do not affect each other.

[0139] Subordinate clients, acting as producers, publish events to a specific topic. WeEvent permanently stores these events on the FISCO-BCOS blockchain, ensuring immutability and supporting post-event verification. Upon successful event publication, WeEvent returns an EventID, which is unique and represents the event. The parent client, acting as a consumer, subscribes to a topic and receives timely event notifications when a producer publishes an event to that topic. Consumers can also unsubscribe.

[0140] Blockchain subscription aggregation includes the following steps:

[0141] Based on the second-level client group, the updated local model is obtained by taking into account the updated local models of all subordinate third-level client groups and the updated local model of the second-level client group itself.

[0142] Based on the first-level client group, the updated local model is obtained by taking into account the updated local models of all the subordinate second-level client groups and the updated local model of the first-level client group itself.

[0143] Aggregate the updated local models of all first-level client groups to obtain the updated global model;

[0144] The specific expression for aggregation via blockchain subscription is as follows:

[0145]

[0146] In the formula, w t+1 w represents the global model obtained by aggregation at the (t+1)th iteration. t This represents the global model obtained by aggregation in the t-th iteration; K represents the number of client groups in the first level. Let the local model of the k-th first-level client group after the second update be represented in the (t+1)-th iteration, as shown in the following expression:

[0147]

[0148] In the formula, M represents the local model of the k-th first-level client group in the (t+1)-th iteration; k This represents the number of second-level client groups under the k-th first-level client group; Let m represent the local model after the second update of the m-th second-level client group in the (t+1)-th iteration, expressed as follows:

[0149]

[0150] In the formula, This represents the local model of the m-th second-level client group at the (t+1)-th iteration; This represents the number of third-level client groups belonging to the m-th second-level client group under the k-th first-level client group; This represents the local model of the nth third-level client group at the (t+1)th iteration.

[0151] Then, the first-level client group, acting as the producer, publishes events to the global model topic. The global model is then sent to the second-level client group that subscribes to the message, and the second-level client group then sends the global model to its own third-level client group.

[0152] In summary, this application presents a secure, layered federated learning architecture based on the FISCO group mechanism. First, we introduce the FISCO BCOS multi-group architecture to build a multi-group blockchain platform. The entire federated learning system is divided into groups of different levels and tiers. Transaction processing, data storage, and block consensus are isolated between groups, ensuring the privacy of the blockchain system. Simultaneously, transactions between different groups can be executed in parallel, improving efficiency. Message communication between cross-groups carries verification information, ensuring trustworthiness and traceability.

[0153] Secondly, we achieved cross-institutional, penetrating oversight. Because FISCO BCOS supports blockchain nodes launching multiple groups, we set up several oversight nodes. Each oversight node can join multiple groups, including an oversight node group and groups of the various clients it oversees. For example, oversight node i belongs to both the oversight node group and several client groups. The main functions of oversight nodes include verification, oversight, and incentive allocation. The introduction of oversight nodes not only enables cross-institutional, penetrating oversight but also significantly improves the efficiency of model aggregation.

[0154] Furthermore, we propose a screening and incentive mechanism based on cosine similarity, which can not only filter out malicious nodes but also fairly allocate incentives. Cosine similarity calculates the cosine of the angle between the update vectors of two local models, regardless of the magnitude of the update vectors; while Euclidean distance refers to the true distance between two points in high-dimensional space. The value of Euclidean distance is affected by dimensionality, and cosine similarity retains the property of having a similarity of 1 when they are completely identical in low dimensions, even in high-dimensional cases. Therefore, cosine similarity is more robust than Euclidean distance.

[0155] Finally, to address the data heterogeneity and system heterogeneity among different institutions, we introduced the τ-inexact solution during the client training phase. This applies a soft constraint between the client's local model and the global model, preventing the client's local model from deviating too far from the global model. This reduces the impact of data heterogeneity and allows the client to determine the number of training epochs based on available local system resources, thus reducing terminal failures caused by system heterogeneity.

[0156] The advantages of the overall system are as follows:

[0157] First, scalability: The entire federated learning system is divided into groups of different levels and grades, using a divide-and-conquer approach to expand while maintaining security and decentralization;

[0158] Second, decoupling: Each group is decoupled from the others and operates independently. Each group independently executes the consensus process, maintains its own transactions and data, and is not affected by other groups, thus achieving better privacy isolation.

[0159] Thirdly, security: a screening and incentive mechanism based on cosine similarity is introduced. Cosine similarity is more robust and can not only screen out malicious nodes, but also fairly distribute incentives.

[0160] Fourth, penetrating supervision: We have set up some supervision nodes, and each supervision node can join multiple groups, one of which is the supervision node group, and the others are the groups of the various clients it supervises, which can realize cross-institutional penetrating supervision.

[0161] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0165] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A federated learning system based on the FISCO group mechanism, characterized by: It includes a group of monitoring nodes and multiple levels of client groups. A client group includes multiple clients, and a client group at a certain level has multiple client groups at the next lower level. The group of monitoring nodes includes multiple levels of monitoring nodes, and a monitoring node is used to monitor one or more client groups at the same level. For any client, the method is to obtain the initialized or previously learned global model; based on the global model, to perform local training and update of the client's local model using the stochastic gradient descent algorithm, and to obtain the updated local model of the client by calculating the preset τ-inexact solution. For any client group, the updated local model of the client group is obtained based on the updated local models of multiple clients in the client group, using a practical Byzantine fault-tolerant consensus mechanism and a malicious node filtering mechanism based on cosine similarity. Based on blockchain or a supervisory node, it is used to aggregate the updated local models of all client groups to obtain an updated global model; in response to the updated global model satisfying the preset federated learning convergence conditions, the federated learning of the global model based on the FISCO group mechanism is completed; otherwise, the next round of federated learning is entered. Obtaining the updated local model from the client includes the following steps: Get the initialized or global model obtained from the previous learning iteration; Based on the global model, the local loss of the client is calculated using the stochastic gradient descent algorithm, and then the local update of the client is calculated. Based on the client's local update, the updated local model of the client is obtained by calculating the preset τ-inexact solution; The expression for the local loss is: In the formula, This represents the local loss of client i; Represents the local model of client i; w t Represents the global model; Representation Model The loss on the data distribution of client i; β represents the strength of the constraints imposed on the local model; ||·|| 2 Represents the 2-norm; The expression for the local update is as follows: In the formula, This indicates a local update for client i; This indicates the gradient update obtained by client i using the stochastic gradient descent algorithm to train the local model; The updated local model expression is as follows: In the formula, Let represent the local model of client i in round t+1; α represent the machine learning rate; and τ represent the local training round. The client groups include first-level client groups, second-level client groups, and third-level client groups; the aggregation methods include blockchain subscription aggregation and regulatory node aggregation. The blockchain subscription aggregation includes the following steps: Based on the second-level client group, the updated local model is obtained by taking into account the updated local models of all subordinate third-level client groups and the updated local model of the second-level client group itself. Based on the first-level client group, the updated local model is obtained by taking into account the updated local models of all the subordinate second-level client groups and the updated local model of the first-level client group itself. Aggregate the updated local models of all first-level client groups to obtain the updated global model; The aggregation of regulatory nodes includes the following steps: sending the local models of all client groups to the regulatory node group based on preset regulatory nodes; aggregating all local models based on the regulatory node group to obtain an updated global model.

2. The federated learning system based on the FISCO group mechanism according to claim 1, characterized in that, Obtaining the updated local model of the client group includes the following steps: For any client group, calculate the cosine similarity score of the updated local model for each client in the client group; The client with the highest cosine similarity score is selected as the central client; Based on the local models updated by the central client and a preset number of clients whose cosine similarity is closest to that of the central client, the updated local model of the client group is obtained through averaging. The preset quantity is set to Nf-2, where N represents the total number of clients in the client group; f represents the number of malicious clients, and satisfies f = (N-1) / 3.

3. The federated learning system based on the FISCO group mechanism according to claim 2, characterized in that, The formula for calculating the cosine similarity score is as follows: In the formula, s(i) represents the cosine similarity score of the local model of client i; i→j indicates that client j belongs to the multiple local models that are closest to the local model of client i; Cos(w i ,w j ) represents the cosine similarity between the local models of client i and client j within the group; w i Represents the local model of client i; w j This represents the local model of client j.

4. The federated learning system based on the FISCO group mechanism according to claim 1, characterized in that, The expression for aggregation via blockchain subscription is as follows: In the formula, w t+1 w represents the global model obtained by aggregation at the (t+1)th iteration. t This represents the global model obtained by aggregation in the t-th iteration; K represents the number of client groups in the first level. Let the local model of the k-th first-level client group after the second update be represented in the (t+1)-th iteration, as shown in the following expression: In the formula, M represents the local model of the k-th first-level client group in the (t+1)-th iteration; k This represents the number of second-level client groups under the k-th first-level client group; Let m represent the local model after the second update of the m-th second-level client group in the (t+1)-th iteration, expressed as follows: In the formula, This represents the local model of the m-th second-level client group at the (t+1)-th iteration; This represents the number of third-level client groups belonging to the m-th second-level client group under the k-th first-level client group; This represents the local model of the nth third-level client group at the (t+1)th iteration.

5. The federated learning system based on the FISCO group mechanism according to claim 1, characterized in that, The expression for aggregation through the regulatory node is as follows: In the formula, w t+1 w represents the global model obtained by aggregation at the (t+1)th iteration. t This represents the global model obtained by aggregation at the t-th iteration; Let K represent the local model of the k-th first-level client group in the (t+1)-th iteration; K represents the number of first-level client groups. M represents the local model of the m-th second-level client group in the (t+1)-th iteration; k This represents the number of second-level client groups under the k-th first-level client group; This represents the local model of the nth third-level client group at the (t+1)th iteration; This represents the number of third-level client groups belonging to the m-th second-level client group under the k-th first-level client group.

6. The federated learning system based on the FISCO group mechanism according to claim 4, characterized in that, The regulatory node group is also used to realize revenue distribution, including the following steps: After aggregating to obtain the updated global model, obtain the total benefit R(t) of the current iteration of federated learning; Based on the total revenue R(t), the revenue of client groups at different levels is calculated, and the revenue is distributed through multiple regulatory nodes based on the regulatory node group.

7. The federated learning system based on the FISCO group mechanism according to claim 1, characterized in that, The expression for the revenue of the first-tier client group is as follows: In the formula, R k (t) represents the profit of the k-th first-level client group in the t-th iteration; Cos(w k ,w t ) represents the local model w of the first-level client group k. k With global model w t The cosine similarity; R(t) represents the total revenue at the t-th iteration; K represents the number of clients in the first-level client group; The expression for the revenue of the second-tier client group is as follows: In the formula, R m (t) represents the profit of the m-th second-level client group in the t-th iteration; Cos(w m ,w t ) represents the local model w of the second-level client group m. m With global model w t Cosine similarity; M k This indicates the number of second-level client groups under the first-level client group k; The expression for the revenue of the third-tier client group is as follows: In the formula, R n (t) represents the profit of the nth third-level client group in the t-th iteration; Cos(w n ,w t ) represents the local model w of the third-level client group n. n With global model w t Cosine similarity; This represents the number of third-level client groups that belong to the second-level client group m under the first-level client group k.

Images