A fully connected neural network model aggregation method in a federated learning scenario

By adopting the fully connected neural network model aggregation method in the federated learning scenario, the user model parameters are calculated layer by layer, and the problem of poor non-IID data processing in the existing technology is solved, and the global accuracy and generalization of the joint model are improved. ability to transform.

CN112749790BActive Publication Date: 2024-12-06DALIAN UNIV OF TECH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202110081944.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-21
Publication Date
2024-12-06
Estimated Expiration
2041-01-21

AI Technical Summary

Technical Problem

The existing federated learning model aggregation method is poor in processing non-IID data, and its performance is greatly affected by the number of local iterations and proximal terms, making it difficult to improve the global accuracy and generalization ability of the joint model.

Method used

Using the fully connected neural network model aggregation method, the central server issues hyperparameters and initial parameters to the user, the user trains and uploads the model parameters, the central server calculates and integrates node parameters layer by layer, and creates a super-determined by generating one-dimensional random vectors as input. Systems of equations solve node parameters to form a joint model.

Benefits of technology

The global accuracy and generalization ability of the joint model are improved, and the global accuracy is obtained higher than that of the FedAvg algorithm.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112749790B_ABST
    Figure CN112749790B_ABST
Patent Text Reader

Abstract

The present invention discloses a fully connected neural network model aggregation method in a federated learning scenario, which belongs to the field of artificial intelligence. The method sends the hyperparameters and parameters of the neural network to all users who provide independent models through a central server, and each user establishes a unified neural network according to the hyperparameters obtained from the central server; each user sets the initial value of the neural network as the parameter obtained from the central server, and uses their own data to perform their own neural network training; after the training of their own neural network is completed, each user uploads their own neural network parameters, that is, user model parameters, to the central server; the central server collects all user model parameters, aggregates them, and obtains a joint model; the central server determines whether the number of cycles is reached; by approximating the features obtained from each client model layer by layer, a unified model suitable for all users can be obtained; the global accuracy and generalization ability of the joint model are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a fully connected neural network model aggregation method in a federated learning scenario. Background Art

[0002] Federated learning, also known as federated machine learning, joint learning, and alliance learning. Federated learning is a machine learning framework that can effectively help multiple institutions use data and conduct machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations. As a distributed machine learning paradigm, federated learning can effectively solve the problem of data silos, allowing participants to jointly model without sharing data, and can technically break data silos and achieve AI collaboration. Federated learning defines a machine learning framework under which the problem of different data owners collaborating without exchanging data is solved by training a joint model. The joint model is the optimal model for all parties to aggregate data together, and each region serves local goals based on the model. Federated learning requires that the modeling result should be infinitely close to the traditional model, that is, the result of aggregating the data of multiple data owners in one place for modeling. Under the federation mechanism, the identity and status of each participant are the same, and a shared data strategy can be established. Since the data is not transferred, it will not leak user privacy or affect data specifications.

[0003] Currently, there are several methods for solving the joint model using the independent models of each participant: (1) FedAvg method [1]: The parameters aggregated from each device are weighted averaged. This method can achieve good results when processing independent and identically distributed (IID) data, but the new model generated may not be effective when processing non-IID data; Proposed framework (FedProx) method: An improvement on FedAvg, adding a correction term called the proximal term, which is essentially a difference restriction on the weight parameters of the local model and the weight parameters of the joint model, to increase the stability of model training and the convergence of the model, but the performance is greatly affected by the number of local iterations and the proximal term; Attention aggregation (FedAtt) method: The contribution of the client model to the joint model is considered, and an attention mechanism is adopted. In the server aggregation process, an optimization technique is used to iteratively update the parameters while paying attention to the distance between the server model and the client model, thereby minimizing the weighted distance between the server model and the client model. The calculation is relatively complex.

[0004] 202010157458.3, the invention name is Differentiated Noise Adding Method and System in Federated Learning Gradient Exchange, which relates to the same field as the present invention, but does not involve a new model aggregation method. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention discloses a fully connected neural network model aggregation method in a federated learning scenario, comprising the following steps:

[0006] S1: The central server sends the hyperparameters and parameters of the neural network to all users who provide independent models, and sets the number of cycles of S3-S5;

[0007] S2: Each user builds a unified neural network based on the hyperparameters obtained from the central server;

[0008] S3: Each user sets the initial value of the neural network to the parameters obtained from the central server, and uses their own data to train their own neural network;

[0009] S4: After the training of their respective neural networks is completed, each user uploads their respective neural network parameters, i.e., user model parameters, to the central server;

[0010] S5: The central server collects all user model parameters, aggregates them, obtains a joint model, and then sends the parameters of the joint model to each user;

[0011] S6: The central server determines whether the number of cycles has been reached. If the number of cycles has not been reached, the system returns to S3. When the number of cycles has been reached, the cycle ends.

[0012] Furthermore, the central server aggregates all the collected neural network parameters to obtain the joint model in the following process:

[0013] S5-1: Assume that there are a user models in total. Among all the layers of the fully connected neural network, the layer with the largest number of nodes is n. max nodes, each data sample’s feature vector contains b features, whose value range is [min, max]. The central server generates a(n max +1) a one-dimensional random vector X i , the size is 1×b, and the value range is [min,max];

[0014] S5-2: The central server uses (n max +1) different one-dimensional vectors X j As the model input, calculate the eigenvalue at each node in the model. If (i-1)·(n max +1) <j≤i·(n max +1), X j As the input of the i-th user model, the feature value of each node is calculated and recorded as F i (X j );

[0015] S5-3: The central server starts from the first middle layer of the joint model and calculates the parameters corresponding to each node in the middle layer layer by layer;

[0016] S5-4: Add the parameters corresponding to all the nodes obtained to the fully connected neural network and integrate them into a joint model.

[0017] Furthermore, the process of calculating the parameters corresponding to each node in the middle layer layer by layer is as follows:

[0018] The calculation method of each node parameter is the same, where x1, x2, ..., x n is the input of the node, that is, the output of the previous layer node. j ((i-1)·(n max +1) <j≤i·(n max +1)) can establish an equation:

[0019]

[0020] Among them, n is the number of nodes in the previous layer, x1,x2,…,x n To X j The k-1th layer output, v, is obtained as the input of the joint model. k To X j The feature value of the location node calculated as the input of the i-th user model, w k1 ,w k2 ,…,w kn and b k is an unknown number, a(n max +1) a one-dimensional random vector X i Establish a(n max +1) equations, establish an overdetermined system of equations, and solve the overdetermined system of equations to obtain w k1 ,w k2 ,…,w kn and b k is the parameter of the joint model regarding the node at this position.

[0021] Due to the adoption of the above technical scheme, a fully connected neural network model aggregation method in a federated learning scenario provided by the present invention can obtain a unified model suitable for all users by approximating the features obtained by each client model layer by layer when solving the parameters of each node of the federated model, and can obtain a higher global accuracy than the FedAvg algorithm; the advantage of this method is that it improves the global accuracy and generalization ability of the federated model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 Schematic diagram of a fully connected neural network. DETAILED DESCRIPTION

[0024] To make the technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below:

[0025] For example, there are two banks in different regions. Their user groups come from their respective regions and have little overlap with each other. However, their businesses are very similar, so the recorded user characteristics are the same. In order to build a more stable and generalized user model, the two banks need to expand the data set, but they cannot pass the data to each other. At this time, horizontal federated learning can be used to build a joint model. First, the two parties negotiate to determine the central server, which can be undertaken by a third party or by one of the parties. Then, perform the following steps:

[0026] S1: The central server sends the hyperparameters and parameters of the neural network to all users who provide independent models, and sets the number of cycles of S3-S5;

[0027] S2: Each user builds a unified neural network based on the hyperparameters obtained from the central server;

[0028] S3: Each user sets the initial value of the neural network to the parameters obtained from the central server, and uses their own data to train their own neural network;

[0029] S4: After the training of their respective neural networks is completed, each user uploads their respective neural network parameters, i.e., user model parameters, to the central server;

[0030] S5: The central server collects all user model parameters, aggregates them, obtains a joint model, and then sends the parameters of the joint model to each user;

[0031] S6: The central server determines whether the number of cycles has been reached. If the number of cycles has not been reached, the system returns to S3. When the number of cycles has been reached, the cycle ends.

[0032] Furthermore, the central server aggregates all the collected neural network parameters to obtain the joint model in the following process:

[0033] S5-1: Because there are two users, so a = 2, if the layer with the most nodes among all the layers of the neural network has n max nodes, each data sample’s feature vector contains b features, whose value range is [min, max]. The central server generates a(n max +1) a one-dimensional random vector X i , the size is 1×b, and the value range is [min,max];

[0034] S5-2: The central server uses (n max +1) different one-dimensional vectors X j As the model input, calculate the eigenvalue at each node in the model. If (i-1)·(n max +1) <j≤i·(n max +1), X j As the input of the i-th user model, the feature value of each node is calculated and recorded as F i (X j );

[0035] S5-3: The central server starts from the first middle layer of the joint model and calculates the parameters corresponding to each node in the middle layer layer by layer;

[0036] S5-4: Add the parameters corresponding to all the nodes obtained to the fully connected neural network and integrate them into a joint model.

[0037] Furthermore, the process of calculating the parameters corresponding to each node in the middle layer layer by layer is as follows:

[0038] The calculation method of each node parameter is the same. For example, the structure and parameters of a node in the kth layer are as follows: Figure 1 As shown, where x1,x2,…,x n is the input of the node (i.e. the output of the previous layer node), any X j ((i-1)·(n max +1) <j≤i·(n max +1)) can establish an equation:

[0039]

[0040] Among them, n is the number of nodes in the previous layer, x1,x2,…,x n To X j The k-1th layer output, v, is obtained as the input of the joint model. k To X j The feature value of the location node calculated as the input of the i-th user model, w k1 ,w k2 ,…,wkn and b k is an unknown number. a(n max +1) a one-dimensional random vector X i A(n max +1) equations, and these equations are combined into an overdetermined system of equations. Solving the overdetermined system of equations yields w k1 ,w k2 ,…,w kn and b k is the parameter of the joint model regarding the node at this position.

[0041] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A fully connected neural network model aggregation method in a federated learning scenario, characterized by: The following steps are involved: S1: Obtain user data of two banks in different regions; S2: The central server sends the hyperparameters and parameters of the neural network to all users who provide independent models, and sets the number of cycles of S4-S6; S3: Each user builds a unified neural network based on the hyperparameters obtained from the central server; S4: Each user sets the initial value of the neural network to the parameters obtained from the central server, and uses their own data to train their own neural network; S5: After the training of their respective neural networks is completed, each user uploads their respective neural network parameters, i.e., user model parameters, to the central server; S6: The central server collects all user model parameters, aggregates them, obtains a joint model, and then sends the parameters of the joint model to each user; S7: The central server determines whether the number of cycles has been reached. If the number of cycles has not been reached, the server returns to S3. When the number of cycles has been reached, the cycle ends, and the data sets of the two banks in different regions are expanded. The central server aggregates all the collected neural network parameters to obtain the joint model in the following process: S5-1: Assume that there are a user models in total. Among all the layers of the fully connected neural network, the layer with the largest number of nodes is n. max nodes, each data sample’s feature vector contains b features, whose value range is [min, max]. The central server generates a(n max +1) a one-dimensional random vector X i , the size is 1×b, and the value range is [min,max]; S5-2: The central server uses (n max +1) different one-dimensional vectors X j As the model input, calculate the eigenvalue at each node in the model. If (i-1)·(n max +1)<j≤i·(n max +1), X j As the input of the i-th user model, the feature value of each node is calculated and recorded as F i (X j ); S5-3: The central server starts from the first middle layer of the joint model and calculates the parameters corresponding to each node in the middle layer layer by layer; S5-4: Add the parameters corresponding to all the nodes obtained to the fully connected neural network and integrate them into a joint model; The process of calculating the parameters corresponding to each node in the middle layer layer by layer is as follows: The calculation method of each node parameter is the same, where x1, x2, ..., x n is the input of the node, that is, the output of the previous layer node. j ((i-1)·(n max +1)<j≤i·(n max +1)) can establish an equation: Among them, n is the number of nodes in the previous layer, x1,x2,…,x n To X j The k-1th layer output, v, is obtained as the input of the joint model. k To X j The feature value of the location node calculated as the input of the i-th user model, w k1 ,w k2 ,…,w kn and b k is an unknown number, a(n max +1) a one-dimensional random vector X i Establish a(n max +1) equations, establish an overdetermined system of equations, and solve the overdetermined system of equations to obtain w k1 ,w k2 ,…,w kn and b k is the parameter of the joint model regarding the node at this position.

Citation Information

Patent Citations

  • Federated learning information processing method and system, storage medium, program and terminal

    CN111611610A