Federal learning prediction model construction method based on data sample weight
By employing a data sample weighting method in federated learning to construct predictive models, problems such as poor model generalization ability and privacy leakage are solved, achieving efficient and secure model training and prediction.
Patent Information
- Application Number
- CN202410987066.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-26
AI Technical Summary
Differences in data characteristics among different institutions in federated learning lead to poor model generalization ability, and deep neural networks are prone to overfitting to biased data. Centralized machine learning is susceptible to privacy leaks, high communication and computing costs, and single points of failure.
A federated learning prediction model construction method based on data sample weights is adopted. By training the model on a local device and weighting the sample loss values, the data transmission is reduced, and a weighted average algorithm is used to integrate the local model parameters and optimize the global model.
It improves model prediction performance, protects user privacy, reduces communication and computing costs, lowers the risk of single points of failure, and solves the problems of data skew and model overfitting.
Smart Images

Figure CN119026670A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for constructing a federated learning prediction model, in particular to a method for constructing a federated learning prediction model based on data sample weights. Background Art
[0002] Federated learning defines a machine learning framework, under which the problem of different data owners collaborating without exchanging data is solved by designing a virtual model (the optimal model for all parties to aggregate data), and each region serves local goals based on the virtual model. However, since the data characteristics of different institutions in federated learning may be different, the performance of the models finally trained by each institution is not equivalent, resulting in poor generalization ability of the model and failure to achieve federal fairness; in addition, the deep neural network used in federated learning can easily over-adapt to biased training data with damaged labels or class imbalance, causing the trained model to shift toward this part of the data, reducing the training results. Although centralized machine learning can filter out biased data by centralizing training data, since all data needs to be transmitted to a central server for model training, there are the following problems:
[0003] 1. Potential infringement of user privacy and data security issues, limiting applications such as medical data;
[0004] 2. Increase communication and computing costs;
[0005] 3. If the central server fails or is attacked, the entire system may be affected. Summary of the invention
[0006] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provide a method for constructing a federated learning prediction model based on data sample weights.
[0007] The technical solution of the present invention is: a method for constructing a federated learning prediction model based on data sample weights, which is carried out in the following steps:
[0008] Step 1. The server selects participating clients from all clients;
[0009] Step 2. The server sends the current global model parameters to the participating clients;
[0010] Step 3. After receiving the global model parameters, the client first uses the global model parameters to calculate the loss value of each sample in the training set, then weights the sample loss values, and then sums up all the weighted sample loss values, applies the optimization algorithm to minimize the summed loss value, and updates the local model;
[0011] The weight for weighting the sample loss value is positively correlated with the size of the sample loss value. The weight corresponding to the sample with the largest loss value in the training set is the maximum value, that is, the weight is assigned to 1;
[0012] Step 4. The client encrypts the updated local model parameters and sends them back to the server;
[0013] Step 5. The server uses an aggregation algorithm to integrate the updated local model parameters received, generates a new global model and sends it to the participating clients;
[0014] Repeat steps 1-5 until convergence or the maximum communication rounds are reached.
[0015] The preferred technical solution is that the weight calculation formula of the weighted step 3 is F(x)=Ax 2 +Bx+C, where x is the loss value of each sample; let min be the minimum sample loss value, corresponding to the minimum value tex of the weight F; the maximum value of the weight F corresponds to the maximum value max of the sample loss value interval [min, +∞), which is assigned a value of 1; then the coefficients A, B, and C are solved according to the following formula:
[0016]
[0017] Amin 2 +Bmin+C=tex (2)
[0018] Amax 2 +Bmax+C=1 (3).
[0019] The present invention uses federated learning technology to train models on local devices without uploading data information, which can well protect user privacy and ensure data security. At the same time, data sample weights are added when training the model, which disperses data and calculations, reduces the demand for data transmission and central server computing resources and reduces the risk of single point failures. It solves the problems of data skew and model over-adaptation in the prior art, and can effectively improve the model prediction performance, especially for medical data involving personal privacy. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 It is a flowchart of an embodiment of the present invention.
[0021] Figure 2 It is a schematic diagram of an embodiment of the present invention for alleviating over-adaptation of biased data during training of a federated learning model.
[0022] Figure 3 Embodiment of the present invention DETAILED DESCRIPTION
[0023] The method for constructing a federated learning prediction model based on data sample weights of the present invention first performs the following processing on real clinical data:
[0024] 1. Data cleaning: delete missing values, handle outliers, and detect and remove duplicate samples;
[0025] 2. Data transformation: Use Min-Max normalization to map data values to [0,1], preserving the structure of the original data.
[0026] Then as Figure 1 Follow the steps below:
[0027] Step 1. The server selects three participating clients (Client1, Client2, Client3) from all clients. Each client has its own data set Data1, Data2, Data3. The participating clients can be selected randomly or based on certain strategies.
[0028] Step 2. The server sends the current global model parameters to participating clients
[0029] Step 3. After receiving the global model parameters, each client first uses the global model parameters to calculate the loss value of each sample in the small batch training set, then weights the sample loss values, and then sums up all the weighted sample loss values, applies the optimization algorithm to minimize the summed loss value, and updates the local model;
[0030] The weight for weighting the sample loss value is positively correlated with the size of the sample loss value. The weight corresponding to the sample with the largest loss value in the training set is the maximum value, that is, the weight is assigned to 1;
[0031] The weighted calculation formula used in this embodiment is F(x)=Ax 2 +Bx+C, where x is the loss value of each sample; F(x)=Ax 2 +Bx+C is a linear function that satisfies the following properties: F increases and is concave in the sample loss value interval [min, +∞) and F is continuously differentiable in the sample loss value interval [min, +∞); let min be the minimum sample loss value, corresponding to the minimum value tex of the weight F; the maximum value of the weight F corresponds to the maximum value max of the sample loss value interval [min, +∞), which is assigned a value of 1; then the coefficients A, B, and C are solved according to the following formula:
[0032]
[0033] Amin 2 +Bmin+C=tex (2)
[0034] Amax 2 +Bmax+C=1 (3)
[0035] Formula (4) is used to calculate the total loss of all samples in the mini-batch,
[0036]
[0037] Where b is the batch size;
[0038] The embodiment of the present invention uses small batch gradient descent parameters to minimize the sum loss value. The parameters of small batch gradient descent are as follows:
[0039]
[0040] Where η represents the learning rate, Represents the loss function relative to the parameter The gradient of
[0041] Step 4. The client updates the local model parameters After being encrypted, it is sent back to the server to protect the privacy of client data;
[0042] Step 5. The server uses an aggregation algorithm to integrate the updated local model parameters received, generates a new global model and sends it to the participating clients;
[0043] Among the existing federated learning algorithms, the most commonly used global model aggregation method is the weighted average algorithm FedAvg, which aggregates the global model W t+1 is defined as follows:
[0044]
[0045] Where W t+1 represents the updated global model, represents the local model updated by the i-th client, m represents the total number of clients, n represents the total number of samples, and ni represents the number of samples of the i-th client;
[0046] Steps 1-5 are repeated until convergence or the maximum number of communication rounds is reached, resulting in a new global model that incorporates contributions from all clients.
[0047] The present invention alleviates the problem of over-adaptation to biased data during federated learning model training by weighting local experimental data samples. Figure 2The hollow dots represent the projections of the actual samples in the two-dimensional coordinates, and the hollow triangles represent the points where the loss value of the global model is zero in the two-dimensional coordinates. Since the deep neural network tends to over-fit the dots closer to the triangle and ignore the dots farther away from the triangle, the sample weight method will attach smaller weights to the dots closer and larger weights to the dots farther away, so that Figure 2 The red dots in the global model are represented by Figure 2 a moves closer to the center and becomes Figure 2 b. Avoid the occurrence of over-adaptation problems.
[0048] Experiment: Algorithm comparison experiment on hospital and public datasets
[0049] The sample weight algorithm is applied to four common federated learning algorithms (FedAvg, FedProx, FedNova, Scaffold), and non-independent and identically distributed (data skew) experiments are conducted on multiple data sets (including real hospital data) and compared with the results of algorithms without sample weighting. The comparative experimental results after adding sample weights in FedNova are shown in Figure 2. Figure 3 As shown in the figure: FedNova-no-sample represents FedNova without adding sample weight, FedNova-Sam-0.1, FedNova-Sam-0.3, FedNova-Sam-0.5, FedNova-Sam-0.7, and FedNova-Sam-0.9 represent the experimental results when the sample weight algorithm is applied to FedNova and the sample weight hyperparameter tex is set to 0.1, 0.3, 0.5, 0.7, and 0.9, respectively.
[0050] The experimental results show that after adding sample weights to the federated learning algorithm, ACC, AUC, and F1-score are better than other algorithms, demonstrating strong predictive capabilities for non-independent and identically distributed data, solving the problem of data skew and alleviating the problem of model over-adaptation to biased data.
Claims
1. A method for constructing a federated learning prediction model based on data sample weights, characterized in that Follow the steps below: Step 1. The server selects participating clients from all clients; Step 2. The server sends the current global model parameters to the participating clients; Step 3. After receiving the global model parameters, the client first uses the global model parameters to calculate the loss value of each sample in the training set, then weights the sample loss values, and then sums up all the weighted sample loss values, applies the optimization algorithm to minimize the summed loss value, and updates the local model; The weight for weighting the sample loss value is positively correlated with the size of the sample loss value. The weight corresponding to the sample with the largest loss value in the training set is the maximum value, that is, the weight is assigned to 1; Step 4. The client encrypts the updated local model parameters and sends them back to the server; Step 5. The server uses an aggregation algorithm to integrate the updated local model parameters received, generates a new global model and sends it to the participating clients; Repeat steps 1-5 until convergence or the maximum communication rounds are reached.
2. The method for constructing a federated learning prediction model based on data sample weights according to claim 1 is characterized in that The weight calculation formula of step 3 is F(x)=Ax 2 +Bx+C, where x is the loss value of each sample; let min be the minimum sample loss value, corresponding to the minimum value tex of the weight F; The maximum value of the weight F corresponds to the maximum value max of the sample loss value interval [min, +∞), which is assigned a value of 1; then the coefficients A, B, and C are solved according to formulas (1)-(3): Amin 2 +Bmin+C=tex (2) Amax 2 +Bmax+C=1 (3)。
Citation Information
Patent Citations
Power data prediction model optimization method and system
CN115048989A
Heterogeneous robust federated learning method based on self-paced learning
CN116228729A
Transverse federated learning-based data classification model training method and related components thereof
CN116340841A
Fair federal learning method for cross-domain social network node classification task
CN117235331A
Deck construction method with spring clip applied
KR102433484B1
