Dynamic weight aggregation federal learning method and system for credit risk assessment

Abstract

The invention discloses a dynamic weight aggregation federated learning method and system for credit risk assessment, and the method comprises the steps: firstly, calculating the cosine similarity of each client model and the last gradient of a global model, so as to assess the consistency of model parameters; secondly, the distribution difference is quantified by calculating the data statistic difference between each client data and the global data, so that the diversity of the data is evaluated; further, a contribution degree comprehensively considering model parameter similarity and data distribution difference is obtained by combining a hyper-parameter with cosine similarity and data distribution difference; and finally, dynamically adjusting the weight of each client in global model updating according to the contribution degree of each client, so that the model pays more attention to the client with higher data quality, thereby improving the overall model performance. The federated learning method is especially suitable for processing non-independent identically distributed data, optimizes the generalization ability of the model, and is suitable for credit risk assessment in the financial field.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular to a dynamic weight aggregation federated learning method for credit risk assessment. Background Art

[0002] In the financial field, credit risk assessment is crucial for banks and financial institutions. Traditional credit risk assessment methods rely on centralized data processing and statistical techniques such as logistic regression and decision trees. These methods have limitations in dealing with nonlinear relationships and high-dimensional data, and are difficult to adapt to rapid changes in data distribution. With the development of machine learning technology, the accuracy of credit risk assessment has been significantly improved. However, these methods usually require a large amount of centralized data for training, which not only involves data privacy and security issues, but also limits the generalization ability of the model when the data distribution is uneven or non-independent and identically distributed (non-IID).

[0003] As an emerging distributed machine learning method, federated learning allows multiple clients to jointly train a global model without sharing original data, effectively protecting the privacy of user data. However, in practical applications, federated learning faces challenges in data distribution heterogeneity and model update strategies. Traditional federated learning algorithms usually adopt a weighted average method based on data volume, which may not accurately reflect the actual contribution of the client to the global model, especially when the data distribution heterogeneity is large. Summary of the invention

[0004] The purpose of the present invention is to provide a dynamic weight aggregation federated learning method for credit risk assessment in response to the above problems. The method dynamically adjusts the weight of the client in the global model update by calculating the client's contribution to the global model, aiming to improve the accuracy and efficiency of credit risk assessment while protecting the privacy of user data.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A dynamic weight aggregation federated learning method for credit risk assessment includes the following:

[0007] Step 10: Build an initialized global model for credit risk assessment based on federated learning, and perform a model distribution operation to distribute the initialized global model to all clients as local models;

[0008] Step 20: Calculate the data distribution difference ψ between the global data and the local data in the first round of update i , including the following:

[0009] Receive the local data statistics in the first round of updates, which are specifically the mean μ of the jth feature of the data calculated on the i-th client. i,j and variance σ i,j ;

[0010] Calculate the global data statistics in the first round of updates, which is specifically the mean μ of the kth feature of the global data g,k and variance σ g,k ;

[0011] Calculate the data distribution difference ψ between the local data and the global data i , the data distribution difference ψ i Specifically, the data mean difference Δμ i and variance difference Δσ i sum;

[0012] Step 30: Calculate the contribution γ of each local model to the global model after training the model i , including the following:

[0013] Receive the updated gradient θ of the last layer of the local model i ;

[0014] According to the gradient θ of the last layer of the local model i and the gradient θ of the last layer of the global model g , calculate the cosine similarity

[0015] According to cosine similarity and data distribution difference ψ i , calculate the contribution of each local model to the global model γ i ;

[0016] Step 40: Normalize the contribution to obtain the dynamic aggregation weight of each local model

[0017] Step 50: Aggregate the received updated local model parameters using dynamic aggregation weights to update the global model;

[0018] Step 60: Execute the model distribution operation to distribute the updated global model to each client as a local model to complete this round of training; then, enter the next round of training, repeat the aforementioned steps 30 to 50 for iterative optimization.

[0019] Among them, in step 20, the data distribution difference ψ i The calculation formula is Among them, μ i,mis the mth feature mean of the i-th client data, μ g,m is the mth feature mean of the global data, σ i,m is the mth feature variance of the i-th client data, σ g,m is the mth feature variance of the global data, and M is the dimension of the feature.

[0020] In step 30, the cosine similarity The calculation formula is

[0021] In step 30, the contribution γ i The calculation formula is Among them, α (0<α<1) is a weight parameter used to adjust the relative importance of cosine similarity and data distribution difference in contribution calculation. is the cosine similarity between the last layer gradient of the i-th local model and the global model, ψ i is the data distribution difference between the i-th local model and the global model.

[0022] In step 40, the dynamic aggregation weight The calculation formula is Among them, γ i is the contribution of local model i to the global model.

[0023] In step 50, the aggregate calculation formula is Where K is the number of local models, are the local model parameters uploaded by local model k after the tth round of training. is the dynamic aggregation weight of the local model k.

[0024] Due to the adoption of the above technical solution, the present invention has the following beneficial effects:

[0025] The present invention firstly calculates the data distribution difference and cosine similarity between the client and the server to accurately quantify the contribution of each client to the global model; secondly, normalizes the contribution of all clients to obtain the weight of each client in the global model update; finally, the global model parameters are updated by aggregating the client model parameters by dynamic weights. The federated learning algorithm proposed in the present invention effectively alleviates the problems of privacy protection and data heterogeneity in credit risk assessment. The method allows the client to participate in the training of the global model without disclosing personal data, and dynamically adjusts the client weights to adapt to different data distributions, thereby improving the generalization ability and evaluation accuracy of the model. In addition, the present invention also enhances the fairness and interpretability of the model, while protecting user privacy, improving the accuracy and efficiency of credit risk assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a system block diagram of the dynamic weight aggregation federated learning method of the present invention.

[0027] Figure 2 It is a flow chart of the method for performing dynamic weight aggregation based on the contribution of the client of the present invention.

[0028] Figure 3 It is a schematic diagram comparing the accuracy of the dynamic weight aggregation federated learning method of the present invention and the traditional federated learning methods FedAvg, FedProx, MOON and SCAFFOLD.

[0029] Figure 4 It is a schematic diagram comparing the loss values ​​of the dynamic weight aggregation federated learning method of the present invention and the traditional federated learning methods FedAvg, FedProx, MOON and SCAFFOLD.

[0030] Figure 5 It is a comparative diagram of the AUC values, precision, recall and F1 scores of the dynamic weight aggregation federated learning method of the present invention and the traditional federated learning methods FedAvg, FedProx, MOON and SCAFFOLD. DETAILED DESCRIPTION

[0031] The specific implementation of the invention is further described below with reference to the accompanying drawings.

[0032] Example

[0033] As described above, the dynamic weight aggregation federated learning method for credit risk assessment of the present invention includes steps 10 to 50, wherein some specific steps can be adjusted in sequence according to actual conditions, and thus combined to form several specific application schemes. An example of a dynamic weight aggregation federated learning method is given below, which includes steps S1 to S9, and this scheme will be used as an example for further explanation.

[0034] Based on the dynamic weight aggregation federated learning method, a dynamic weight aggregation federated learning device for credit risk assessment can be formed, which is specifically applied on a server, and includes a processor, a memory, and a dynamic weight aggregation federated learning program; the dynamic weight aggregation federated learning program is stored in the memory and is configured to be executed by the processor to complete the dynamic weight aggregation federated learning method for credit risk assessment of the present invention. The server and its memory, processor, etc. all adopt existing technologies, which will not be repeated here.

[0035] On the other hand, based on the dynamic weight aggregation federated learning method, a dynamic weight aggregation federated learning system for credit risk assessment can be formed, which includes a server, several clients and a dynamic weight aggregation federated learning program used in conjunction with it. The dynamic weight aggregation federated learning program can be executed by the server and several clients in cooperation with each other to complete the dynamic weight aggregation federated learning method for credit risk assessment of the present invention. Among them, the client is responsible for performing local model calculations, including but not limited to: data preprocessing and model training, uploaded local model parameters and data statistics; the server is responsible for performing global model updates, including but not limited to: receiving local model parameters and data statistics uploaded by the client, calculating contribution and dynamic aggregation weights, aggregation model parameters, and distributing initialized global models and updated global model parameters; the interaction between the server and the client is limited to the transmission of model parameters and data statistics to protect data privacy. The client, server and its memory, processor, communication connection, etc. all use existing technologies, which will not be repeated here.

[0036] The following will further explain the dynamic weight aggregation federated learning method in combination with the server and the client.

[0037] The present invention provides a dynamic weight aggregation federated learning method for credit risk assessment, comprising the following steps:

[0038] Step S1: Build a credit risk model based on federated learning, initialize a global model on the server, and distribute it to multiple participating clients as the local models of the clients.

[0039] The system diagram of building a credit risk assessment model based on federated learning is as follows: Figure 1 As shown in the figure, there are roughly: ① downloading the global model; ② training the model on the local client; ③ uploading the local model; ④ calculating the contribution of each client; ⑤ dynamically aggregating the model based on the contribution. The method of dynamic weight aggregation based on the contribution of the client is as follows Figure 2 As shown in the figure, it roughly includes: calculating contribution; calculating dynamic weights; aggregation. The model includes a server and multiple client devices. The server is responsible for initializing the global model and distributing it to all participating clients. The client device is responsible for performing local calculations, including data preprocessing, model training, and calculation of local data statistics. The server is also responsible for updating the global model, including receiving model parameters and data statistics uploaded by the client, calculating client contribution, dynamically adjusting weights and aggregating model parameters, and distributing updated global model parameters.

[0040] Step S2: Each client calculates the statistics of its local data in the first round of updates.

[0041] Each client calculates the statistics of its local data in the first round of updates. For the i-th client, its data set Contains n i samples, and the feature vector of each sample is X i,j (j=1,2,...,n i ),pass Calculate the mean μ of the jth feature of its data on the i-th client i,j ,pass Calculate the variance σ of the jth feature of its data on the i-th client i,j , and upload it to the server after the calculation is completed.

[0042] Step S3: The server calculates the statistics of the global data in the first round of updates, and calculates it with the statistics of the local data of each client to obtain the data distribution difference between each client and the server.

[0043] The server calculates the statistics of global data in the first round of updates, and calculates the statistics of each client's local data to obtain the data distribution difference between each client and the server. contains N samples, and the feature vector of each sample is X g,k (k=1,2,...,N), through Calculate the mean μ of the kth feature of the global data g,k ,pass Calculate the variance σ of the kth feature of the global data g,k The mean difference between the data of client i and the server is defined as Δμ i , the data variance difference between client i and the server is defined as Δσ i , the expression is as follows:

[0044]

[0045] The data mean difference and variance difference between the client and the server are added to get the distribution difference measure. The data distribution difference between client i and the server is defined as ψ i , the expression is as follows:

[0046]

[0047] Among them, μ i,m is the mth feature mean of the i-th client data, μ g,m is the mth feature mean of the global data, σ i,m is the mth feature variance of the i-th client data, σ g,m is the mth feature variance of the global data, and M is the dimension of the feature.

[0048] Step S4: Each client trains the local model using its local data to obtain updated model parameters, and uploads the updated model parameters and the gradient of the last layer of the model to the server.

[0049] Each client trains the local model using its local data, obtains updated model parameters, and uploads the updated model parameters and the gradient of the last layer of the model to the server. Assume that the initial parameters of the global model are w t , in round t, each client i uses local data Minimize its local loss function Update the model parameters by gradient descent, the expression is as follows:

[0050]

[0051] in, is the updated local model parameter of client i, η is the learning rate, is about the global model w t Compute the gradient of the loss function on client i.

[0052] Step S5: Calculate the cosine similarity value between each client and the server based on the gradient of the last layer of the model uploaded by the client and the gradient of the last layer of the global model.

[0053] The cosine similarity between each client and the server is calculated based on the gradient of the last layer of the model uploaded by the client and the gradient of the last layer of the global model. The cosine similarity is determined by calculating the dot product and modulus length of the two vectors. The cosine similarity between the last layer gradient of the model of client i and the server is defined as The expression is as follows:

[0054]

[0055] Among them, θ i is the gradient of the last layer of the client i model, θ g is the gradient of the last layer of the server model.

[0056] Step S6: The server calculates the contribution of each client to the global model.

[0057] The server calculates the contribution of each client to the global model. By introducing a hyperparameter α, the cosine similarity and data distribution difference between each client and the server are combined to obtain the contribution of client i to the server γ i , the expression is as follows

[0058]

[0059] Among them, α (0<α<1) is a weight parameter used to adjust the relative importance of cosine similarity and data distribution difference in contribution calculation. is the cosine similarity of the last layer gradient of the client i and the server model, ψ i is the data distribution difference between client i and the server.

[0060] Step S7: Standardize the contribution of each client to obtain the weight of each client model in the global model update.

[0061] Normalize the contribution of each client to obtain the dynamic aggregation weight The calculation formula is as follows:

[0062]

[0063] Among them, γ i is the contribution of client i to the server.

[0064] Step S8: Update the global model using the dynamically adjusted weights.

[0065] The global model is updated using the dynamically adjusted weights. The server aggregates the models uploaded by the client according to the dynamic aggregation weights. The final aggregation formula is:

[0066]

[0067] Where K is the number of clients, are the model parameters uploaded by client k after the tth round of training, is the aggregate weight of the client k model parameters

[0068] Step S9: Distribute the updated global model parameters to each client for the next round of training, and repeat steps S4 to S8 for iterative optimization.

[0069] In order to verify the performance improvement and advantages of the dynamic weight aggregation federated learning method (the method of the present invention) in credit risk assessment compared with the traditional federated learning method, please refer to Figures 3 to 5 This paper uses the European credit card dataset creditcard provided by the Kaggle platform to compare with several traditional federated learning methods, namely Fedavg, FedProx, MOON and SCAFFOLD, and uses the following indicators for experimental comparison:

[0070] 1. Accuracy: It measures the proportion of correct predictions made by the model, that is, the number of samples predicted correctly by the model divided by the total number of samples.

[0071] 2. Loss: Measures the difference between the model’s predicted value and the actual value.

[0072] 3.AUC value (Area Under the Curve): The area under the receiver operating characteristic curve (ROC Curve), which measures the ability of the model to distinguish different categories.

[0073] 4. Precision: The proportion of positive categories predicted by the model that are actually positive.

[0074] 5. Recall: The proportion of positive categories that are correctly predicted by the model to be positive categories.

[0075] 6. F1 Score: The harmonic mean of precision and recall, which comprehensively considers the balance between precision and recall.

[0076] The method proposed in the present invention is implemented based on the PyTorch framework and Python 3.8. The number of clients is 5, the data batch size is set to 64, the model optimizer is SGD, the learning rate is set to 0.001, and the CrossEntropyLoss cross entropy loss function is selected to optimize the loss.

[0077] Figure 3 The accuracy difference between the method of the present invention and traditional federated learning methods (FedAvg, FedProx, MOON and SCAFFOLD) is compared. The method of the present invention reduces the number of required iterations while maintaining high accuracy, which shows that the method of the present invention has advantages in improving the accuracy of credit risk assessment.

[0078] Figure 4 The changing trends of the loss values ​​(Loss) of the method of the present invention and the traditional federated learning method during the training process are demonstrated. The method of the present invention achieves a rapid decrease in the loss value and the loss value is the lowest within a relatively small number of iterations, which proves the effectiveness of the method of the present invention in optimizing model performance.

[0079] Figure 5The performance comparison between the method of the present invention and the traditional federated learning method in terms of several key performance indicators, such as AUC value, precision, recall rate, and F1 score, is demonstrated. The method of the present invention achieves the best performance in terms of AUC value, recall rate, and F1 score. Although the SCAFFOLD algorithm performs more prominently in terms of precision, the method of the present invention maintains a high level in all key indicators. Taking all factors into consideration, the performance of the method of the present invention is more balanced and excellent. In general, the dynamic weight aggregation federated learning method of the present invention provides a more effective solution in credit risk assessment, which not only improves the predictive performance of the model, but also enhances the generalization ability and practicality of the model.

[0080] As mentioned above, the present invention aims at the problems of heterogeneity of client data distribution and dynamic changes in the model update process in federated learning, and provides a dynamic weight aggregation federated learning method for credit risk assessment, which aims to improve the accuracy and efficiency of credit risk assessment while protecting user data privacy. First, the cosine similarity of the last layer gradient of each client model and the global model is calculated to evaluate the consistency of model parameters; secondly, the distribution difference is quantified by calculating the difference in data statistics between each client data and the global data to evaluate the diversity of data; further, a hyperparameter is used to combine cosine similarity and data distribution difference to obtain a contribution that comprehensively considers model parameter similarity and data distribution difference; finally, the weight of each client in the global model update is dynamically adjusted according to the contribution of each client, so that the model pays more attention to clients with higher data quality, thereby improving the overall model performance. The federated learning method proposed in the present invention is particularly suitable for processing non-independent and identically distributed data and optimizing the generalization ability of the model, and is suitable for credit risk assessment in the financial field.

[0081] It should be pointed out that the examples of the above-mentioned embodiments can be preferably combined with one or more of them according to actual needs, and multiple examples use a set of drawings to illustrate the combined technical features, which will not be described one by one here.

[0082] The above descriptions are detailed descriptions and illustrations of the preferred embodiments of the present invention, but these descriptions are not intended to limit the scope of protection required by the present invention. All equivalent changes or modified modifications accomplished under the technical teachings suggested by the present invention should fall within the scope of patent protection covered by the present invention.

Claims

1. A dynamic weight aggregation federated learning method for credit risk assessment, characterized in that: Includes the following: Step 10: Build an initialized global model for credit risk assessment based on federated learning, and perform a model distribution operation to distribute the initialized global model to all clients as local models; Step 20: Calculate the data distribution difference ψ between the global data and the local data in the first round of update i , including the following: Receive the local data statistics in the first round of updates, which are specifically the mean μ of the jth feature of the data calculated on the i-th client. i,j and variance σ i,j ; Calculate the global data statistics in the first round of updates, which is specifically the mean μ of the kth feature of the global data g,k and variance σ g,k ; Calculate the data distribution difference ψ between the local data and the global data i , the data distribution difference ψ i Specifically, the data mean difference Δμ i and variance difference Δσ i sum; Step 30: Calculate the contribution γ of each local model to the global model after training the model. i , including the following: Receive the updated gradient θ of the last layer of the local model i ; According to the gradient θ of the last layer of the local model i and the gradient θ of the last layer of the global model g , calculate the cosine similarity According to cosine similarity and data distribution difference ψ i , calculate the contribution of each local model to the global model γ i ; Step 40: Normalize the contribution to obtain the dynamic aggregation weight of each local model Step 50: Aggregate the received updated local model parameters using dynamic aggregation weights to update the global model; Step 60: Execute the model distribution operation to distribute the updated global model to each client as a local model to complete this round of training; then, enter the next round of training, repeat the aforementioned steps 30 to 50 for iterative optimization.

2. The dynamic weight aggregation federated learning method for credit risk assessment according to claim 1, characterized in that: In step 20, the data distribution difference ψ i The calculation formula is Among them, μ i,m is the mth feature mean of the i-th client data, μ g,m is the mth feature mean of the global data, σ i,m is the mth feature variance of the i-th client data, σ g,m is the mth feature variance of the global data, and M is the dimension of the feature.

3. The dynamic weight aggregation federated learning method for credit risk assessment according to claim 1, characterized in that: In step 30, the cosine similarity The calculation formula is 4. The dynamic weight aggregation federated learning method for credit risk assessment according to claim 1, characterized in that: In step 30, the contribution γ i The calculation formula is Among them, α (0<α<1) is a weight parameter used to adjust the relative importance of cosine similarity and data distribution difference in contribution calculation. is the cosine similarity between the last layer gradient of the i-th local model and the global model, ψ i is the data distribution difference between the i-th local model and the global model.

5. The dynamic weight aggregation federated learning method for credit risk assessment according to claim 1, characterized in that: In step 40, the dynamic aggregation weight The calculation formula is Among them, γ i is the contribution of local model i to the global model.

6. The method for dynamic weight aggregation federated learning for credit risk assessment according to claim 1, characterized in that: In step 50, the aggregate calculation formula is Where K is the number of local models, are the local model parameters uploaded by local model k after the tth round of training. is the dynamic aggregation weight of the local model k.

7. A dynamic weight aggregation federated learning device for credit risk assessment, characterized in that: It includes a processor, a memory and a dynamic weight aggregation federated learning program; the dynamic weight aggregation federated learning program is stored in the memory and is configured to be executed by the processor to complete the dynamic weight aggregation federated learning method as described in any one of claims 1-6.

8. A dynamic weight aggregation federated learning system for credit risk assessment, characterized by: It includes a server, several clients and a dynamic weight aggregation federated learning program used in conjunction with them. The dynamic weight aggregation federated learning program can be executed by the server and several clients in conjunction with each other to complete the dynamic weight aggregation federated learning method as described in any one of claims 1-6.

9. A dynamic weight aggregation federated learning system for credit risk assessment according to claim 8, characterized in that: The client is responsible for performing local model calculations, including but not limited to: data preprocessing and model training, uploaded local model parameters and data statistics; The server is responsible for executing global model updates, including but not limited to: receiving local model parameters and data statistics uploaded by the client, calculating contribution and dynamic aggregation weights, aggregating model parameters, and distributing initialized global models and updated global model parameters; the interaction between the server and the client is limited to the transmission of model parameters and data statistics to protect data privacy.

Images