Personalized Federated Learning Method with Dual Perception of Metadata and Image Features

Abstract

The present invention belongs to the field of intelligent healthcare, and proposes a personalized federated learning method with dual perception of metadata and image features. By constructing an intelligent diagnosis model, users can obtain fast and accurate diagnosis services through this model. Secondly, a personalized federated learning algorithm is designed to customize a high-quality personalized image classification model for each edge node by using the knowledge learned from other edge nodes with greater contribution. At the same time, a Naive Bayes classifier is designed to classify the metadata of patients. Then, the diagnosis results of images and metadata are aggregated with different weights to improve the accuracy of intelligent diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent healthcare, and specifically relates to a personalized federated learning method for co-perceiving metadata and image features for intelligent healthcare. Background Art

[0002] In recent years, with the continuous development of technologies such as the Internet of Things, artificial intelligence, and big data, medical services have gradually become intelligent. Computer-Aided Diagnosis (CAD) has been widely used in the field of intelligent healthcare to provide effective diagnostic decision support for clinicians.

[0003] Deep learning has shown great potential as an auxiliary tool in computer-aided diagnosis and has become feasible in medical image analysis, such as skin disease and mild cognitive impairment classification. In particular, Convolutional Neural Network (CNN) has been proven to be very effective for medical image classification and segmentation tasks. However, most of the work relies entirely on the features of medical images for decision-making, ignoring the clinical information of patients, and thus may limit the accuracy of diagnosis.

[0004] In real medical scenarios, doctors often comprehensively consider the medical images and clinical information of patients when making a diagnosis. For example, when diagnosing skin diseases, dermatologists not only analyze dermoscopic images based on experience but also comprehensively consider the clinical text information of patients, such as the patient's age, lesion location, main symptoms, etc. Currently, there are mainly two strategies for the fusion of images and metadata. One is the fusion at the feature level, which fuses the features extracted by neural networks or manually. The other is to classify and make decisions on patient metadata and images through different models respectively, and then aggregate the classification results at the decision-making level. Although the above fusion methods have improved the classification accuracy to a certain extent, there is still room for improvement to achieve better performance; at the same time, the above methods use a centralized method to extract features and store the information of different institutions or users in the same server, which poses a threat to the privacy of patients.

[0005] To learn knowledge from multiple medical institutions while protecting privacy, CNN can be trained under the framework of Federated Learning (FL). FL is a decentralized machine learning framework proposed by Google in 2017 and has been widely used in medical image classification.

[0006] In practical applications, the training data across different user terminals is often non-independent and identically distributed, which is also known as data heterogeneity. When the data distribution differences between user terminals are large, if a user terminal directly obtains the knowledge learned from other user terminals, the performance of the user terminal model will be greatly reduced. Therefore, relevant researchers have proposed personalized federated learning, which, while solving the problems brought by data heterogeneity, customizes a personalized federated learning model for each client to make the global federated model adapt to each distributed user terminal. However, the personalized operations of most personalized federated learning schemes are mainly carried out locally, and the global model on the server still adopts the direct aggregation method. Therefore, in the case of statistical heterogeneity, direct aggregation will reduce the performance of the model, resulting in a low accuracy of intelligent diagnosis. Summary of the Invention

[0007] To solve the above technical problems, the present invention provides a personalized federated learning method with dual perception of metadata and image features. It constructs an intelligent diagnosis model, and the personalization is carried out on the server side. By using the knowledge learned from other edge nodes with greater contribution, a high-quality personalized image classification model is customized for each edge node. At the same time, a Naive Bayes classifier is designed to classify the patient's metadata, and the diagnostic results of the image and metadata are aggregated with different weights to improve the accuracy of intelligent diagnosis.

[0008] The personalized federated learning method with dual perception of metadata and image features according to the present invention includes the following steps:

[0009] Step 1: The user uploads the lesion image and patient metadata to the edge layer and requests to obtain diagnostic result information;

[0010] Step 2: The edge nodes in the edge layer use the metadata set M i and the image data set D i in the data infrastructure to perform local training on the metadata classifier and the image classifier respectively; each edge node uploads the trained image classifier model to the cloud layer for personalized aggregation to obtain the aggregation model of each edge node in the edge layer, and continuously iterates until convergence;

[0011] Step 3: Edge node i, , uses the trained image classifier and metadata classifier to predict the user-uploaded lesion image and patient metadata respectively, and then performs a weighted sum operation on the prediction results of the lesion image and metadata to obtain the final diagnostic result and return it to the user.

[0012] Further, in step 2, in each round of communication, the contribution degree of each edge node to each edge node is calculated through parameterized knowledge transfer, and then the aggregation coefficient of each edge node is calculated; according to the different aggregation coefficients, the aggregation model is assigned to the edge nodes; after T rounds of iteration, each edge node finally obtains a high-quality aggregation model.

[0013] Further, the specific content of step 2 is as follows:

[0014] Suppose there are N edge nodes in total, and i represents the i-th edge node, that is ; in addition, each edge node i has a local private labeled image dataset , where I i represents the number of samples in D i , x ij , y ij respectively represent the data and label of the j-th lesion image sample; a public dataset is configured in the cloud server, where R represents the total number of lesion image samples in the public dataset, b r , z r respectively represent the data and label of the r-th lesion image sample;

[0015] In the t-th round of communication process, the edge node i downloads the aggregation model calculated by the server in the (t - 1)-th round to the local, and then updates the aggregation model i using the local private dataset D ; the cross-entropy loss function and the optimization formula are as follows:

[0016]

[0017]

[0018] Among them, is the predicted value of the lesion image, is the learning rate, and the updated local model is obtained; subsequently, the edge node i participating in this round of communication uploads the local model to the server.

[0019] Further, in the server, the models of the edge nodes respectively predict the samples of the public dataset D p , and the prediction result of the sample b r is ; the n-th weight coefficient c in of the aggregation model of the edge node i represents the contribution degree of the edge node n to the edge node i, and its calculation formula is:

[0020] ,

[0021] ,

[0022] where ε is a constant used to avoid the denominator being zero; is the KL divergence; thus the larger it is, the smaller the difference between the prediction result of the edge node n and the edge node i; similarly, when is larger, the difference between the prediction results of the edge node i and the edge node n is smaller, which indicates that the heterogeneity between the edge node i and the edge node n is smaller, so the edge node i can learn more knowledge from the edge node n;

[0023] Then, the weight coefficient is normalized to obtain:

[0024] ,

[0025] Finally, the aggregation model of the edge node i in the t-th round is:

[0026] ,

[0027] ;

[0028] After T rounds of iteration, the model on the edge node i converges; then the prediction result for the medical image x ij is:

[0029] .

[0030] Furthermore, step 3 is specifically:

[0031] Assume that the patient metadata dataset on the edge node i is , where and y ij represent the data and label of the j-th metadata sample respectively, T i represents the number of patient metadata features, I i is the number of samples in M i ; in addition, the total number of label types in M i is K i ; then the maximum likelihood estimate of the prior probability of K i types of diseases is:

[0032] ,

[0033] ;

[0034] where I is an indicator function;

[0035] Suppose the t-th feature of the metadata sample is a discrete variable, and its value set is: , where S t is the total number of values of the t-th feature. Then, given the classification result is k, the conditional probability that the feature takes the value h tl , l = 1, 2, …, Sj is calculated as:

[0036]

[0037] For a continuous feature , consider the probability function as follows:

[0038]

[0039] where and represent the variance and mean of the metadata samples of the k-th class for the t-th feature, respectively;

[0040] For a given metadata sample , according to the naive Bayes feature conditional independence assumption, the conditional probability is obtained:

[0041]

[0042] According to the naive Bayes formula, for each class k, the denominator is the same. Further, the probability that the classification result of the input is k is:

[0043] ,

[0044] ,

[0045] Subsequently, the result predicted by the naive Bayes classifier for the patient metadata is further normalized to :

[0046] ,

[0047] ;

[0048] To improve the diagnostic accuracy by simultaneously using the patient's medical image and clinical text information, the prediction result of the patient metadata and the prediction result ij of the medical image x are aggregated; the final prediction result is:

[0049] ,

[0050] Among them, then the final classification result is:

[0051] .

[0052] The beneficial effects of the present invention are as follows:

[0053] (1) The present invention constructs a personalized federated learning model that jointly perceives metadata and image features for intelligent healthcare. The model consists of a user layer, an edge layer, a cloud layer, and a data infrastructure. Compared with manual consultation, users can obtain fast and accurate diagnoses through this model;

[0054] (2) The present invention designs a personalized federated learning based on parameterized knowledge transfer, which makes full use of the knowledge learned from other edge nodes with greater contributions to customize high-quality personalized classification models for each edge node, better training the image classifier in the edge node, and improving the accuracy of image classification;

[0055] (3) The present invention designs a Naive Bayes classifier to classify patient metadata. Based on the joint perception of metadata and image features, the diagnosis results of medical images and patient metadata are aggregated with different weights to improve the accuracy of intelligent diagnosis;

[0056] (4) The simulation results show that the algorithm proposed by the present invention has a higher classification accuracy compared with related algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a schematic diagram of the personalized federated learning model that jointly perceives metadata and image features designed by the present invention;

[0058] Figure 2 is a schematic diagram of the personalized federated learning framework designed by the present invention;

[0059] Figure 3 In the (a) small figure of is a schematic diagram of the change of the balanced accuracy of the present invention with the number of communications; in the (b) small figure is a schematic diagram of the change of the loss of the present invention with the number of communications;

[0060] Figure 4 is a schematic diagram of the change of the accuracy of the present invention under different aggregation ratios ;

[0061] Figure 5 is a schematic diagram of the confusion matrix of the image feature diagnosis algorithm; a schematic diagram of the confusion matrix of the co-perception diagnosis algorithm of metadata and image features;

[0062] Figure 6 is a flowchart of the method of the present invention. Specific embodiments

[0063] In order to make the content of the present invention easier to be clearly understood, the present invention will be further described in detail below according to specific embodiments in conjunction with the accompanying drawings.

[0064] As Figure 6 shown, the personalized federated learning method for dual perception of metadata and image features according to the present invention includes the following steps:

[0065] S1. A personalized federated learning model for co-perception of metadata and image features for intelligent healthcare is constructed, and the model consists of four components: a user layer, an edge layer, a cloud layer, and a data infrastructure. Figure 1 The relationships between these layers are presented, and the function definitions of each layer are as follows:

[0066] User layer: This layer consists of C users, mainly divided into two categories: patients and clinicians, and the two types of users have different diagnostic requirements. Specifically, patients send service requests containing lesion images taken by smartphones or cameras and personal information such as age, onset location, and main symptoms to the edge layer to obtain diagnostic results. Similarly, clinicians send medical images and corresponding patient information to the edge layer through a dedicated computer to obtain auxiliary diagnostic results and improve the accuracy of patient diagnosis;

[0067] Edge layer: This layer consists of multiple edge nodes, and each edge node contains an image classifier and a metadata classifier; the edge nodes mainly respond to service requests and return the automatic diagnostic results to the users. Each edge node has two functions: training the diagnostic network and providing diagnostic services; (1) Training the diagnostic network: Before providing diagnostic services, the edge node obtains patient metadata M i from the data infrastructure for training the metadata classifier (i.e., the naive Bayes classifier); at the same time, edge node i will use the image data D iThe trained image classifier model (i.e., the model trained by personalized federated learning) is uploaded to the cloud for personalized aggregation, and then the cloud returns the aggregated model to the edge nodes; the above process is iterated until the image classifier model converges. (2) Provide diagnostic services: When edge node i receives a service request from a user, the image classifier and the metadata classifier respectively make predictions on the medical image and the patient metadata, and then perform a weighted sum operation on the image and metadata results to obtain the final diagnostic result. Subsequently, the diagnostic result is returned to the user. At the same time, after the edge node completes the automatic diagnosis, it sends all diagnostic samples to the data infrastructure and stores the high-confidence samples with labeled results to enrich the training dataset;

[0068] Cloud: The cloud consists of a cloud server with powerful computing capabilities and dataset D p which is composed. The cloud server receives the image classifier model from the edge nodes, and then uses dataset D p to perform personalized aggregation based on parametric knowledge transfer and return the aggregated model to the edge nodes. The above process is iterated until the model converges;

[0069] Data infrastructure: The data infrastructure is mainly used to store the labeled image data D i and metadata M i . Generally speaking, there are two data sources to maintain the dataset. One is the medical images and corresponding patient metadata from various medical institutions, and the other is the data uploaded by users for diagnosis.

[0070] S2. In order to make full use of the knowledge learned from other more contributing edge nodes and customize high-quality classification models for each edge node, the present invention designs a personalized federated learning algorithm based on parametric knowledge transfer, that is, in each round of communication, the contribution degree of all edge nodes to each edge node is calculated through parametric knowledge transfer, and then the aggregation coefficient of each edge node is calculated. According to the different aggregation coefficients, the aggregated model is allocated to the edge nodes. After T rounds of iteration, each edge node can obtain a high-quality personalized federated learning model.

[0071] Suppose there are N edge nodes in total, and i represents the i-th edge node, that is . In addition, for each edge node i, there is a locally private labeled image dataset , where I i represents the number of samples in D i , x ij , y ij respectively represent the data and label of the j-th lesion image sample; a public dataset is configured in the server of the cloud , where R represents the total number of lesion image samples in the public dataset, and b r , z r represent the data and label of the r-th lesion image sample respectively. The personalized federated learning framework is as shown in Figure 2 .

[0072] During the t-th round of communication, the edge node i downloads the aggregated model calculated by the server in the (t - 1)-th round to the local, and then updates the aggregated model i using the local private dataset D . The cross-entropy loss function and the optimization formula are as follows:

[0073]

[0074]

[0075] where is the predicted value of the lesion image, is the learning rate.

[0076] Subsequently, the edge node i participating in this round of communication uploads the local model to the server. In the server, the models of the edge nodes (n is any edge node, and the edge node n and the edge node i can be the same node or different nodes) respectively predict the samples of the public dataset D p , and the prediction result for the sample b r is . Different from traditional federated learning, the personalized federated learning designed in the present invention customizes the aggregation weight coefficients for each edge node, enabling each edge node to learn more from the edge nodes with greater contributions and less from the edge nodes with smaller contributions, rather than allocating the same aggregation weight coefficients to each edge node according to the sample ratio. For example, the n-th weight coefficient c in of the aggregated model of the edge node i represents the contribution degree of the edge node n to the edge node i, and its calculation formula is:

[0077] ,

[0078] ,

[0079] where ε is a constant used to avoid the case where the denominator is zero; is the KL divergence; therefore the larger it is, the smaller the difference between the prediction result of the edge node n and the edge node i; similarly, when The larger it is, the smaller the difference between the prediction results of the edge node i and the edge node n, which indicates that the heterogeneity between the edge node i and the edge node n is smaller. Therefore, the edge node i can learn more knowledge from the edge node n.

[0080] Then, the weight coefficient is normalized to obtain:

[0081] ,

[0082] Finally, the aggregation model of the edge node i in the t-th round is:

[0083] ,

[0084] ;

[0085] After T rounds of iteration, the model on the edge node i converges; then ij the prediction result for the medical image x

[0086] .

[0087] S3. In order to utilize the patient's metadata to assist intelligent diagnosis, the present invention designs a Naive Bayes classifier to diagnose lesions through the patient's metadata (such as age, gender, disease location, and basic symptoms).

[0088] Assume that the patient metadata dataset on the edge node i is , where and y ij represent the data and label of the j-th metadata sample respectively, T i represents the number of patient metadata features, and I i is the number of samples in M i ; in addition, the total number of label types in M i is K i ; then the maximum likelihood estimate of the prior probability of the K i types of diseases is:

[0089] ,

[0090] ;

[0091] where I is the indicator function.

[0092] Assume that the t-th feature is a discrete variable, and the set of possible values is: , where S t is the total number of values of the t-th feature. Then, given that the classification result is k, the feature takes the value of h tl (l = 1, 2, …, S J ) the conditional probability is calculated as:

[0093]

[0094] For continuous features , consider the probability function as follows:

[0095]

[0096] where and represent the variance and mean of the k-th class metadata sample of the t-th feature respectively.

[0097] For a given metadata sample , according to the naive Bayes feature conditional independence assumption, the conditional probability is obtained:

[0098]

[0099] According to the naive Bayes formula, for each class k, the denominator is the same. Further, the probability that the classification result of the input is k :

[0100] ,

[0101] ,

[0102] Subsequently, the result predicted by the naive Bayes classifier for the patient metadata is further normalized to :

[0103] ,

[0104] ;

[0105] To improve the diagnostic accuracy by simultaneously using the patient's medical image and clinical text information, the prediction result of the patient metadata and the prediction result ij of the medical image x are aggregated; the final prediction result is:

[0106] ,

[0107] where , then the final classification result is:

[0108] .

[0109] Finally, the specific implementation verification of the method of the present invention proves the effectiveness and superiority of the algorithm proposed by the present invention. The dataset used in the experiment is the PAD-UFES-20 dermatosis dataset. Figure 3 The small graph (a) of shows the change results of the balanced accuracy (BACC) with the number of communication rounds under different learning rates (LR) when the number of edge nodes N = 3. It can be found that as the number of communication rounds increases, the BACC continuously increases and then tends to converge. Similarly, in Figure 3 the small graph (b) of, the loss (LOSS) continuously decreases with the increase of the number of communication rounds and gradually converges to 0. This proves the effectiveness of the personalized federated learning based on image features proposed by the present invention.

[0110] In addition, Table 1 shows the change results of ACC, BACC, and Sensitivity with the increase of the number of edge nodes N. It can be observed that as N increases, the above three indicators are significantly improved. This is because as more edge nodes participate in the training, the edge nodes can learn more knowledge from each other, solve the problem of local sample imbalance, and thus improve the performance of the model.

[0111] Table 1 Performance Results

[0112]

[0113] To further verify the superiority of the personalized federated learning algorithm based on image features of the present invention, the model of the present invention is compared with three existing federated learning models: Federated Averaging (FedAvg), Clustered Federated Learning (CFL), and Personalized Retrogress Resilient-Federated Learning (PRR-FL).

[0114] Table 2 gives the prediction results of ACC, BACC, and Sensitivity of the above four federated learning model methods. According to Table 2, it can be observed that the method of the present invention has obvious advantages over other methods in all three indicators. In addition, the BACC values of other methods are all lower than ACC, while the method proposed by the present invention is the opposite, which indicates that the method of the present invention successfully solves the influence brought by data sample imbalance, thus improving the BACC value.

[0115] Table 2 Performance Results - Federation Mechanism

[0116]

[0117] Figure 4The ACC values of the metadata results and the image results are given, as well as the aggregation of the metadata and image results under different α. It can be seen that when α = 0, it is the classification accuracy of the image results. As α increases, the ACC of the aggregated results continuously improves until it reaches a peak when α = 0.48, indicating the effectiveness of the aggregation. Subsequently, as α continues to increase, the ACC after aggregation gradually decreases until it equals the accuracy of the metadata results when α = 1. This is because the image classifier trained by image feature personalized federated learning has a higher classification accuracy than the metadata classifier. Therefore, as the weight of the image results decreases, the ACC after aggregation gradually becomes lower than that of only the image results. Therefore, the present invention sets α = 0.48 as the optimal aggregation.

[0118] Figure 5 The small figure (a) shows the confusion matrix of the image feature diagnosis algorithm, which only considers image features and does not fuse with metadata features; the correct classification ratio of skin cancer by 3 edge nodes reaches over 95%. Figure 5 The small figure (b) shows the confusion matrix of the co - perception diagnosis algorithm of metadata and image features, which achieves a higher accuracy rate of over 98% in skin cancer classification than using the image feature diagnosis algorithm. Thus, aggregating the metadata results and the image results can effectively improve the accuracy of skin cancer diagnosis.

[0119] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above - mentioned embodiments. Within the scope of knowledge possessed by those of ordinary skill in the art, various changes can be made without departing from the purpose of the present invention.

Claims

1. A personalized federated learning method with dual perception of metadata and image features, characterized in that, the method includes the following steps: Step 1: The user uploads the lesion image and patient metadata to the edge layer and requests to obtain the diagnostic result information; Step 2: Edge nodes in the edge layer use the metadata set M i and the image data set D i to locally train the metadata classifier and the image classifier respectively; Each edge node uploads the trained image classifier model to the cloud layer for personalized aggregation to obtain the aggregation model of each edge node in the edge layer, and continuously iterates until convergence; specifically: Suppose there are N edge nodes in total. Let i represent the i-th edge node, that is, i ∈ {1, 2,..., N}. In addition, each edge node i has a locally private labeled image dataset D i ={x ij , y ij | j = 1, 2,..., I i}, where I i represents the number of samples in D i , x ij , y ij represent the data and label of the j-th lesion image sample respectively. A public dataset D p ={(b r , z r ) | r = 1, 2,..., R} is configured in the cloud server, where R represents the total number of lesion image samples in this public dataset, and b r , z r represent the data and label of the r-th lesion image sample respectively; During the t-th round of communication, edge node i downloads the aggregated model calculated by the server in the (t - 1)-th round to the local, and then updates the aggregated model using the locally private dataset D i The cross-entropy loss function and the optimization formula are as follows:​​ Among them, is the predicted value of the lesion image, η is the learning rate, and the updated local model is obtained Subsequently, the edge node i participating in this round of communication will send the local model to the server; In the server, the models of edge nodes \(n\in\{1,2,\cdots,N\}\) respectively make predictions on the samples of the common dataset \(D\) p , and the prediction result for the sample \(b\) r is \(y' n \); the \(n\)-th weight coefficient \(c\) of the aggregated model of edge node \(i\) r represents the contribution degree of edge node \(n\) to edge node \(i\), and its calculation formula is: in ​ where ε is a constant used to avoid the denominator being zero; D KL () is the KL divergence; Then, normalize the weight coefficients to obtain: Finally, the aggregation model of edge node i in the t-th round is: After T rounds of iteration, the model on edge node i converges; then the prediction result for medical image x ij is as follows: Step 3: Edge node i, i ∈ {1, 2,..., N}, uses the trained image classifier and metadata classifier to respectively predict the lesion image and patient metadata uploaded by the user, and then performs a weighted sum operation on the prediction results of the lesion image and metadata to obtain the final diagnostic result and return it to the user.

2. The personalized federated learning method with dual perception of metadata and image features according to claim 1, characterized in that, In step 2, in each round of communication, calculate the contribution degree of all edge nodes to each edge node through parameterized knowledge transfer, and then calculate the aggregation coefficient of each edge node; according to the different aggregation coefficients, allocate the aggregation model to the edge nodes; after T rounds of iteration, each edge node finally obtains an aggregation model.

3. The personalized federated learning method with dual perception of metadata and image features according to claim 1, characterized in that, Step 3 is specifically: Suppose the patient metadata dataset on edge node i is where and y ij represent the data and label of the j-th metadata sample respectively, T i represents the number of patient metadata features, and I i is the number of samples in M i ; in addition, the total number of label types in M i is K i ; then the maximum likelihood estimate of the prior probability of K i types of diseases is: where I is the indicator function; Suppose the t-th feature X of the metadata sample t is a discrete variable, and the value set is: where S t is the total number of values of the t-th feature. Then, given the classification result is k, the conditional probability that the feature X t takes the value h tl , l = 1, 2,..., S j is calculated as: For the continuous feature X t , consider the probability function as follows: where δ kt and μ kt represent the variance and mean of the k-th metadata sample of the t-th feature, respectively; For a given metadata sample The conditional probability is obtained based on the feature conditional independence assumption of Naive Bayes: According to the Naive Bayes formula, for each class k, the denominator is the same. Further, obtain the probability that the classification result of the input is k Subsequently, patient metadata The results predicted by the Naive Bayes classifier are further standardized to To improve the accuracy of diagnosis by simultaneously utilizing the medical images and clinical text information of a patient, the patient metadata prediction result and the prediction result of the medical image x ij are aggregated; the final prediction result is: ​ where 0 < α < 1, then the final classification result is:

Images