Adaptive Privacy-Preserving Method and Device for Heterogeneous Federated Learning Based on Momentum Optimization

Abstract

The present invention belongs to the technical field of privacy protection for federated learning, and more specifically, relates to an adaptive privacy-preserving method and device for heterogeneous federated learning based on momentum optimization. The method includes: receiving the feature extractor parameters broadcast by the central server, calculating the local gradient of the current iteration round based on the local dataset, and introducing a momentum mechanism in the local gradient update for the classifier part; applying the gradient descent algorithm to update the local model parameters using the calculated local gradient; for the feature extractor part, calculating the Fisher information matrix; normalizing each layer of Fisher information and screening strategies; then performing differential privacy processing, uploading it to the central server for parameter aggregation, and broadcasting the updated global model parameters back to each local client for the next round of training. The present invention solves the negative impact of the differential privacy mechanism on model training and the problems of slow model convergence and poor privacy protection effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of privacy protection in federated learning, and more specifically, relates to an adaptive privacy-preserving method and device for heterogeneous federated learning based on momentum optimization. Background Art

[0002] Federated learning is a distributed machine learning paradigm in which multiple clients train models locally and perform parameter aggregation under the coordination of a central server to build a global model or personalized local models. This method reduces the privacy risks associated with centralized data storage while ensuring model performance. However, research has shown that even without direct data sharing, federated learning may still leak users' sensitive information, exposing them to potential privacy threats. Therefore, introducing the differential privacy mechanism has become an important means of protecting data security in federated learning.

[0003] Chinese invention patent document CN116050509A discloses a clustering federated learning method based on momentum gradient descent, proposes a clustering method based on minimizing the loss function, and improves the stochastic gradient descent (SGD) algorithm in the local update stage of federated learning by adopting an optimization algorithm of momentum gradient descent. In the global update stage, gradient averaging and model averaging are proposed; the clustering federated learning algorithm based on momentum gradient descent (CFL-MGD) of the present invention not only obtains theoretical guarantees of exponential convergence in convex settings, but is also more efficient than other algorithms in non-convex settings such as neural networks; the framework structure is simple and the algorithm is efficient, combining clustering and polyak's momentum.

[0004] The core idea of differential privacy is to introduce random noise during the model update process to reduce the possibility for attackers to infer the original data through parameters, thereby improving the privacy protection ability. In the federated learning framework, differential privacy is mainly achieved by injecting noise into the model parameters or gradients uploaded by clients. However, this noise will inevitably affect the model optimization process, resulting in a decrease in the convergence speed and a reduction in model accuracy. Therefore, how to find the optimal balance between privacy protection and model performance remains a key challenge. In addition, federated learning usually faces the problem of data heterogeneity, that is, the data distributions of different clients are inconsistent (Non-IID). Traditional federated learning methods use a unified global model and are difficult to adapt to the data distribution differences of all clients. Summary of the Invention

[0005] The present invention aims to overcome at least one defect of the above-mentioned prior art, and provides an adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization to achieve a better trade-off between privacy protection and model performance, and to solve the negative impact of the differential privacy mechanism on model training and the problems of slow model convergence and poor privacy protection effect.

[0006] The detailed technical solution of the present invention is as follows:

[0007] An adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization, the method comprising:

[0008] S1: The client receives the feature extractor parameters broadcast by the central server, introduces a momentum mechanism into the local loss function, and calculates the local gradient of the current iteration round based on the local dataset;

[0009] S2: Apply the gradient descent algorithm to update the local model parameters using the calculated local gradient;

[0010] S3: For the feature extractor parameters, calculate the Fisher information matrix, normalize the Fisher information of each layer, and identify and sparsify the parameters that contribute less to the loss function based on the parameter importance screening strategy to reduce the impact of redundant parameters while maintaining the model performance;

[0011] S4: Perform differential privacy processing on the feature extractor parameters after Fisher information screening and sparsification, then upload them to the central server for parameter aggregation, and broadcast the updated global model parameters back to each local client for the next round of training.

[0012] Further, the S1 specifically includes:

[0013] S11. Improve the classifier part based on the momentum mechanism, and optimize the loss function of the classifier part to:

[0014] (1);

[0015] In formula (1), is the cross-entropy function, represents the feature extractor parameters and classifier parameters in the r-th local iteration of the t-th global iteration of the i -th client, represents the momentum coefficient, which measures the influence degree of the momentum term on the optimization process, represents the classifier parameters in the r-th local iteration of the t-th global iteration of the i -th client, represents the classifier parameters in the (r-1)-th local iteration of the t-th global iteration of the i -th client.

[0016] For the classifier parameters using the momentum mechanism , after T global iterations, where T is the total number of global iterations, the following convergence bound is obtained:

[0017] (2);

[0018] In formula (2), E represents the expectation, represents the gradient, represents the classifier parameters of the i -th client in the t-th round of the global iteration; , represents the learning rate, represents the momentum coefficient; , represents the classifier parameters of the i -th client in the initial global iteration, represents the classifier parameters of the i -th client in the (T - 1)-th round of the global iteration, represents the total number of local iteration rounds, L represents the Lipschitz coefficient of the cross-entropy function, and m represents the variance of the batch gradient estimation of the local data;

[0019] Among them, , .

[0020] When the convergence bound is minimized, there exists an optimal , that is, the convergence effect is the best; when the learning rate is small enough, equation (4) is convex with respect to the momentum coefficient , so there exists an optimal that minimizes the convergence bound.

[0021] For the first term in formula (2), introducing the momentum mechanism can reduce the impact of the initial error on the convergence speed, thereby accelerating the optimization process. The momentum term accumulates historical gradient information during the update process, making the optimization direction smoother, reducing gradient oscillations, and ultimately improving the convergence speed.

[0022] S12. Client i uses the local dataset to calculate the local gradient of the current iteration round;

[0023] (3);

[0024] (4);

[0025] In formulas (3) - (4), represents the local gradient of the classifier in the r-th local iteration of the t-th iteration of client i ; represents the local gradient of the feature extractor in the r-th local iteration of the t-th iteration of client i ; represents the iThe feature extractor parameters in the r-th local iteration of the t-th global iteration of a client; Represents the i Classifier parameters in the r-th local iteration of the t-th global iteration of a client; Indicates the In the local dataset The Indicates the cross-entropy function The sample gradient obtained based on sample b, Indicates the client The local dataset of The total number of samples in; Represents the momentum coefficient.

[0026] Furthermore, the S3 specifically includes:

[0027] S31. For the feature extractor parameters, evaluate and screen the importance of the feature extractor parameters based on the Fisher information matrix:

[0028] The Fisher information matrix of the feature extractor parameters Is calculated as the expected value of the second derivative of the log-likelihood function, and the formula is as follows:

[0029] (5);

[0030] In formula (5), Indicates The log-likelihood function of, Is the feature extractor parameter;

[0031] S32. Normalize each layer of the Fisher information matrix, and based on the parameter importance screening strategy, identify the parameters that contribute less to the loss function;

[0032] Apply the min-max normalization process between the weight layers of the Fisher information matrix, and the normalization process of the Fisher information is as follows:

[0033] (6);

[0034] In formula (6), Indicates the In the layer The normalized Fisher information of the parameter Indicates the In the layer The Fisher information of the parameter , l Indicates the l Layer in the model.

[0035] The Fisher information matrix represents the contribution degree of parameters. By setting a threshold and selecting the Fisher information matrix greater than the threshold, this can be achieved.

[0036] S33. Sparsify the parameters that contribute less to the loss function, specifically including:

[0037] After local training, the client i obtains the updated value of the feature extractor parameters :

[0038] (7);

[0039] In formula (7), and respectively correspond to the feature extractor parameters of the i th client in the t-th round and the (t - 1)-th round.

[0040] For the parameters with small Fisher information values, set them to zero to reduce their impact on pruning. Specifically, define the sparsification mask as follows:

[0041] (8);

[0042] In formula (8), represents the th percentile of the Fisher information value, represents the mask matrix for the update of the feature extractor.

[0043] Then, in the t-th round, calculate the masked update using the element-wise Hadamard product, that is, update formula (7):

[0044] (9).

[0045] Through this process, parameters with low information content are preferentially removed, while those that contribute more to the model performance are retained. This method not only helps to maintain the model performance but also significantly reduces the storage requirements and communication costs.

[0046] Furthermore, the weight layer includes: the convolutional layer, batch normalization layer, fully connected layer, and the weight layer after the activation layer in the convolutional neural network.

[0047] Furthermore, perform differential privacy processing on the feature extractor parameters after Fisher information screening and sparsification, specifically including:

[0048] Perform clipping processing and differential privacy noise addition on the feature extractor parameters to be uploaded to obtain :

[0049] (10);

[0050] In formula (10), represents the clipping threshold of.

[0051] For the obtained by clipping , differential privacy noise addition is further performed:

[0052] (11);

[0053] In formula (11), represents the client i the feature extractor parameters after clipping and noise addition in the t-th global iteration, represents Gaussian noise with an expectation of 0 and a variance of , is a vector with all scalars equal to 1.

[0054] In another aspect of the present invention, there is provided an apparatus for an adaptive privacy-preserving method of heterogeneous federated learning based on momentum optimization, the apparatus comprising:

[0055] at least one processor;

[0056] and a memory, the memory stores instructions, when the instructions are executed by the at least one processor, causing the at least one processor to execute the above-mentioned adaptive privacy-preserving method of heterogeneous federated learning based on momentum optimization.

[0057] In another aspect of the present invention, there is also provided a computer-readable storage medium, which stores executable instructions, and when the instructions are executed, causes the machine to execute the above-mentioned adaptive privacy-preserving method of heterogeneous federated learning based on momentum optimization.

[0058] Compared with the prior art, the beneficial effects of the present invention are:

[0059] The adaptive privacy-preserving method and device for heterogeneous federated learning based on momentum optimization provided by the present invention fuse the adaptive differential privacy mechanism and the momentum optimization strategy to achieve a better trade-off between privacy protection and model performance. During the update process of the shared feature extractor, a sparsification strategy based on the Fisher information matrix is introduced to identify and prune the parameters that have little impact on the model performance, thereby effectively reducing the scale of the parameters to which noise needs to be added, reducing the negative impact of the differential privacy mechanism on model training, effectively reducing the accumulation of noise errors, while reducing the communication overhead and improving the communication efficiency of the system. During the classifier update stage, the momentum optimization strategy is adopted to alleviate the gradient fluctuation caused by noise interference and accelerate the model convergence, and it is proved that there is an optimal momentum coefficient, and the model can achieve the optimal convergence performance. Brief Description of the Drawings

[0060] Figure 1 is a flowchart of the adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization according to the present invention.

[0061] Figure 2 is a comparison diagram of the effects of the benchmark scheme and the scheme of the present invention in Embodiment 1 of the present invention. Detailed Embodiments

[0062] The present invention will be further described below with reference to the drawings and embodiments.

[0063] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0064] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0065] The embodiments in the present invention and the features in the embodiments may be combined with each other without conflict.

[0066] Embodiment 1

[0067] Refer Figure 1 , this embodiment provides an adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization, and the method includes:

[0068] S1. The local client calculates the local gradient based on the momentum mechanism: The client receives the feature extractor parameters broadcast by the central server, introduces the momentum mechanism into the local loss function, and calculates the local gradient for the current iteration round based on the local dataset.

[0069] The model parameters are divided into feature extractor parameters, i.e., shared parameters, and the classifier part. Among them, the feature extractor parameters need to be uploaded to the central server; for the classifier part, it is retained locally, that is, retained in the client, to enhance the convergence stability. The type of the local dataset is image data.

[0070] Specifically, the S1 specifically includes:

[0071] S11. Improve the classifier part based on the momentum mechanism, and optimize the loss function of the classifier part to:

[0072] (1);

[0073] In formula (1), is the cross-entropy function, represents the feature extractor parameters and classifier parameters in the r-th local iteration of the t-th global iteration of the i -th client, represents the momentum coefficient, which measures the influence degree of the momentum term on the optimization process, represents the classifier parameters in the r-th local iteration of the t-th global iteration of the i -th client, represents the classifier parameters in the (r - 1)-th local iteration of the t-th global iteration of the i -th client.

[0074] For the classifier parameters using the momentum mechanism , after T global iterations, where T is the total number of global iterations, the following convergence bound is obtained:

[0075] (2);

[0076] In formula (2), E represents the expectation, represents the gradient, represents the classifier parameters of the i -th client in the t-th global iteration; , represents the learning rate, represents the momentum coefficient; , represents the classifier parameters of the i-th client in the initial global iteration, represents the classifier parameters of the i-th client in the (T - 1)-th global iteration, represents the total number of local iterations, L represents the Lipschitz coefficient of the cross-entropy function, and m represents the variance of the batch gradient estimation of the local data. , .

[0077] When the convergence bound is minimized, there exists an optimal , that is, the convergence effect is the best; when the learning rate is small enough, Equation (2) is convex with respect to the momentum coefficient , so there exists an optimal that minimizes the convergence bound.

[0078] For the first term in Equation (2), introducing the momentum mechanism can reduce the impact of the initial error on the convergence speed, thereby accelerating the optimization process. The momentum term accumulates historical gradient information during the update process, making the optimization direction smoother, reducing gradient oscillations, and ultimately improving the convergence speed. However, although the momentum mechanism can effectively accumulate useful gradient information, it will inevitably accumulate random noise in the gradient. When the momentum coefficient is too large, the impact of historical noise will be amplified, which may lead to instability in the optimization process. Especially in mini-batch training, the gradient calculated each time is only based on a limited number of samples, so the gradient estimation will deviate from the true gradient, introducing additional random noise.

[0079] As the training progresses, this noise will continuously accumulate in the momentum update, further amplifying its impact, and then affecting the convergence stability and final performance of the model. Therefore, choosing an appropriate momentum coefficient is crucial and it is necessary to find the best balance between smooth updates and noise suppression.

[0080] S12. Client i uses the local dataset to calculate the local gradient of the current iteration round;

[0081] (3)

[0082] (4)

[0083] In Formulas (3) to (4), represents the local gradient of the classifier in the r-th local iteration of the t-th round of iteration of client i ; represents the local gradient of the feature extractor in the r-th local iteration of the t-th round of iteration of client i ; represents the parameter of the feature extractor in the r-th local iteration of the t-th global iteration of the i -th client; represents the iThe classifier parameters in the r-th local iteration of the t-th global iteration of a client; Denote the local dataset in the b-th sample; Denote the cross-entropy function The sample gradient obtained based on sample b, Denote the client local dataset the total number of samples in; Represents the momentum coefficient.

[0084] S2. Apply the gradient descent algorithm and update the local model parameters using the calculated local gradients, specifically:

[0085] After obtaining the local gradients, based on stochastic gradient descent, update the model parameters of the client i using the obtained local gradients, that is:

[0086] (12);

[0087] (13);

[0088] In formulas (12) to (13), and respectively represent the feature extractor parameters and classifier parameters in the r-th local iteration of the t-th global iteration updated by the client i ; and respectively represent the learning rates of the feature extractor parameters and classifier parameters.

[0089] S3. Sparsify the feature extractor parameters: For the feature extractor parameters, calculate the Fisher information matrix, normalize the Fisher information of each layer, and based on the parameter importance screening strategy, identify the parameters that contribute less to the loss function and sparsify them to reduce the impact of redundant parameters while maintaining the model performance.

[0090] Preferably, the S3 specifically includes:

[0091] S31. For the feature extractor parameters , in order to further reduce the bias introduced by pruning and improve the communication efficiency of federated learning (FL), evaluate and screen the importance of the feature extractor parameters based on the Fisher information matrix, and reduce the pruning bias caused by pruning by removing the parameters with low information content or negligible contribution to the model.

[0092] Specifically, the Fisher information matrix of the feature extractor parameters ​ is calculated as the expected value of the second derivative of the log-likelihood function, and the formula is as follows:

[0093] (5);

[0094] In formula (5), denotes the log-likelihood function of is the parameter of the feature extractor. To reduce the computational complexity and improve the efficiency, the diagonal elements of the Fisher information matrix are used for approximation. In many cases, the diagonal elements better reflect the influence of the model parameters than the non-diagonal elements.

[0095] S32. Normalize the Fisher information matrix of each layer, and based on the parameter importance screening strategy, identify the parameters that contribute less to the loss function and perform post-processing to reduce their influence.

[0096] To address the problem that the Fisher information may be too concentrated due to data heterogeneity, which may lead to excessive sparsification in some layers, a min-max normalization process is applied between the weight layers of the Fisher information matrix. The weight layers include: convolutional layers (Conv layers), batch normalization layers (BatchNorm), fully connected layers (Fully Connected layers, FC), and the weight layers after the activation layer in the convolutional neural network (CNN). This normalization ensures that the evaluation of the parameter importance within each layer is independent and fair, thereby reducing the bias in the pruning process. The normalization process of the Fisher information is as follows:

[0097] (6);

[0098] In formula (6), denotes the normalized Fisher information of the parameter in the -th layer, denotes the Fisher information of the parameter in the -th layer, , l denotes the l -th layer in the model.

[0099] The Fisher information matrix represents the contribution degree of the parameters. Set a threshold and select the Fisher information matrix greater than the threshold.

[0100] S33. Sparsify the parameters that contribute less to the loss function, specifically including:

[0101] After local training, client i obtains the updated value of the feature extractor parameters :

[0102] (7);

[0103] In formula (7), and respectively correspond to the feature extractor parameters of the t-th round and the (t - 1)-th round of the i th client.

[0104] For parameters with small Fisher information values, set them to zero to reduce their impact on pruning. Specifically, define the sparsification mask as follows:

[0105] (8);

[0106] In formula (8), represents the percentile of the Fisher information value, represents the mask matrix for the feature extractor update.

[0107] Then, in the t-th round, calculate the masked update using the element-wise Hadamard product, that is, update formula (7):

[0108] (9).

[0109] Through this process, parameters with low information content are preferentially removed, while those that contribute more to the model performance are retained. This method not only helps to maintain the model performance but also significantly reduces the storage requirements and communication costs.

[0110] S4. After differential privacy processing, perform parameter aggregation and iterate in this way: Perform differential privacy processing on the feature extractor parameters after Fisher information screening and sparsification, then upload them to the central server for parameter aggregation, and broadcast the updated global model parameters back to each local client for the next round of training.

[0111] Performing differential privacy processing on the feature extractor parameters after Fisher information screening and sparsification specifically includes: performing clipping processing and adding differential privacy noise to the feature extractor parameters to be uploaded to obtain :

[0112] (10);

[0113] In formula (10), represents the clipping threshold of.

[0114] For the cropped , further perform differential privacy noise addition:

[0115] (11);

[0116] In formula (11), represents the feature extractor parameters of the client i after cropping and noise addition in the t-th global iteration, represents Gaussian noise with an expected value of 0 and a variance of , is a vector with all scalar values being 1, i.e., .

[0117] Since this patent introduces sparsification processing in the S2 and S3 stages, in the S4 stage, the cropping bias is effectively reduced; or a smaller cropping threshold can be used because the variance of the Gaussian noise is regulated by , effectively reducing the impact of noise on the model performance. This strategy not only optimizes the privacy protection mechanism but also improves the convergence performance of the model while ensuring privacy.

[0118] The experiments of this embodiment are demonstrated as follows:

[0119] The CIFAR-10 dataset is a classic image classification dataset constructed by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton from the University of Toronto, Canada, and is widely used in the field of computer vision. The dataset contains 60,000 color images in 10 different categories, with 6,000 images in each category. The resolution of each image is 32×32 pixels. These categories cover common objects in daily life, such as airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. The dataset is divided into a training set and a test set, where the training set contains 50,000 images and the test set consists of 10,000 images. Each image is represented using the RGB color channels, with pixel values ranging from 0 to 255, and each image corresponds to a label between 0 and 9 to indicate its category. The CIFAR-10 dataset is commonly used in distributed learning environments, such as when multiple institutions or enterprises collaborate to train an image classification model without directly sharing the original data. In this scenario, each participant uses its own data subset for local training and exchanges model update information through methods such as federated learning instead of transmitting the original image data. To enhance data privacy protection, differential privacy technology can be introduced to add noise during the parameter exchange process, thereby reducing the risk of sensitive information leakage. This privacy protection mechanism has important value in applications such as intelligent monitoring, autonomous driving object recognition, and augmented reality, as it can not only ensure data security but also improve model performance.

[0120] This embodiment conducts experiments on the CIFAR10 dataset. In the experiment, this patent sets the number of clients to 6 and the number of local iteration rounds , and the learning rate . For the momentum parameter , according to formula (2), search on 0 - 1 with a step size of 0.05, and find that the optimal = 0.05.

[0121] The results of the comparative experiment are as Figure 2 shown, where:

[0122] Baseline method 1 is the FedDPA algorithm, which screens personalized parameters in the broadcast stage and combines an adaptive constraint strategy;

[0123] Baseline method 2 is the DPFL - BL algorithm, a method that combines regularization and sparsification, while improving the generalization ability to alleviate the problems brought by unbalanced data distribution and differential privacy clipping;

[0124] Baseline method 3 is the DP - FedAvg algorithm, that is, introducing a differential privacy mechanism on the classical FedAvg scheme to enhance the data privacy protection ability;

[0125] Baseline method 4 is the PPSGD algorithm, which improves the model adaptability through a personalized privacy protection strategy while ensuring privacy security.

[0126] Under the same privacy protection level, the scheme of the present invention improves the accuracy by 3.6%, 2.5%, 6.5% and 6.3% respectively compared with FedDPA, DPFL - BL, DP - FedAvg and PPSGD.

[0127] Embodiment 2

[0128] This embodiment provides a device for implementing an adaptive privacy - preserving method for heterogeneous federated learning based on momentum optimization. The device includes:

[0129] At least one processor;

[0130] And a memory that stores instructions. When the instructions are executed by the at least one processor, the at least one processor executes the above - mentioned adaptive privacy - preserving method for heterogeneous federated learning based on momentum optimization.

[0131] In this embodiment, the electronic device includes, but is not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.

[0132] Embodiment 3

[0133] This embodiment also provides a computer-readable storage medium storing executable instructions that, when executed, cause the machine to execute the privacy-preserving method for heterogeneous federated learning based on momentum optimization as described above.

[0134] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program code for implementing the functions of any one of the above embodiments is stored, and the computer or processor of the system or device is caused to read and execute the instructions stored in the readable storage medium.

[0135] In this case, the program code read from the readable medium itself can implement the functions of any one of the above embodiments, so the computer-readable code and the readable storage medium storing the computer-readable code constitute a part of this specification.

[0136] Examples of the readable storage medium include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer or the cloud via a communication network.

[0137] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, rather than limitations on the specific implementation manners of the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the claims of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

1. An adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization, characterized in that, The method includes: S1: The client receives the feature extractor parameters broadcast by the central server, introduces a momentum mechanism into the local loss function, and calculates the local gradient of the current iteration round based on the local dataset; S2: Apply the gradient descent algorithm to update the local model parameters using the calculated local gradient; S3: For the feature extractor parameters, calculate the Fisher information matrix, normalize the Fisher information of each layer, and based on the parameter importance screening strategy, identify the parameters with less contribution to the loss function and sparsify them; S4: Perform differential privacy processing on the feature extractor parameters after Fisher information screening and sparsification, then upload them to the central server for parameter aggregation, and broadcast the updated global model parameters back to each local client for the next round of training; The specific content of S1 includes: S11. Improve the classifier part based on the momentum mechanism and optimize the loss function of the classifier part to: （1）； In formula (1), is the cross-entropy function, represents the feature extractor parameters and classifier parameters in the r-th local iteration of the t-th global iteration for the i -th client, represents the momentum coefficient, represents the classifier parameters in the r-th local iteration of the t-th global iteration for the i -th client, represents the classifier parameters in the (r-1)-th local iteration of the t-th global iteration for the i -th client; For the classifier parameters using the momentum mechanism , after T global iterations, where T is the total number of global iterations, the following convergence bounds are obtained: ； In formula (2), E represents the expectation, represents the gradient, represents the classifier parameters for the i-th client at the t-th round of global iteration; , represents the learning rate, represents the momentum coefficient; , represents the i -th client's classifier parameters at the initial global iteration, represents the i -th client's classifier parameters at the (T - 1)-th round of global iteration, represents the total number of local iteration rounds, L represents the Lipschitz coefficient of the cross-entropy function, and m represents the variance of the batch gradient estimate of the local data; Among them, , ; When the convergence limit is minimized, there is an optimal , that is, the convergence effect is the best; S12. The client i uses the local dataset to calculate the local gradient of the current iteration round; （3）； （4）； In Formulas (3) to (4), represents the local gradient of the classifier in the r-th local iteration of the t-th round of iteration of the client; i In Formulas (3) to (4), represents the local gradient of the feature extractor in the r-th local iteration of the t-th round of iteration of the client; i In Formulas (3) to (4), represents the feature extractor parameters in the r-th local iteration of the t-th global iteration of the i -th client; In Formulas (3) to (4), i represents the classifier parameters in the r-th local iteration of the t-th global iteration of the -th client; represents the -th sample in the local dataset; represents the sample gradient obtained by the cross-entropy function based on sample b; In Formulas (.) to (.), represents the total number of samples in the local dataset of the client; represents the momentum coefficient.

2. The privacy-preserving method for heterogeneous federated learning based on momentum optimization according to claim 1, wherein The specific content of S3 includes: S31: For the feature extractor parameters, evaluate and screen the importance of the feature extractor parameters based on the Fisher information matrix: Fisher information matrix of feature extractor parameters is calculated as the expected value of the second derivative of the log-likelihood function, as shown in the following formula: （5）； In formula (5), represents the log-likelihood function of which is the feature extractor parameter; S32: Normalize the Fisher information matrix of each layer, and based on the parameter importance screening strategy, identify the parameters with less contribution to the loss function; Apply the min-max normalization process between the weight layers of the Fisher information matrix. The normalization process of the Fisher information is as follows: （6）； In formula (6), represents the normalized Fisher information of parameter in the -th layer, and represents the Fisher information of parameter in the l -th layer of the model; l ​ The Fisher information matrix represents the contribution degree of the parameters. Set a threshold and select the Fisher information matrix greater than the threshold; S33: Sparsify the parameters with less contribution to the loss function, specifically including: After local training, client i obtains an updated value of the feature extractor parameters : （7）； In formula (7), and respectively correspond to the feature extractor parameters of the i th client in the t-th round and the (t - 1)-th round; Define the sparsification mask as follows: （8）； In formula (8), represents the percentile of the Fisher information value, represents the mask matrix for updating the feature extractor; Then, in the t-th round, calculate the masked update using the element-wise Hadamard product; （9）。 3. The privacy-preserving method for heterogeneous federated learning based on momentum optimization according to claim 2, wherein The differential privacy processing of the feature extractor parameters after Fisher information screening and sparsification specifically includes: Perform clipping processing and differential privacy noise addition on the feature extractor parameters to be uploaded to obtain : （10）； In Formula (10), represents the clipping threshold; For the cropped , differential privacy noise addition is further performed: （11）； In formula (11), represents the client i feature extractor parameters after clipping and adding noise in the t-th global iteration, represents Gaussian noise with an expectation of 0 and a variance of and is a vector with all scalar elements equal to 1.

4. The heterogeneous federated learning adaptive privacy preservation method based on momentum optimization according to claim 2, characterized in that The weight layers include, but are not limited to, the convolutional layer, batch normalization layer, fully connected layer, and weight layer after the activation layer in the convolutional neural network.

5. An apparatus for implementing an adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization, characterized in that, The device includes: A processor; A memory, on which a computer program that can run on the processor is stored; Wherein, when the computer program is executed by the processor, it implements the steps of the heterogeneous federated learning adaptive privacy-preserving method based on momentum optimization as described in any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images