Heterogeneous federated learning adaptive privacy protection method and device based on momentum optimization

Abstract

The invention belongs to the technical field of federated learning privacy protection, and particularly relates to a heterogeneous federated learning adaptive privacy protection method and device based on momentum optimization. The method comprises the steps that feature extractor parameters broadcasted by a central server are received, the local gradient of a current iteration round is calculated based on a local data set, and a momentum mechanism is introduced into local gradient updating for a classifier part; a gradient descent algorithm is applied, and local model parameters are updated through the local gradient obtained through calculation; for the feature extractor part, calculating a Fisher information matrix; performing normalization processing on the Fisher information of each layer, and screening strategies; and then differential privacy processing is carried out, the global model parameters are uploaded to a central server for parameter aggregation, and updated global model parameters are broadcasted back to each local client for next round of training. According to the method, the problems of negative influence of a differential privacy mechanism on model training, slow model convergence and poor privacy protection effect are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of privacy protection in federated learning, and more specifically, relates to an adaptive privacy-preserving method and device for heterogeneous federated learning based on momentum optimization. Background Art

[0002] Federated learning is a distributed machine learning paradigm. Multiple clients train models locally and perform parameter aggregation under the coordination of a central server to build a global model or personalized local models. This method reduces the privacy risks brought by centralized data storage while ensuring model performance. However, research shows that even if data is not directly shared, federated learning may still leak users' sensitive information, making it face potential privacy threats. Therefore, introducing the differential privacy mechanism has become an important means to protect data security in federated learning.

[0003] Chinese invention patent document CN116050509A discloses a clustering federated learning method based on momentum gradient descent, proposes a clustering method based on minimizing the loss function, and improves the stochastic gradient descent (SGD) algorithm in the local update stage of federated learning, using the optimization algorithm of momentum gradient descent. In the global update stage, gradient averaging and model averaging are proposed; the clustering federated learning algorithm based on momentum gradient descent (CFL-MGD) of the present invention not only obtains the theoretical guarantee of exponential convergence in convex settings, but is also more efficient than other algorithms in non-convex settings such as neural networks; the framework structure is simple and the algorithm is efficient, combining clustering and polyak's momentum.

[0004] The core idea of differential privacy is to introduce random noise during the model update process to reduce the possibility for attackers to infer the original data through parameters, thereby improving the privacy protection ability. In the federated learning framework, differential privacy is mainly achieved by injecting noise into the model parameters or gradients uploaded by clients. However, this noise will inevitably affect the model optimization process, resulting in a decrease in the convergence speed and a reduction in model accuracy. Therefore, how to find the optimal balance between privacy protection and model performance remains a key challenge. In addition, federated learning usually faces the problem of data heterogeneity, that is, the data distributions of different clients are inconsistent (Non-IID). Traditional federated learning methods use a unified global model and are difficult to adapt to the data distribution differences of all clients. Summary of the Invention

[0005] The present invention aims to overcome at least one defect of the above-mentioned prior art, and provides an adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization to achieve a better trade-off between privacy protection and model performance, and to solve the negative impact of the differential privacy mechanism on model training and the problems of slow model convergence and poor privacy protection effect.

[0006] The detailed technical solution of the present invention is as follows: An adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization, the method comprising: S1: The client receives the feature extractor parameters broadcast by the central server, introduces a momentum mechanism into the local loss function, and calculates the local gradient of the current iteration round based on the local dataset; S2: Apply the gradient descent algorithm to update the local model parameters using the calculated local gradient; S3: For the feature extractor parameters, calculate the Fisher information matrix, normalize the Fisher information of each layer, and identify and sparsify the parameters that contribute less to the loss function based on the parameter importance screening strategy to reduce the impact of redundant parameters while maintaining the model performance; S4: Perform differential privacy processing on the feature extractor parameters after Fisher information screening and sparsification, then upload them to the central server for parameter aggregation, and broadcast the updated global model parameters back to each local client for the next round of training.

[0007] Further, the S1 specifically includes: S11. Improve the classifier part based on the momentum mechanism, and optimize the loss function of the classifier part to: (1); In formula (1), is the cross-entropy function, represents the feature extractor parameters and classifier parameters in the r-th local iteration of the t-th global iteration of the i -th client, represents the momentum coefficient, which measures the influence degree of the momentum term on the optimization process, represents the classifier parameters in the r-th local iteration of the t-th global iteration of the i -th client, represents the classifier parameters in the (r-1)-th local iteration of the t-th global iteration of the i -th client.

[0008] For the classifier parameters using the momentum mechanism , after T global iterations, where T is the total number of global iterations, the following convergence bound is obtained: (2); In formula (2), E represents the expectation, represents the gradient, represents the classifier parameters in the t-th global iteration of the i -th client; , denotes the learning rate, represents the momentum coefficient; , denotes the classifier parameters of the i -th client at the initial global iteration, denotes the classifier parameters of the i -th client at the (T - 1)-th global iteration, represents the total number of local iteration rounds, L represents the Lipschitz coefficient of the cross-entropy function, and m represents the variance of the batch gradient estimation of the local data; Among them, , .

[0009] When the convergence bound is minimized, there exists an optimal , that is, the convergence effect is the best; when the learning rate is small enough, Equation (4) is convex with respect to the momentum coefficient , so there exists an optimal that minimizes the convergence bound.

[0010] For the first term in Equation (2), introducing the momentum mechanism can reduce the impact of the initial error on the convergence speed, thereby accelerating the optimization process. The momentum term accumulates historical gradient information during the update process, making the optimization direction smoother, reducing gradient oscillations, and ultimately enhancing the convergence speed.

[0011] S12. Client i uses the local dataset to calculate the local gradient of the current iteration round; (3); (4); In Formulas (3) to (4), denotes the local gradient of the classifier in the i -th local iteration of the -th client at the t-th iteration; denotes the local gradient of the feature extractor in the i -th local iteration of the -th client at the t-th iteration; represents the feature extractor parameters in the i -th local iteration of the -th client at the t-th global iteration; represents the classifier parameters in the i -th local iteration of the -th client at the t-th global iteration; denotes the -th sample in the local dataset ; denotes the sample gradient of the cross-entropy function obtained based on sample b, Represents the client of the local dataset The total number of samples in; Represents the momentum coefficient.

[0012] Furthermore, the specific steps of S3 are as follows: S31. For the feature extractor parameters, evaluate and screen the importance of the feature extractor parameters based on the Fisher information matrix: The Fisher information matrix of the feature extractor parameters Is calculated as the expected value of the second derivative of the log-likelihood function, and the formula is as follows: (5); In formula (5), Represents The log-likelihood function of, Is the feature extractor parameter; S32. Normalize each layer of the Fisher information matrix, and based on the parameter importance screening strategy, identify the parameters that contribute less to the loss function; Apply the min-max normalization process between the weight layers of the Fisher information matrix. The normalization process of the Fisher information is as follows: (6); In formula (6), Represents the normalized Fisher information of the parameter In the layer, Represents the Fisher information of the parameter In the layer, , l Represents the l layer in the model.

[0013] The Fisher information matrix represents the contribution degree of the parameter. Set a threshold, and select the Fisher information matrix greater than the threshold.

[0014] S33. Sparsify the parameters that contribute less to the loss function, which specifically includes: After local training, the client i Obtains the updated value of the feature extractor parameter : (7); In formula (7), And Respectively correspond to the feature extractor parameters of the i th client in the t-th round and the (t - 1)-th round.

[0015] For parameters with small Fisher information values, set them to zero to reduce their impact on pruning. Specifically, define the sparsification mask as follows: (8); In formula (8), represents the percentile of the Fisher information value, and represents the mask matrix for updating the feature extractor.

[0016] Then, in the t-th round, calculate the masked update using the element-wise Hadamard product, that is, update formula (7): (9).

[0017] Through this process, parameters with low information content are preferentially removed, while those that contribute significantly to the model performance are retained. This method not only helps to maintain the model performance but also significantly reduces the storage requirements and communication costs.

[0018] Furthermore, the weight layer includes: the convolutional layer, batch normalization layer, fully connected layer, and the weight layer after the activation layer in the convolutional neural network.

[0019] Furthermore, perform differential privacy processing on the feature extractor parameters after Fisher information screening and sparsification, specifically including: Perform pruning processing and differential privacy noise addition on the feature extractor parameters to be uploaded to obtain : (10); In formula (10), represents the pruning threshold of

[0020] For the pruned , further perform differential privacy noise addition: (11); In formula (11), represents the feature extractor parameters of the client i after pruning and noise addition in the t-th global iteration, represents Gaussian noise with an expected value of 0 and a variance of , and is a vector with all scalars equal to 1.

[0021] In another aspect of the present invention, there is provided a device for an adaptive privacy-preserving method of heterogeneous federated learning based on momentum optimization. The device includes: At least one processor; And a memory storing instructions which, when executed by the at least one processor, cause the at least one processor to execute the privacy-preserving method for heterogeneous federated learning with momentum optimization as described above.

[0022] In another aspect of the present invention, there is also provided a computer-readable storage medium storing executable instructions which, when executed, cause the machine to execute the privacy-preserving method for heterogeneous federated learning with momentum optimization as described above.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows: The privacy-preserving method and device for heterogeneous federated learning with momentum optimization provided by the present invention fuse the adaptive differential privacy mechanism and the momentum optimization strategy to achieve a better trade-off between privacy protection and model performance. In the update process of the shared feature extractor, a sparsification strategy based on the Fisher information matrix is introduced to identify the parameters with less impact on the model performance and perform pruning, thereby effectively reducing the scale of parameters to which noise needs to be added, reducing the negative impact of the differential privacy mechanism on model training, effectively reducing the accumulation of noise errors, and at the same time reducing the communication overhead and improving the communication efficiency of the system; in the classifier update stage, the momentum optimization strategy is adopted to alleviate the gradient fluctuation caused by noise interference and accelerate the model convergence, and it is proved that there is an optimal momentum coefficient and the model can achieve the optimal convergence performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flowchart of the privacy-preserving method for heterogeneous federated learning with momentum optimization according to the present invention.

[0025] Figure 2 is a comparison diagram of the effects of the benchmark scheme and the scheme of the present invention in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described below with reference to the drawings and embodiments.

[0027] It should be noted that the following detailed description is exemplary and is intended to provide further illustration of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0028] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0029] Without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0030] Embodiment 1 Refer Figure 1 , this embodiment provides an adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization, and the method includes: S1. The local client calculates the local gradient based on the momentum mechanism: The client receives the feature extractor parameters broadcast by the central server, introduces the momentum mechanism into the local loss function, and calculates the local gradient of the current iteration round based on the local dataset.

[0031] The model parameters are divided into feature extractor parameters, that is, shared parameters, and the classifier part. Among them, the feature extractor parameters need to be uploaded to the central server; for the classifier part, it is retained locally, that is, retained in the client, to enhance the convergence stability. The type of the local dataset is image data.

[0032] Specifically, the S1 specifically includes: S11. Improve the classifier part based on the momentum mechanism, and optimize the loss function of the classifier part to: (1); In formula (1), is the cross-entropy function, represents the feature extractor parameters and classifier parameters in the r-th local iteration of the t-th global iteration of the i -th client, represents the momentum coefficient, which measures the influence degree of the momentum term on the optimization process, represents the classifier parameters in the r-th local iteration of the t-th global iteration of the i -th client, represents the classifier parameters in the (r - 1)-th local iteration of the t-th global iteration of the i -th client.

[0033] For the classifier parameters using the momentum mechanism , after T global iterations, where T is the total number of global iterations, the following convergence bound is obtained: (2); In formula (2), E represents expectation, represents the gradient, represents the classifier parameters of the i -th client in the t-th round of the global iteration; , represents the learning rate, represents the momentum coefficient; , represents the classifier parameters of the i-th client in the initial global iteration, represents the classifier parameters of the i-th client in the (T - 1)-th round of the global iteration, represents the total number of local iteration rounds, L represents the Lipschitz coefficient of the cross-entropy function, m represents the variance of the batch gradient estimation of the local data, , .

[0034] When the convergence bound is minimized, there exists an optimal , that is, the convergence effect is the best; when the learning rate is small enough, formula (2) is convex with respect to the momentum coefficient , so there exists an optimal that minimizes the convergence bound.

[0035] For the first term in formula (2), introducing the momentum mechanism can reduce the impact of the initial error on the convergence speed, thereby accelerating the optimization process. The momentum term accumulates historical gradient information during the update process, making the optimization direction smoother, reducing gradient oscillations, and ultimately improving the convergence speed. However, although the momentum mechanism can effectively accumulate useful gradient information, it will inevitably accumulate random noise in the gradient. When the momentum coefficient is too large, the impact of historical noise will be amplified, which may lead to instability in the optimization process. Especially in mini-batch training, the gradient calculated each time is only based on a limited number of samples, so the gradient estimation will deviate from the true gradient, introducing additional random noise.

[0036] As the training progresses, this noise will continuously accumulate in the momentum update, further amplifying its impact, and then affecting the convergence stability and final performance of the model. Therefore, choosing an appropriate momentum coefficient is crucial and it is necessary to find the best balance between smooth updates and noise suppression.

[0037] S12. Client i uses the local dataset to calculate the local gradient of the current iteration round; (3) (4) In Formulas (3) to (4), represents the local gradient of the classifier in the r-th local iteration of the t-th round of iteration of the client i ; represents the local gradient of the feature extractor in the r-th local iteration of the t-th round of iteration of the client i ; represents the i -th feature extractor parameter in the r-th local iteration of the t-th global iteration of the -th client; i represents the -th classifier parameter in the r-th local iteration of the t-th global iteration of the -th client; represents the -th sample in the local dataset ; represents the sample gradient obtained by the cross-entropy function based on sample b, represents the total number of samples in the local dataset of the client;

[0038] S2. Apply the gradient descent algorithm and update the local model parameters using the calculated local gradients, specifically: After obtaining the local gradients, based on stochastic gradient descent, update the model parameters of the client i using the obtained local gradients, that is: (12); (13); In Formulas (12) to (13), and respectively represent the feature extractor parameter and the classifier parameter in the r-th local iteration of the t-th global iteration updated by the client i ; and respectively represent the learning rates of the feature extractor parameter and the classifier parameter.

[0039] S3. Sparsify the feature extractor parameters: For the feature extractor parameters, calculate the Fisher information matrix, normalize the Fisher information of each layer, and based on the parameter importance screening strategy, identify the parameters that contribute less to the loss function and sparsify them to reduce the influence of redundant parameters while maintaining the model performance.

[0040] Preferably, the S3 specifically includes: S31. For the feature extractor parameters , To further reduce the bias introduced by pruning and improve the communication efficiency of federated learning (FL), the importance of feature extractor parameters is evaluated and screened based on the Fisher information matrix. By removing parameters with low information content or negligible contribution to the model, the pruning bias caused by pruning is alleviated.

[0041] Specifically, the Fisher information matrix of the feature extractor parameters is calculated as the expected value of the second derivative of the log-likelihood function, and the formula is as follows: (5); In formula (5), represents the log-likelihood function of , and

[0042] are the feature extractor parameters. To reduce the computational complexity and improve the efficiency, the diagonal elements of the Fisher information matrix are used for approximation. In many cases, the diagonal elements better reflect the influence of the model parameters than the non-diagonal elements.

[0043] S32. Normalize the Fisher information matrix of each layer, and based on the parameter importance screening strategy, identify the parameters with less contribution to the loss function for post-processing to reduce their influence.

[0043] To address the problem that the Fisher information may be too concentrated due to data heterogeneity, which may lead to excessive sparsity in some layers, a min-max normalization process is applied between the weight layers of the Fisher information matrix. The weight layers include: convolutional layers (Conv layers), batch normalization layers (BatchNorm), fully connected layers (Fully Connected layers, FC), and the weight layers after the activation layer in a convolutional neural network (CNN). This normalization ensures that the evaluation of the parameter importance within each layer is independent and fair, thereby reducing the bias in the pruning process. The normalization process of the Fisher information is as follows: (6); In formula (6), represents the normalized Fisher information of the parameter in the -th layer, represents the Fisher information of the parameter in the -th layer, , l represents the l -th layer in the model.

[0044] The Fisher information matrix represents the contribution degree of parameters. Set a threshold and select the Fisher information matrix greater than the threshold.

[0045] S33. Sparsify the parameters with less contribution to the loss function, specifically including: After local training, client i obtains the updated value of the feature extractor parameters : (7); In formula (7), and correspond to the feature extractor parameters of the i th client in the t-th round and the (t - 1)-th round respectively.

[0046] For the parameters with small Fisher information values, set them to zero to reduce their impact on pruning. Specifically, define the sparsification mask as follows: (8); In formula (8), represents the th percentile of the Fisher information value, represents the mask matrix for the update of the feature extractor.

[0047] Then, in the t-th round, calculate the masked update using the element-wise Hadamard product, that is, update formula (7): (9).

[0048] Through this process, preferentially remove the parameters with low information content while retaining those that contribute more to the model performance. This method not only helps to maintain the model performance but also significantly reduces the storage requirements and communication costs.

[0049] S4. After differential privacy processing, perform parameter aggregation and iterate as follows: Perform differential privacy processing on the feature extractor parameters after Fisher information screening and sparsification, then upload them to the central server for parameter aggregation, and broadcast the updated global model parameters back to each local client for the next round of training.

[0050] Performing differential privacy processing on the feature extractor parameters after Fisher information screening and sparsification specifically includes: performing clipping processing and adding differential privacy noise to the feature extractor parameters to be uploaded to obtain : (10); In formula (10), represents 's clipping threshold.

[0051] For the cropped , further perform differential privacy noise addition: (11); In formula (11), represents the feature extractor parameters after cropping and noise addition of the client i at the t-th global iteration, represents Gaussian noise with an expectation of 0 and a variance of , is a vector with all scalars being 1, that is .

[0052] Since this patent introduces sparsification processing in the S2 and S3 stages, in the S4 stage, the cropping bias is effectively reduced; or a smaller cropping threshold can be used because the variance of the Gaussian noise is regulated by , effectively reducing the impact of noise on the model performance. This strategy not only optimizes the privacy protection mechanism but also improves the convergence performance of the model while ensuring privacy.

[0053] The experiments of this embodiment are demonstrated as follows: The CIFAR-10 dataset is a classic image classification dataset constructed by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton from the University of Toronto in Canada and is widely used in the field of computer vision. The dataset contains 60,000 color images in 10 different classes, with 6,000 images in each class. The resolution of each image is 32×32 pixels. These classes cover common objects in daily life, such as airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. The dataset is divided into a training set and a test set, where the training set contains 50,000 images and the test set consists of 10,000 images. Each image is represented using the RGB color channel, with pixel values ranging from 0 to 255, and each picture corresponds to a label between 0 and 9 to indicate its class. The CIFAR-10 dataset is commonly used in distributed learning environments, such as when multiple institutions or enterprises collaborate to train an image classification model without directly sharing the original data. In this scenario, each participating party uses its own data subset for local training and exchanges model update information through methods such as federated learning instead of transmitting the original image data. To enhance data privacy protection, differential privacy technology can be introduced to add noise during the parameter exchange process, thereby reducing the risk of sensitive information leakage. This privacy protection mechanism has important value in applications such as intelligent monitoring, autonomous driving object recognition, and augmented reality, as it can both ensure data security and improve model performance.

[0054] This embodiment conducts experiments on the CIFAR10 dataset. In the experiment, the number of clients in this patent is set to 6, and the number of local iteration rounds , learning rate . For the momentum parameter , according to formula (2), search is performed on 0 - 1 with a step size of 0.05, and the optimal = 0.05 is found.

[0055] The results of the comparative experiment are as Figure 2 shown, where: Benchmark method 1 is the FedDPA algorithm, which screens personalized parameters in the broadcast stage and combines an adaptive constraint strategy; Benchmark method 2 is the DPFL - BL algorithm, a method that combines regularization and sparsification, while improving the generalization ability to alleviate the problems brought by unbalanced data distribution and differential privacy clipping; Benchmark method 3 is the DP - FedAvg algorithm, that is, a differential privacy mechanism is introduced on the basis of the classical FedAvg scheme to enhance the data privacy protection ability; Benchmark method 4 is the PPSGD algorithm, which improves the model adaptability through a personalized privacy protection strategy while ensuring privacy security.

[0056] Under the same privacy protection level, the scheme of the present invention improves the accuracy by 3.6%, 2.5%, 6.5% and 6.3% respectively compared with FedDPA, DPFL - BL, DP - FedAvg and PPSGD.

[0057] Embodiment 2 This embodiment provides a device for implementing an adaptive privacy - preserving method for heterogeneous federated learning based on momentum optimization. The device includes: At least one processor; And a memory that stores instructions. When the instructions are executed by the at least one processor, the at least one processor executes the above - mentioned adaptive privacy - preserving method for heterogeneous federated learning based on momentum optimization.

[0058] In this embodiment, the electronic device includes but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile computing device, smart phone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable computing device, consumer electronic device, etc.

[0059] Embodiment 3 This embodiment also provides a computer-readable storage medium storing executable instructions that, when executed, cause the machine to perform the privacy-preserving heterogeneous federated learning adaptive method based on momentum optimization as described above.

[0060] Specifically, a system or device equipped with a readable storage medium may be provided, on which software program code for implementing the functions of any one of the above embodiments is stored, and the computer or processor of the system or device is caused to read and execute the instructions stored in the readable storage medium.

[0061] In this case, the program code read from the readable medium itself can implement the functions of any one of the above embodiments, so the computer-readable code and the readable storage medium storing the computer-readable code constitute a part of this specification.

[0062] Examples of the readable storage medium include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code may be downloaded from a server computer or a cloud via a communication network.

[0063] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, rather than limitations on the specific implementation manners of the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the claims of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

1. An adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization, characterized in that: The method comprises: S1: The client receives the feature extractor parameters broadcast by the central server, introduces the momentum mechanism into the local loss function, and calculates the local gradient of the current iteration round based on the local data set; S2: Apply the gradient descent algorithm to update the local model parameters using the calculated local gradients; S3: For the feature extractor parameters, the Fisher information matrix is ​​calculated and the Fisher information of each layer is normalized. Based on the parameter importance screening strategy, the parameters that contribute less to the loss function are identified and sparsely processed. S4: The feature extractor parameters after Fisher information filtering and sparsification are differentially privately processed and then uploaded to the central server for parameter aggregation. The updated global model parameters are broadcast back to each local client for the next round of training.

2. The adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization according to claim 1, characterized in that: The S1 specifically includes: S11. Improve the classifier part based on the momentum mechanism and change the loss function of the classifier part Optimized to: （1）； In formula (1), is the cross entropy function, Representative i The feature extractor parameters and classifier parameters in the rth local iteration of the tth global iteration of the client, represents the momentum coefficient, Representative i The classifier parameters in the rth local iteration of the tth global iteration of the client, Representative i The classifier parameters in the r-1th local iteration of the tth global iteration of the client; For the classifier parameters using momentum mechanism , after T global iterations, where T is the total number of global iterations, the following convergence bounds are obtained: （2）； In formula (2), E represents expectation, represents the gradient, represents the classifier parameters of the tth global iteration of the i-th client; , represents the learning rate, represents the momentum coefficient; , Indicates i The classifier parameters for the initial global iteration of the client, Indicates i The classifier parameters of the T-1th round of global iteration of the client, Represents the total number of local iterations, L represents the Lipschitz coefficient of the cross entropy function, and m represents the variance of the batch gradient estimate of the local data; in, , ; When the convergence bound is minimized, there exists an optimal , that is, the convergence effect is the best; S12, client i uses local data set Calculate the local gradient of the current iteration round; （3）； （4）； In formulas (3)~(4), Represents the client i The local gradient of the classifier at the rth local iteration of the tth iteration; Represents the client i The local gradient of the feature extractor at the rth local iteration of the tth iteration; Representative i The feature extractor parameters in the rth local iteration of the tth global iteration of the client; Representative i The classifier parameters in the rth local iteration of the tth global iteration of the client; Represents a local dataset The samples; Represents the cross entropy function The sample gradient obtained based on sample b, Represents the client Local dataset The total number of samples in represents the momentum coefficient.

3. The adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization according to claim 2 is characterized in that: The S3 specifically includes: S31. For feature extractor parameters, the importance of feature extractor parameters is evaluated and screened based on the Fisher information matrix: Fisher information matrix of feature extractor parameters is calculated as the expected value of the second derivative of the log-likelihood function, as follows: （5）； In formula (5), express The log-likelihood function of is the feature extractor parameter; S32, normalize the Fisher information matrix of each layer, and identify the parameters that contribute less to the loss function based on the parameter importance screening strategy; The maximum and minimum normalization process is applied between the weight layers of the Fisher information matrix. The normalization process of Fisher information is as follows: （6）； In formula (6), Indicates Layer Parameters The normalized Fisher information of Indicates Layer Parameters Fisher information, , l Indicates the model l layer; The Fisher information matrix represents the contribution of the parameters. We set a threshold and select the Fisher information matrix that is greater than the threshold. S33, the parameters that contribute less to the loss function are sparse, including: After local training, client i obtains the updated values ​​of the feature extractor parameters : （7）； In formula (7), and Corresponding to i The feature extractor parameters of the t-th and t-1-th rounds of each client; The sparseness mask is defined as follows: （8）； In formula (8), The Fisher information value percentile, The mask matrix representing the feature extractor update; Then, the masked update is computed in round t using the element-wise Hadamard product: （9）。 4. The adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization according to claim 3 is characterized in that: The feature extractor parameters after Fisher information screening and sparseness are subjected to differential privacy processing, specifically including: The feature extractor parameters to be uploaded are trimmed and differential privacy noise is added to obtain : （10）； In formula (10), represent The clipping threshold; For the cut , further perform differential privacy noise addition: （11）； In formula (11), Represents the client i The feature extractor parameters after pruning and denoising at the tth global iteration, represents an expected value of 0 and a variance of Gaussian noise, is a vector where each scalar is all 1.

5. The adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization according to claim 3 is characterized in that: The weight layer includes but is not limited to: a convolutional layer, a batch normalization layer, a fully connected layer, and a weight layer after an activation layer in a convolutional neural network.

6. A device for implementing an adaptive privacy-preserving method for heterogeneous federated learning based on momentum optimization, characterized in that: The device comprises: processor; a memory having stored thereon a computer program executable on the processor; Wherein, when the computer program is executed by the processor, the steps of the adaptive privacy preservation method for heterogeneous federated learning based on momentum optimization as described in any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images