Federal learning acceleration method, system and device based on auto-encoder and residual quantization

Abstract

The invention discloses a federal learning acceleration method, system and device based on an auto-encoder and residual quantization. A central node starts a federal learning task and distributes an initial global model, each participant node performs initial training by using local data, and an encoder and a quantizer are not used in the stage. Meanwhile, the center node trains an auto-encoder and a residual quantizer in parallel, the trained auto-encoder and residual quantizer models are distributed to each participant node, model parameters are compressed by using the auto-encoder, the compression precision is improved through a residual quantization technology, the center node collects compressed and quantized parameters, and the compression precision is improved. And reconstructing and aggregating by using a decoder, distributing the updated global model to each node for the next round of training, and repeating the process until the preset number of times of training rounds is reached or the performance of the model meets the requirement. Through the auto-encoder and the residual quantization technology, the data transmission quantity is remarkably reduced, model training is accelerated, and federal learning efficiency and practicability are improved.

Description

Technical Field

[0001] The present invention relates to the field of federated learning for privacy computing, and specifically to a federated learning acceleration method and device based on autoencoders and residual quantization. Background Art

[0002] Federated learning technology has emerged to build a machine learning framework that enables artificial intelligence systems to efficiently and accurately collaborate on multi-party data while meeting data privacy, security, and compliance requirements. Federated learning is a distributed machine learning paradigm whose core is to distribute the model training process to multiple devices and achieve data privacy protection by only uploading model parameter updates rather than raw data. Under this framework, the raw data of each client is always stored locally, without the need for data exchange or transmission, ensuring data locality and privacy. By transmitting only local model parameters, federated learning maximizes data privacy and security. In addition, compared to models that rely on training on a single local dataset, federated learning can significantly improve the overall performance and generalization capabilities of the model through multi-party collaborative training.

[0003] However, in practical applications, while federated learning can effectively protect the confidentiality and integrity of data, data availability still faces significant challenges. In particular, when using provably secure secret-state computation methods (such as secure multi-party computation and homomorphic encryption), extremely high protocol communication overhead is often introduced, resulting in a significant decrease in training efficiency. To address this issue, existing research focuses on two optimization approaches: improving communication efficiency and reducing communication volume. Improving communication efficiency often requires significant investment in resources (such as high-performance hardware or complex algorithms), while reducing communication volume is a more practical strategy that can achieve immediate results without significantly increasing costs.

[0004] An autoencoder, as an unsupervised learning model, consists of two parts: an encoder and a decoder. The encoder maps input data to a low-dimensional representation space, while the decoder reconstructs the original data based on the low-dimensional representation. Residual quantization is an advanced quantization technique that significantly improves the accuracy of the quantized model by preserving the residual information between the input data and the model prediction. Both techniques have been widely used in the field of deep learning. Combining autoencoders and residual quantization into the federated learning training framework not only effectively protects data privacy but also enables efficient feature learning. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a federated learning acceleration method, system, and device for autoencoders and residual quantization, which can solve the problem of low existing communication efficiency.

[0006] The objective of the present invention is achieved through the following technical solutions: In the first aspect, a federated learning acceleration method based on autoencoders and residual quantization is provided, wherein the steps of the method are as follows:

[0007] S1. The central node in the federated learning network initiates a federated learning task to the network, where the federated learning task information includes the task type, the relevant data types used for training, the initial global model, and the number of training rounds;

[0008] S2. Each participant node in the federated learning network receives the initial global model and trains the model using local data;

[0009] S3. Use the local data of each participant node to train the autoencoder model and the residual quantizer model in parallel on the central node; distribute the trained autoencoder model and residual quantizer model to each participant node;

[0010] S4. After receiving the autoencoder model and residual quantizer model, each participating node uses the autoencoder model to compress the model parameters trained in step S2, mapping the model parameters to a low-dimensional representation. The compressed model parameters are further processed using residual quantization technology to improve the accuracy of data compression. The residual quantization technology consists of a stack of multiple sub-quantizers, each of which is responsible for quantizing the residual value of the previous quantizer.

[0011] S5. The central node collects the compressed and quantized model parameters uploaded by each participant node, and decodes them using the corresponding decoder to restore the original model parameters, and then aggregates the model and updates the global model.

[0012] S6. The updated global model is distributed to each participant node for the next round of training, and steps S4 to S5 are repeated until the predetermined number of training rounds is reached or the model performance meets the requirements.

[0013] Furthermore, the autoencoder model in S3 consists of two parts: an encoder and a decoder. The encoder part is responsible for mapping the trained model parameters of each participating node into a low-dimensional representation space. This process is achieved through a series of linear and nonlinear transformations, capturing the essential characteristics of the model parameters while removing redundant information. The encoder consists of multiple hidden layers, each of which contains weights W and bias parameters b. These parameters are optimized during the training process using the backpropagation algorithm and gradient descent method.

[0014] Furthermore, the decoder is the inverse process of the encoder, which reconstructs the low-dimensional representation output by the encoder back to the original high-dimensional model parameters; the design structure of the decoder is symmetrical with the encoder to ensure that the model parameters can be effectively recovered from the compressed representation; in the reconstruction process, the decoder uses the nonlinear activation function ReLU to enhance the expressiveness of the model, thereby maintaining the key information of the model parameters during compression and decompression; the decoder training also uses the back propagation algorithm to adjust the decoder parameters by minimizing the reconstruction error; the reconstruction error L recon It can be quantified by the mean square error MSE as follows:

[0015]

[0016] Where D is the dimension of the model parameters, z i are the original model parameters, are the model parameters after reconstruction.

[0017] The training of the autoencoder is an unsupervised learning process, whose goal is to minimize the difference between the model parameters and their reconstructed version. This difference is quantified by the mean squared error (MSE) and used as the loss function for training. During the training process, the autoencoder adjusts the parameters of the encoder and decoder to minimize the loss function, thereby learning effective data compression and reconstruction strategies. The total loss function of the autoencoder is L autoencoder Expressed as:

[0018] L autoencoder =L encoder +L recon

[0019] L encoder is the encoder loss, by minimizing L autoencoder , the autoencoder is able to learn how to effectively compress the model parameters into a low-dimensional representation while retaining enough information to accurately reconstruct the original parameters.

[0020] Furthermore, the residual quantizer in S3 includes a multi-layer quantizer, which uses residual vector quantization (RVQ) technology to achieve high-precision compression. RVQ is a hierarchical quantization technology that achieves high-precision compression by gradually approximating the residual. Its core idea is to progressively quantize the residual using a multi-layer codebook. The specific process is divided into two stages:

[0021] 1) Codebook training: given a set of feature vectors v iRepresents the i-th eigenvector. These eigenvectors are low-dimensional representations of the input data and are used to train the codebook of residual vector quantization (RVQ); n represents the total number of eigenvectors; RVQ constructs an L-layer codebook by progressive residual clustering; for the l-th layer codebook The training process is expressed as:

[0022]

[0023] Among them, d represents the dimension of the feature vector, that is, each feature vector v i length, r represents the residual vector; in each layer of RVQ, the residual vector is the error after quantization of the previous layer, that is, r = v i -Q(v), where Q(v) is the quantized vector, c k represents the kth codebook vector, where codebook C is a set of vectors used to quantize the residual vector r; Represents the residual vector set of the l-1th layer, K is the capacity of the codebook of each layer; the initial residual Subsequent residuals are calculated recursively:

[0024]

[0025] where Q l (·) represents the quantization function of the lth layer;

[0026] 2) Quantization process: In the quantization stage, the input vector is quantized through each layer of codebook. Each layer of quantizer is responsible for quantizing the residual of the previous layer. Specifically, for the input vector v, the quantization output of the lth layer is:

[0027] v l =Q l (v l-1 )

[0028] Where v0 = v, v l Represents the quantized output of the lth layer; the final quantized result v quantized is the accumulation of quantized outputs of all layers:

[0029]

[0030] Furthermore, in S5, the central node receives the compressed and quantized model parameters z from each participant node. q,i ; First, the quantized model parameters z are inverse quantized q,i Convert to low-dimensional representation z i ; Then, use the decoder The low-dimensional representation z i Reconstructed to approximate original model parameters The whole process is expressed as:

[0031]

[0032] in, Represents the inverse quantization operation of the lth layer residual quantization, v q,l is the quantized output of layer l, is the decoder function used to reconstruct the low-dimensional representation into the original model parameters;

[0033] Subsequently, the federated averaging algorithm FedAvg is used to aggregate all model parameters, and a weighted strategy is combined to improve aggregation efficiency and model convergence speed. The form of the federated averaging algorithm is:

[0034]

[0035] Among them, w agg is the model parameter after aggregation, w i is the model parameter uploaded by the i-th client, and m is the number of clients participating in the aggregation;

[0036] To deal with the problem of uneven data distribution, the weighted federated averaging algorithm is adopted, in which the model parameters of each client are weighted according to the number of its data samples:

[0037]

[0038] Among them, n i is the number of data samples of the i-th client, and n is the total number of samples of all clients.

[0039] Furthermore, the updating and distribution of the global model in S6 further includes the following three key steps:

[0040] 1) Two-stage compression: The client first compresses the model parameters θ into a low-dimensional representation z through the autoencoder, and then quantizes z using residual vector quantization (RVQ) to obtain the final transmitted compressed representation z q :

[0041]

[0042] 2) Error propagation analysis: The total reconstruction error consists of two parts: the autoencoder error and the quantization propagation error:

[0043]

[0044] Hypothetical decoder It's L g -Lipschitz continuous, then the quantized propagation error satisfies:

[0045]

[0046] Therefore, the upper bound of the total error is:

[0047]

[0048] Among them, ∈ quant is the quantization error; total The upper bound of the total error is the maximum possible error between the reconstructed parameters and the original parameters after the model is compressed by the autoencoder and the residual quantized. This upper bound consists of two parts: the reconstruction error of the autoencoder and the propagation of the quantization error. AE is the loss function of the autoencoder, which is the goal to be minimized during the training process of the autoencoder. It is defined as the mean square error (MSE) between the input data and the output reconstructed by the autoencoder. The mean square error is the average of the squares of the errors. Therefore, in order to get the average of the errors, we need to take the square root.

[0049] 3) Joint optimization goal: To balance compression efficiency and model accuracy, the following multi-objective optimization problem is used:

[0050]

[0051] in, represents the communication cost function, λ1,λ2,λ3 are trade-off coefficients that control the importance of autoencoding error, quantization error, and communication overhead, respectively.

[0052] In a second aspect, the present invention further provides a federated learning acceleration system based on autoencoders and residual quantization, which includes the following modules:

[0053] The autoencoder module consists of two sub-modules: an encoder and a decoder. The encoder is responsible for mapping the model parameters to a low-dimensional representation space, and the decoder is responsible for reconstructing the low-dimensional representation back to the original high-dimensional model parameters. The autoencoder module is trained using the backpropagation algorithm and gradient descent to minimize the difference between the model parameters and their reconstructed versions.

[0054] The residual quantization module includes a multi-layer quantizer that maps the encoder output to the vector in the quantizer codebook with the smallest Euclidean distance to the output, and then quantizes the residual layer by layer to further reduce the dimension of the model parameters and retain key information;

[0055] The federated averaging algorithm module is used to perform aggregation of model parameters at the central node, taking into account the number of data samples of each client through a weighted federated averaging algorithm to generate global model updates;

[0056] Dynamic learning rate adjustment module, which dynamically adjusts the contribution of each participating node to the global model update based on the local training effect of each participating node to optimize the training efficiency and stability of the model;

[0057] The data preprocessing module is used to standardize and normalize the data of each participating node to improve the efficiency and accuracy of model training; and encryption technology is used to protect the privacy and integrity of data during transmission.

[0058] The performance monitoring module monitors the model performance during the federated learning process, including key indicators such as accuracy and loss rate, and triggers retraining or adjustment of the training strategy when the performance does not meet the preset requirements.

[0059] The user interface module provides a user interaction interface that allows users to configure training parameters, view training progress and performance indicators, and deploy models.

[0060] In the third aspect, the present invention also provides a federated learning acceleration device based on autoencoder and residual quantization, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the federated learning acceleration method based on autoencoder and residual quantization.

[0061] In a fourth aspect, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned federated learning acceleration method based on autoencoders and residual quantization.

[0062] The beneficial effects of the present invention are as follows: the present invention combines autoencoders and residual quantization technology into the federated learning training framework to significantly reduce the amount of data transmission and effectively protect data privacy, achieve efficient feature learning, accelerate model training, and improve the efficiency and practicality of federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 This is a flow chart of a federated learning acceleration method based on autoencoders and residual quantization provided by the present invention.

[0065] Figure 2 A structural diagram of a federated learning acceleration device based on autoencoder and residual quantization provided by the present invention. DETAILED DESCRIPTION

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0067] like Figure 1 As shown, the present invention provides a federated learning acceleration method based on autoencoder and residual quantization, and the steps of the method are as follows:

[0068] S1. The central node in the federated learning network initiates a federated learning task to the network, where the federated learning task information includes the task type, the relevant data types used for training, the initial global model, and the number of training rounds;

[0069] S2. Each participant node in the federated learning network receives the initial global model and trains the model using local data;

[0070] S3. On the central node, use the local data of each participant node to parallel train the autoencoder model and the residual quantizer model;

[0071] The autoencoder model consists of two parts: an encoder and a decoder. The encoder is responsible for mapping the trained model parameters of each participating node into a low-dimensional representation space. This process is achieved through a series of linear and nonlinear transformations, capturing the essential characteristics of the model parameters while removing redundant information. The encoder consists of multiple hidden layers, each of which contains weights W and bias parameters b. These parameters are optimized during training using backpropagation and gradient descent.

[0072] The residual quantizer includes a multi-layer quantizer and uses residual vector quantization (RVQ) technology to achieve high-precision compression. RVQ is a layered quantization technology that achieves high-precision compression by gradually approximating the residual. Its core idea is to progressively quantize the residual using a multi-layer codebook. The specific process is divided into two stages:

[0073] 1) Codebook training: given a set of feature vectors v i Represents the i-th eigenvector. These eigenvectors are low-dimensional representations of the input data and are used to train the codebook of residual vector quantization (RVQ); n represents the total number of eigenvectors; RVQ constructs an L-layer codebook by progressive residual clustering; for the l-th layer codebook The training process is expressed as:

[0074]

[0075] Among them, d represents the dimension of the feature vector, that is, each feature vector v i length, r represents the residual vector; in each layer of RVQ, the residual vector is the error after quantization of the previous layer, that is, r = v i -Q(v), where Q(v) is the quantized vector, c k represents the kth codebook vector, where codebook C is a set of vectors used to quantize the residual vector r; Represents the residual vector set of the l-1th layer, K is the capacity of the codebook of each layer; the initial residual Subsequent residuals are calculated recursively:

[0076]

[0077] where Q l (·) represents the quantization function of the lth layer;

[0078] 2) Quantization process: In the quantization stage, the input vector is quantized through each layer of codebook. Each layer of quantizer is responsible for quantizing the residual of the previous layer. Specifically, for the input vector v, the quantization output of the lth layer is:

[0079] v l =Q l (v l-1 )

[0080] Where v0 = v, v l Represents the quantized output of the lth layer; the final quantized result v quantized is the accumulation of quantized outputs of all layers:

[0081]

[0082] Distribute the trained autoencoder model and residual quantizer model to each participating node;

[0083] S4. After receiving the autoencoder model and residual quantizer model, each participating node uses the autoencoder model to compress the model parameters trained in step S2, mapping the model parameters to a low-dimensional representation. The compressed model parameters are further processed using residual quantization technology to improve the accuracy of data compression. The residual quantization technology consists of a stack of multiple sub-quantizers, each of which is responsible for quantizing the residual value of the previous quantizer.

[0084] S5. The central node collects the compressed and quantized model parameters uploaded by each participant node, and decodes them using the corresponding decoder to restore the original model parameters, and then aggregates the model and updates the global model.

[0085] The central node receives the compressed and quantized model parameters z from each participant node. q,i ; First, the quantized model parameters z are inverse quantized q,i Convert to low-dimensional representation z i ; Then, use the decoder The low-dimensional representation z i Reconstructed to approximate original model parameters The whole process is expressed as:

[0086]

[0087] in, Represents the inverse quantization operation of the lth layer residual quantization, v q,l is the quantized output of layer l, is the decoder function used to reconstruct the low-dimensional representation into the original model parameters;

[0088] Subsequently, the federated averaging algorithm FedAvg is used to aggregate all model parameters, and a weighted strategy is combined to improve aggregation efficiency and model convergence speed. The form of the federated averaging algorithm is:

[0089]

[0090] Among them, w agg is the model parameter after aggregation, w i is the model parameter uploaded by the i-th client, and m is the number of clients participating in the aggregation;

[0091] To deal with the problem of uneven data distribution, the weighted federated averaging algorithm is adopted, in which the model parameters of each client are weighted according to the number of its data samples:

[0092]

[0093] Among them, n i is the number of data samples of the i-th client, and n is the total number of samples of all clients.

[0094] The decoder is the inverse process of the encoder, which reconstructs the low-dimensional representation output by the encoder back to the original high-dimensional model parameters. The design structure of the decoder is symmetrical with that of the encoder to ensure that the model parameters can be effectively recovered from the compressed representation. During the reconstruction process, the decoder uses the nonlinear activation function ReLU to enhance the expressiveness of the model, thereby maintaining the key information of the model parameters during the compression and decompression process. The decoder is trained using the back propagation algorithm to adjust the decoder parameters by minimizing the reconstruction error. The reconstruction error L recon It can be quantified by the mean square error MSE as follows:

[0095]

[0096] Where D is the dimension of the model parameters, z i are the original model parameters, are the model parameters after reconstruction.

[0097] The training of the autoencoder is an unsupervised learning process, whose goal is to minimize the difference between the model parameters and their reconstructed version. This difference is quantified by the mean squared error (MSE) and used as the loss function for training. During the training process, the autoencoder adjusts the parameters of the encoder and decoder to minimize the loss function, thereby learning effective data compression and reconstruction strategies. The total loss function of the autoencoder is L autoencoder Expressed as:

[0098] L autoencoder =L encoder +L recon

[0099] L encoder is the encoder loss, by minimizing L autoencoder , the autoencoder is able to learn how to effectively compress the model parameters into a low-dimensional representation while retaining enough information to accurately reconstruct the original parameters.

[0100] S6. The updated global model is distributed to all participating nodes for the next round of training. Steps S4 to S5 are repeated until the predetermined number of training rounds is reached or the model performance meets the requirements. The update and distribution of the global model further includes the following three key steps:

[0101] 1) Two-stage compression: The client first compresses the model parameters θ into a low-dimensional representation z through the autoencoder, and then quantizes z using residual vector quantization (RVQ) to obtain the final transmitted compressed representation z q :

[0102]

[0103] 2) Error propagation analysis: The total reconstruction error consists of two parts: the autoencoder error and the quantization propagation error:

[0104]

[0105] Hypothetical decoder It's L g -Lipschitz continuous, then the quantized propagation error satisfies:

[0106]

[0107] Therefore, the upper bound of the total error is:

[0108]

[0109] Among them, ∈ quant is the quantization error; total The upper bound of the total error is the maximum possible error between the reconstructed parameters and the original parameters after the model is compressed by the autoencoder and the residual quantized. This upper bound consists of two parts: the reconstruction error of the autoencoder and the propagation of the quantization error. AE is the loss function of the autoencoder, which is the goal to be minimized during the training process of the autoencoder. It is defined as the mean square error (MSE) between the input data and the output reconstructed by the autoencoder. The mean square error is the average of the squares of the errors. Therefore, in order to get the average of the errors, we need to take the square root.

[0110] 3) Joint optimization goal: To balance compression efficiency and model accuracy, the following multi-objective optimization problem is used:

[0111]

[0112] in, represents the communication cost function, λ1,λ2,λ3 are trade-off coefficients that control the importance of autoencoding error, quantization error, and communication overhead, respectively.

[0113] On the other hand, the present invention also provides a federated learning acceleration system based on autoencoders and residual quantization, which includes the following modules:

[0114] The autoencoder module consists of two sub-modules: an encoder and a decoder. The encoder is responsible for mapping the model parameters to a low-dimensional representation space, and the decoder is responsible for reconstructing the low-dimensional representation back to the original high-dimensional model parameters. The autoencoder module is trained using the backpropagation algorithm and gradient descent to minimize the difference between the model parameters and their reconstructed versions.

[0115] The residual quantization module includes a multi-layer quantizer that maps the encoder output to the vector in the quantizer codebook with the smallest Euclidean distance to the output, and then quantizes the residual layer by layer to further reduce the dimension of the model parameters and retain key information;

[0116] The federated averaging algorithm module is used to perform aggregation of model parameters at the central node, taking into account the number of data samples of each client through a weighted federated averaging algorithm to generate global model updates;

[0117] Dynamic learning rate adjustment module, which dynamically adjusts the contribution of each participating node to the global model update based on the local training effect of each participating node to optimize the training efficiency and stability of the model;

[0118] The data preprocessing module is used to standardize and normalize the data of each participating node to improve the efficiency and accuracy of model training; and encryption technology is used to protect the privacy and integrity of data during transmission.

[0119] The performance monitoring module monitors the model performance during the federated learning process, including key indicators such as accuracy and loss rate, and triggers retraining or adjustment of the training strategy when the performance does not meet the preset requirements.

[0120] The user interface module provides a user interaction interface that allows users to configure training parameters, view training progress and performance indicators, and deploy models.

[0121] One embodiment of the present invention is applied in a medical scenario. A hospital wants to use machine learning to build a disease prediction model. However, the hospital's internal data volume is insufficient to train a high-precision model. Therefore, the hospital needs to collaborate with other hospitals to build a model. Based on this scenario, assume that there are five hospitals in the federated learning network, represented by H1, H2, H3, H4, and H5.

[0122] In the federated learning network, the hospital H5 with the above scenario requirements acts as the task initiator node to initiate a joint disease prediction model building task to other hospitals. Task information includes: task type, whether the data used in this task is disease prediction related data, the deep learning model used in the task, the number of training rounds, etc. The representation of disease prediction data is (x i ,y i ), x i Represents the patient's characteristic data (such as age, gender, medical history and other medical data), y i The data is labeled and used to determine whether each patient has a certain disease. The disease prediction model established in this task is a binary classification task. The deep learning model used can determine whether the patient has a certain disease based on the input patient feature information.

[0123] The central node in the federated learning network (which can be a hospital or an independent coordination agency) initiates the federated learning task. The task information includes the task type, the relevant data types involved, the initial global model, and the scheduled number of training rounds. After receiving the initial global model, each participating hospital uses local data for training to generate local model parameters. Autoencoders and quantizers are not used in this stage to ensure that the initial training of the model is not affected by compression. Synchronously, on the central node, the autoencoder and residual quantizer are trained in parallel based on the local data of each participating hospital. The autoencoder maps the model parameters to a low-dimensional space through multi-layer transformations, removing redundant information while retaining the core features. The residual quantizer further quantizes the compressed parameters to reduce the amount of data transmission.

[0124] After the autoencoder is trained, the central node distributes the trained autoencoder model to each participating hospital. After the residual quantizer is trained, the central node distributes the trained residual quantizer model to each participating hospital. Upon receiving the autoencoder and residual quantizer models, each participating hospital uses the autoencoder to compress the trained model parameters, mapping them to a low-dimensional space. The model parameters compressed by the autoencoder are further processed using residual quantization technology. Residual quantization technology consists of a stack of multiple sub-quantizers, each of which sequentially quantizes the residual value of the previous quantizer.

[0125] The central node collects the compressed and quantized model parameters uploaded by each participating hospital and decodes them using the corresponding decoder to reconstruct the original model parameters, thereby performing model aggregation. This model parameter aggregation uses the FedAvg algorithm, combined with a weighted strategy to improve aggregation efficiency and model convergence speed.

[0126]

[0127] Among them, w agg is the model parameter after aggregation, n i is the number of data samples of the i-th client, and n is the total number of samples of all clients.

[0128] The updated global model is distributed to participating hospitals for the next round of training. This process is repeated until the predetermined number of training rounds is reached or the model performance meets the specified requirements. Throughout the federated learning process, hospitals do not need to share raw data; instead, they share only compressed and quantized model parameters. This achieves the goal of joint modeling, improving model performance while protecting patient data privacy.

[0129] See also Figure 2 An embodiment of the present invention provides a federated learning acceleration device based on an autoencoder and residual quantization, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement the federated learning acceleration method based on an autoencoder and residual quantization in the above embodiment.

[0130] The embodiment of the federated learning acceleration device based on autoencoder and residual quantization provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located to read the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 2 As shown in the figure, a hardware structure diagram of a federated learning acceleration device based on autoencoder and residual quantization provided by the present invention is provided in any device with data processing capability, except Figure 2 In addition to the processor, memory, network interface and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware according to the actual functions of the device with data processing capabilities, which will not be described in detail.

[0131] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically detailed in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here. For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying any creative work.

[0132] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a federated learning acceleration method based on an autoencoder and residual quantization in the above-mentioned embodiment. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the above-mentioned embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0133] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A federated learning acceleration method based on autoencoders and residual quantization, characterized in that: The steps of this method are as follows: S1. The central node in the federated learning network initiates a federated learning task to the network, where the federated learning task information includes the task type, the relevant data types used for training, the initial global model, and the number of training rounds; S2. Each participant node in the federated learning network receives the initial global model and trains the model using local data; S3. Use the local data of each participant node to train the autoencoder model and the residual quantizer model in parallel on the central node; distribute the trained autoencoder model and residual quantizer model to each participant node; S4. After receiving the autoencoder model and residual quantizer model, each participating node uses the autoencoder model to compress the model parameters trained in step S2, mapping the model parameters to a low-dimensional representation. The compressed model parameters are further processed using residual quantization technology to improve the accuracy of data compression. The residual quantization technology consists of a stack of multiple sub-quantizers, each of which is responsible for quantizing the residual value of the previous quantizer. S5. The central node collects the compressed and quantized model parameters uploaded by each participant node, and decodes them using the corresponding decoder to restore the original model parameters, and then aggregates the model and updates the global model. S6. The updated global model is distributed to each participant node for the next round of training, and steps S4 to S5 are repeated until the predetermined number of training rounds is reached or the model performance meets the requirements.

2. The method for accelerating federated learning based on autoencoders and residual quantization according to claim 1, wherein: The autoencoder model in S3 consists of two parts: an encoder and a decoder. The encoder part is responsible for mapping the model parameters trained by each participant node into a low-dimensional representation space. This process is achieved through a series of linear and nonlinear transformations to capture the essential characteristics of the model parameters while removing redundant information; the encoder consists of multiple hidden layers, each layer contains weights W and bias b parameters, which are optimized during training through backpropagation algorithm and gradient descent method.

3. The method for accelerating federated learning based on autoencoders and residual quantization according to claim 2, wherein: The decoder is the inverse process of the encoder, which reconstructs the low-dimensional representation output by the encoder back to the original high-dimensional model parameters. The decoder is designed to be symmetrical with the encoder to ensure that the model parameters can be effectively recovered from the compressed representation. During the reconstruction process, the decoder uses the nonlinear activation function ReLU to enhance the expressiveness of the model, thereby maintaining the key information of the model parameters during compression and decompression; The decoder is trained using the back propagation algorithm to adjust the decoder parameters by minimizing the reconstruction error. recon It can be quantified by the mean square error MSE as follows: Where D is the dimension of the model parameters, z i are the original model parameters, are the model parameters after reconstruction. The training of the autoencoder is an unsupervised learning process, whose goal is to minimize the difference between the model parameters and their reconstructed version. This difference is quantified by the mean squared error (MSE) and used as the loss function for training. During the training process, the autoencoder minimizes the loss function by adjusting the parameters of the encoder and decoder, thereby learning effective data compression and reconstruction strategies; the total loss function of the autoencoder L autoencoder Expressed as: L autoencoder =L encoder +L recon L encoder is the encoder loss, by minimizing L autoencoder , the autoencoder is able to learn how to effectively compress the model parameters into a low-dimensional representation while retaining enough information to accurately reconstruct the original parameters.

4. The method for accelerating federated learning based on autoencoders and residual quantization according to claim 1, wherein: The residual quantizer in S3 includes a multi-layer quantizer and uses residual vector quantization (RVQ) technology to achieve high-precision compression. RVQ is a hierarchical quantization technology that achieves high-precision compression by gradually approximating the residual. Its core idea is to progressively quantize the residual using a multi-layer codebook. The specific process is divided into two stages: 1) Codebook training: given a set of feature vectors v i represents the i-th eigenvector. These feature vectors are low-dimensional representations of the input data and are used to train the codebook of residual vector quantization (RVQ). n represents the total number of feature vectors. RVQ constructs an L-layer codebook by progressive residual clustering. For the l-th layer codebook C l , and its training process is expressed as: Among them, d represents the dimension of the feature vector, that is, each feature vector v i length, r represents the residual vector; in each layer of RVQ, the residual vector is the error after quantization of the previous layer, that is, r = v i -Q(v), where Q(v) is the quantized vector, c k represents the kth codebook vector, where codebook C is a set of vectors used to quantize the residual vector r; Represents the residual vector set of the l-1th layer, K is the capacity of the codebook of each layer; the initial residual Subsequent residuals are calculated recursively: where Q l (·) represents the quantization function of the lth layer; 2) Quantization process: In the quantization stage, the input vector is quantized through each layer of codebook. Each layer of quantizer is responsible for quantizing the residual of the previous layer. Specifically, for the input vector v, the quantization output of the lth layer is: v l =Q l (v l-1 ) Where v0 = v, v l Represents the quantized output of the lth layer; the final quantized result v quantized is the accumulation of quantized outputs of all layers:

5. The method for accelerating federated learning based on autoencoders and residual quantization according to claim 1, wherein: In S5, the central node receives the compressed and quantized model parameters from each participant node; first, the quantized model parameters are converted into low-dimensional representations through inverse quantization operations; then, the decoder is used to Reconstruct the low-dimensional representation into an approximation of the original model parameters The whole process is expressed as: in, Represents the inverse quantization operation of the lth layer residual quantization, v q,l is the quantized output of layer l, is the decoder function used to reconstruct the low-dimensional representation into the original model parameters; Subsequently, the federated averaging algorithm FedAvg is used to aggregate all model parameters, and a weighted strategy is combined to improve aggregation efficiency and model convergence speed. The form of the federated averaging algorithm is: Among them, w agg is the model parameter after aggregation, w i is the model parameter uploaded by the i-th client, and m is the number of clients participating in the aggregation; To deal with the problem of uneven data distribution, the weighted federated averaging algorithm is adopted, in which the model parameters of each client are weighted according to the number of its data samples: Among them, n i is the number of data samples of the i-th client, and n is the total number of samples of all clients.

6. The method for accelerating federated learning based on autoencoders and residual quantization according to claim 1, wherein: The updating and distribution of the global model in S6 further includes the following three key steps: 1) Two-stage compression: The client first passes the autoencoder f e Compress the model parameter θ into a low-dimensional representation z, and then quantize z using Residual Vector Quantization (RVQ) to obtain the final transmitted compressed representation z q : 2) Error propagation analysis: The total reconstruction error consists of two parts: the autoencoder error and the quantization propagation error: Hypothetical decoder is the Lipschitz continuity condition L g If it is continuous, the quantized propagation error satisfies: Therefore, the upper bound of the total error is: Among them, ∈ quant is the quantization error, and total represents the upper bound of the total error. The upper bound of the total error is the maximum possible error between the reconstructed parameters and the original parameters of the model after autoencoder compression and residual quantization. This upper bound consists of two parts: the reconstruction error of the autoencoder and the propagation of the quantization error. AE is the loss function of the autoencoder, which is the goal to be minimized during the training process of the autoencoder and is defined as the mean squared error (MSE) between the input data and the output reconstructed by the autoencoder. The mean squared error is the average of the squared errors. Therefore, in order to obtain the average value of the error, it is necessary to take the square root. 3) Joint optimization goal: To balance compression efficiency and model accuracy, the following multi-objective optimization problem is used: in, represents the communication cost function, λ1,λ2,λ3 are trade-off coefficients that control the importance of autoencoding error, quantization error, and communication overhead, respectively.

7. A federated learning acceleration system based on autoencoders and residual quantization, characterized by: The system includes the following modules: The autoencoder module consists of two sub-modules: an encoder and a decoder. The encoder is responsible for mapping the model parameters to a low-dimensional representation space, and the decoder is responsible for reconstructing the low-dimensional representation back to the original high-dimensional model parameters. The autoencoder module is trained using the backpropagation algorithm and gradient descent to minimize the difference between the model parameters and their reconstructed versions. The residual quantization module includes a multi-layer quantizer that maps the encoder output to the vector in the quantizer codebook with the smallest Euclidean distance to the output, and then quantizes the residual layer by layer to further reduce the dimension of the model parameters and retain key information; The federated averaging algorithm module is used to perform aggregation of model parameters at the central node, taking into account the number of data samples of each client through a weighted federated averaging algorithm to generate global model updates; Dynamic learning rate adjustment module, which dynamically adjusts the contribution of each participating node to the global model update based on the local training effect of each participating node to optimize the training efficiency and stability of the model; The data preprocessing module is used to standardize and normalize the data of each participating node to improve the efficiency and accuracy of model training; and encryption technology is used to protect the privacy and integrity of data during transmission. The performance monitoring module monitors the model performance during the federated learning process, including key indicators such as accuracy and loss rate, and triggers retraining or adjustment of the training strategy when the performance does not meet the preset requirements. The user interface module provides a user interaction interface that allows users to configure training parameters, view training progress and performance indicators, and deploy models.

8. A federated learning acceleration device based on an autoencoder and residual quantization, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a federated learning acceleration method based on autoencoder and residual quantization according to any one of claims 1 to 6.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a federated learning acceleration method based on an autoencoder and residual quantization according to any one of claims 1 to 6 is implemented.

Images