Residual adaptive compression efficient communication federal learning model training method and system

Abstract

The invention relates to the technical field of distributed learning and wireless communication, in particular to an efficient communication federated learning model training method and system based on residual adaptive compression. The client obtains and updates the global model; obtaining a local model residual error by utilizing the local model memory set and the updated model weight; carrying out adaptive sparse compression on the residual error and transmitting the residual error to a server; the server selects a reliable client to carry out model weight prediction and aggregation; and broadcasting the updated global model to the client, and repeating the steps until convergence. Through adaptive compression and reliable client selection, the communication overhead is remarkably reduced, and the model convergence speed and the final performance are improved. The method is particularly suitable for edge computing scenes with limited resources, effectively solves the key problems of communication efficiency, model performance, privacy protection and the like in federated learning, and provides new possibility for wide application of federated learning.

Description

Technical Field

[0001] The present invention relates to the field of distributed learning and wireless communication technology, and in particular to a method and system for training an efficient communication federated learning model with residual adaptive compression. Background Art

[0002] Federated learning is an efficient distributed machine learning framework that uses distributed data on client devices through network connections to train various models without uploading the original data. Federated learning can protect user privacy while using distributed network data for federated model training, reaching or approaching the performance of centralized training. During model training, user clients do not need to transmit original data, but instead transmit model weights. As the scale of machine learning models continues to expand, the existing mobile communication infrastructure does not always meet the requirements of federated learning in terms of bandwidth and latency, which limits the widespread deployment of federated learning. Communication efficiency is a key factor in deploying federated learning, especially in wireless networks.

[0003] Most existing federated learning compression methods directly compress and quantize the current model weights or gradients or discard some parameters, and send the compressed model weights or gradients to the server for aggregation. These compression methods use a uniform compression rate for all clients and do not consider the communication heterogeneity in practical applications. Therefore, under the condition that the client's communication resources are limited, it is necessary to adaptively compress the client's transmission parameters to reduce the uplink communication overhead and improve the efficiency of global model training. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is that most of the existing federated learning compression methods directly compress and quantize the current model weights or gradients or discard some parameters, and send the compressed model weights or gradients to the server for aggregation. These compression methods use a uniform compression rate for all clients and do not consider the communication heterogeneity in practical applications. Therefore, under the condition that the communication resources of the client are limited, it is necessary to adaptively compress the transmission parameters of the client to reduce the communication overhead of the uplink and improve the efficiency of global model training.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides an efficient communication federated learning model training method with residual adaptive compression, which includes: a client obtains a federated learning global model and uses a local data set to update the model;

[0008] The client uses the local model memory set and updates the model weights to obtain the local model residual;

[0009] The client performs adaptive sparse compression on the local model residual, transmits it to the server via the uplink, and generates historical model weights and saves them in the local model memory set;

[0010] The server selects a reliable client to make local model weight predictions, saves them in the global model memory set, and completes the federated learning model aggregation;

[0011] The server broadcasts the aggregated updated global model to each client through the downlink, and repeats the above steps multiple times until each client model converges to the optimal global model.

[0012] As a preferred solution of the efficient communication federated learning model training method for residual adaptive compression of the present invention, wherein: the client obtains the federated learning global model, and uses the local data set to update the model, including the following steps:

[0013] Each client updates the local model based on the local data set. On any client, the stochastic gradient descent algorithm is used to reduce the loss function of the local model. The local model is updated using the following formula:

[0014]

[0015] Among them, w t,m With w t-1,m Respectively represent the client U m The local model weights updated in round t and round t-1 training, D t,m Indicates the update of w t,m The training data set is provided by the client U m The local dataset D m Randomly generated, η t is the learning rate of the tth iteration; is the loss function F(w t-1,m ,D t,m )About w t-1,m gradient.

[0016] As a preferred solution of the efficient communication federated learning model training method for residual adaptive compression of the present invention, wherein: the client obtains the local model residual by using the local model memory set and updating the model weight, including the following steps:

[0017] The client inputs the K local historical model weights in the local model memory set into the model prediction module to obtain the prediction model weight. The prediction module calculates the model weight of the output prediction according to the following formula:

[0018]

[0019] in, Indicates that client U is m The prediction model weights, Indicates that the client U is saved in the local model memory during the tkth round of training m The historical model weight, λ k is the weight coefficient, and

[0020] The client uses the prediction model weight and the update model weight to calculate the local model residual. The local model residual is calculated using the following formula:

[0021]

[0022] in, represents the prediction model weight; w t,m Indicates updating model weights; r t,m Indicates client U m The local model residual obtained in the tth round of training.

[0023] As a preferred solution of the efficient communication federated learning model training method with residual adaptive compression of the present invention, the client adaptively sparsely compresses the local model residual, transmits it to the server through the uplink, and generates historical model weights and stores them in the local model memory set, including the following steps:

[0024] Sparse compression representation converts the original vector into a sparse vector with a small number of non-zero elements by removing some unimportant elements in the original vector;

[0025] The sparse compression method reduces the bandwidth occupied by communication by sending elements in the local model residual that are greater than a certain threshold to the server. Different sparse thresholds will affect the communication overhead and model accuracy. The sparse threshold needs to be set dynamically, and the optimal sparse threshold needs to be selected based on the model accuracy and iteration rounds. To achieve adaptive selection of the sparse threshold, the sparse compression rate needs to be set dynamically:

[0026]

[0027] Among them, ρ t,m and acc t,m Respectively represent the client U m The sparse compression rate and model accuracy in the tth round of training, T represents the total number of training rounds, α and β are weight coefficients;

[0028] The sparsification threshold is determined based on the sparsity compression ratio and the local model residual:

[0029]

[0030] Among them, θ t,m Indicates client U m The sparsification threshold at the tth round of training, J t,m =ρ t,m d represents the number of remaining elements after model residual compression, and d represents the local model residual r t,m The number of elements of Indicates that |r t,m |After sorting in descending order, take the order in the Jth t,m The parameter value corresponding to each position, |·| means taking the absolute value;

[0031] The elements in the local model residual are calculated according to the following formula to obtain the compressed model residual:

[0032]

[0033] in, and r t,m, i Represent the compression model residuals and the local model residual r t,m The i-th element of client U m Compress the model residual Sent to the server via uplink;

[0034] The client uses the compressed model residual and the predicted model weight to obtain the historical model weight:

[0035]

[0036] in, Indicates that the client U is saved in the local model memory during the tth round of training m The historical model weights.

[0037] As a preferred solution of the efficient communication federated learning model training method for residual adaptive compression of the present invention, the server selects a reliable client to perform local model weight prediction, saves it in the global model memory set, and completes the federated learning model aggregation, including the following steps:

[0038] The server evaluates the client's communication reliability based on the packet error rate between the client and the server;

[0039] The server inputs the received local model residual and K global historical model weights into the model prediction module to obtain the predicted local model weights;

[0040] The server aggregates the prediction model weights of each client using a federated learning model;

[0041] Due to the unreliability of the wireless uplink, the client will generate errors when transmitting the model residual. The server needs to select reliable clients to participate in the model aggregation of federated learning.

[0042] The server evaluates the client's communication reliability based on the packet error rate between the client and the server. The packet error rate between the client and the server is calculated using the following formula:

[0043]

[0044] Among them, q m It is client U m Packet error rate between the server and the server, h m It is client U m Channel gain to the server, Yes m The mathematical expectation of B UL is the client uplink transmission bandwidth, N 0 is the noise power spectral density, P m It is client U m The transmit power of the client and the server is τ, and τ is the inverse of the coding gain for data transmission between the client and the server.

[0045] As a preferred solution of the efficient communication federated learning model training method for residual adaptive compression of the present invention, the server inputs the received local model residual and K global historical model weights into the model prediction module to obtain the predicted local model weight, which is expressed as follows:

[0046]

[0047] in, Represents the client U generated by the server prediction m The prediction model weight at iteration t, Indicates that the client U is saved in the global model memory set at the tkth iteration m The historical model weight, λ k is the weight coefficient, and

[0048] As a preferred solution of the efficient communication federated learning model training method for residual adaptive compression of the present invention, the server performs federated learning model aggregation on the prediction model weights of each client:

[0049]

[0050] Among them, wt, F represents the model weight obtained by server aggregation in the tth round of training, N m Indicates that the server gets the client U m The size of the local dataset;

[0051] Indicates client U m Whether to participate in the tth round of federated learning training, Indicates client U m Participate in this round of federated learning training, otherwise Indicates client U m Whether there is an error when transmitting the model residual to the server;

[0052] Indicates client U m No error occurred when transmitting the model residual to the server. The probability of no error is 1-q m ,otherwise

[0053] In a second aspect, an embodiment of the present invention provides an efficient communication federated learning model training system with residual adaptive compression, which includes an acquisition module, a client obtains a global model of federated learning, and uses a local data set to update the model;

[0054] Update module, the client uses the local model memory set and updates the model weights to obtain the local model residual;

[0055] In the transmission module, the client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates historical model weights and saves them in the local model memory set;

[0056] In the prediction module, the server selects a reliable client to make local model weight predictions, saves them in the global model memory set, and completes the federated learning model aggregation;

[0057] In the convergence module, the server broadcasts the aggregated updated global model to each client through the downlink, and repeats the above steps multiple times until each client model converges to the optimal global model.

[0058] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the efficient communication federated learning model training method with residual adaptive compression as described in the first aspect of the present invention are implemented.

[0059] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the efficient communication federated learning model training method with residual adaptive compression as described in the first aspect of the present invention are implemented.

[0060] The beneficial effect of the present invention is that the present invention can significantly reduce the communication overhead of the federated learning uplink and improve the training efficiency of federated learning by adaptively compressing the model residual of the client during the model training process.

[0061] In the process of federated learning model aggregation, the present invention can effectively improve the convergence speed of the federated learning model by selecting reliable clients and reduce the adverse effects of unreliable users on model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 Flowchart of the method for training a communication-efficient federated learning model with residual adaptive compression;

[0064] Figure 2 Computer device diagram for efficient communication federated learning model training method with residual adaptive compression;

[0065] Figure 3 Model training diagram for a communication-efficient federated learning model training method with residual adaptive compression. DETAILED DESCRIPTION

[0066] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0067] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0068] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0069] Example 1

[0070] Reference Figure 1-2 , which is the first embodiment of the present invention, and provides an efficient communication federated learning model training method with residual adaptive compression, comprising:

[0071] S100: The client obtains the global model of federated learning and uses the local dataset to update the model;

[0072] S101: The client obtains the global model of federated learning and uses the local dataset to update the model, including the following steps:

[0073] Each client updates the local model based on the local data set. On any client, the stochastic gradient descent algorithm is used to reduce the loss function of the local model. The local model is updated using the following formula:

[0074]

[0075] Among them, w t,m With w t-1,m Respectively represent the client U m The local model weights updated in round t and round t-1 training, D t,m Indicates the update of w t,m The training data set is provided by the client U m The local dataset D m Randomly generated, η t is the learning rate of the tth iteration; is the loss function F(w t-1,m ,D t,m )About w t-1,m gradient.

[0076] S200: The client obtains the local model residual by using the local model memory set and updating the model weights;

[0077] S201: The client uses the local model memory set and updates the model weights to obtain the local model residual, including the following steps:

[0078] The client inputs the K local historical model weights in the local model memory set into the model prediction module to obtain the prediction model weight. The prediction module calculates the model weight of the output prediction according to the following formula:

[0079]

[0080] in, Indicates that client U is m The prediction model weights, Indicates that the client U is saved in the local model memory during the tkth round of training m The historical model weight, λ k is the weight coefficient, and

[0081] The client uses the prediction model weight and the update model weight to calculate the local model residual. The local model residual is calculated using the following formula:

[0082]

[0083] in, represents the prediction model weight; w t,m Indicates updating model weights; r t,m Indicates client U m The local model residual obtained in the tth round of training.

[0084] S300: The client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates historical model weights and saves them in the local model memory set;

[0085] S301: The client performs adaptive sparse compression on the local model residual, transmits it to the server via the uplink, and generates historical model weights and saves them in the local model memory set, including the following steps:

[0086] Sparse compression representation converts the original vector into a sparse vector with a small number of non-zero elements by removing some unimportant elements in the original vector;

[0087] The sparse compression method reduces the bandwidth occupied by communication by sending elements in the local model residual that are greater than a certain threshold to the server. Different sparse thresholds will affect the communication overhead and model accuracy. The sparse threshold needs to be set dynamically, and the optimal sparse threshold needs to be selected based on the model accuracy and iteration rounds. To achieve adaptive selection of the sparse threshold, the sparse compression rate needs to be set dynamically:

[0088]

[0089] Among them, ρ t,m and acc t,m Respectively represent the client U m The sparse compression rate and model accuracy in the tth round of training, T represents the total number of training rounds, α and β are weight coefficients;

[0090] The sparsification threshold is determined based on the sparsity compression ratio and the local model residual:

[0091]

[0092] Among them, θ t,m Indicates client U m The sparsification threshold at the tth round of training, J t,m =ρ t,m d represents the number of remaining elements after model residual compression, and d represents the local model residual r t,m The number of elements of Indicates that |r t,m |After sorting in descending order, take the order in the Jth t,m The parameter value corresponding to each position, |·| means taking the absolute value;

[0093] The elements in the local model residual are calculated according to the following formula to obtain the compressed model residual:

[0094]

[0095] in, and r t,m,i Represent the compression model residuals and the local model residual r t,m The i-th element of client U m Compress the model residual Sent to the server via uplink;

[0096] The client uses the compressed model residual and the predicted model weight to obtain the historical model weight:

[0097]

[0098] in, Indicates that the client U is saved in the local model memory during the tth round of training m The historical model weights.

[0099] S400: The server selects a reliable client to perform local model weight prediction, saves it in the global model memory set, and completes the federated learning model aggregation;

[0100] S401: The server selects a reliable client to perform local model weight prediction, saves it in the global model memory set, and completes the federated learning model aggregation, including the following steps:

[0101] The server evaluates the client's communication reliability based on the packet error rate between the client and the server;

[0102] The server inputs the received local model residual and K global historical model weights into the model prediction module to obtain the predicted local model weights;

[0103] The server aggregates the prediction model weights of each client using a federated learning model;

[0104] Due to the unreliability of the wireless uplink, the client will generate errors when transmitting the model residual. The server needs to select reliable clients to participate in the model aggregation of federated learning.

[0105] The server evaluates the client's communication reliability based on the packet error rate between the client and the server. The packet error rate between the client and the server is calculated using the following formula:

[0106]

[0107] Among them, q m It is client U m Packet error rate between the server and the server, h m It is client U m Channel gain to the server, Yes m The mathematical expectation of B UL is the client uplink transmission bandwidth, N 0 is the noise power spectral density, P m It is client U m The transmit power of the client and the server is τ, and τ is the inverse of the coding gain for data transmission between the client and the server.

[0108] S402: The server inputs the received local model residual and K global historical model weights into the model prediction module to obtain the predicted local model weight, which is expressed as:

[0109]

[0110] in, Represents the client U generated by the server prediction m The prediction model weight at iteration t, Indicates that the client U is saved in the global model memory set at the tkth iteration m The historical model weight, λ k is the weight coefficient, and

[0111] S403: The server aggregates the prediction model weights of each client using a federated learning model:

[0112]

[0113] Among them, wt,F represents the model weight obtained by server aggregation in the tth round of training, N m Indicates that the server gets the client U m The size of the local dataset;

[0114] Indicates client U m Whether to participate in the tth round of federated learning training, Indicates client U m Participate in this round of federated learning training, otherwise Indicates client U m Whether there is an error when transmitting the model residual to the server;

[0115] Indicates client U m No error occurred when transmitting the model residual to the server. The probability of no error is 1-q m ,otherwise

[0116] S500: The server broadcasts the aggregated updated global model to each client via a downlink, and repeats the above steps multiple times until each client model converges to the optimal global model.

[0117] Model residual: refers to the difference between the local model and the prediction model, reflecting the information of model update.

[0118] Adaptive sparse compression: A compression method that dynamically adjusts the sparsification threshold based on model accuracy and iteration rounds.

[0119] Adaptive sparse compression step: By dynamically adjusting the sparsification threshold, the communication overhead is minimized while ensuring model performance.

[0120] Reliable client selection step: By evaluating the reliability of client communication, the negative impact of the client on model training is avoided.

[0121] Furthermore, this embodiment also provides an efficient communication federated learning model training system with residual adaptive compression, including:

[0122] Acquisition module: The client obtains the global model of federated learning and uses the local dataset to update the model;

[0123] Update module, the client uses the local model memory set and updates the model weights to obtain the local model residual;

[0124] In the transmission module, the client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates historical model weights and saves them in the local model memory set;

[0125] In the prediction module, the server selects a reliable client to make local model weight predictions, saves them in the global model memory set, and completes the federated learning model aggregation;

[0126] In the convergence module, the server broadcasts the aggregated updated global model to each client through the downlink, and repeats the above steps multiple times until each client model converges to the optimal global model.

[0127] This embodiment also provides a computer device, which is suitable for the case of an efficient communication federated learning model training method with residual adaptive compression, including a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the efficient communication federated learning model training method with residual adaptive compression as proposed in the above embodiment.

[0128] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0129] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, an efficient communication federated learning model training method for realizing residual adaptive compression as proposed in the above embodiment is implemented.

[0130] In summary, by obtaining the global model of federated learning through the client and using the local data set to update the model, it is possible to use distributed data for model training while protecting data privacy. This method avoids the direct transmission of original data, effectively reduces the risk of data leakage, and can also make full use of the large amount of data resources scattered on various clients, ultimately achieving the beneficial effect of protecting privacy while improving the model training effect.

[0131] The extraction and compression of model update information is achieved by the client using the local model memory set and updating the model weights to obtain the local model residual. This residual-based method can effectively capture the key information of model changes, greatly reduce the amount of data that needs to be transmitted, and thus significantly reduce the communication overhead, ultimately achieving the beneficial effect of improving the efficiency of federated learning under limited bandwidth conditions.

[0132] The client performs adaptive sparse compression on the local model residuals to achieve a dynamic balance between communication efficiency and model performance. This adaptive method can automatically adjust the compression rate according to the current model training status and communication conditions, while ensuring model convergence and minimizing data transmission, ultimately achieving the beneficial effect of optimizing federated learning performance in resource-constrained scenarios.

[0133] The server selects reliable clients to predict the local model weights, which enables adaptation to unstable communication environments. This screening mechanism based on communication reliability can effectively reduce the negative impact of unreliable clients on the global model, improve the robustness of federated learning in complex network environments, and ultimately achieve the beneficial effect of improving model training quality and convergence speed.

[0134] The server broadcasts the aggregated updated global model to each client through a downlink and performs multiple rounds of iterations to achieve collaborative optimization of the distributed learning process. This iterative learning method can continuously integrate the knowledge of each client and gradually improve the performance of the global model, ultimately achieving the beneficial effect of high-quality model training while protecting privacy.

[0135] In summary, this solution cleverly solves several key issues in federated learning, including communication efficiency, model performance, and privacy protection. In particular, the two steps of adaptive sparse compression and reliable client selection have brought unexpected technical effects: they not only significantly reduce communication overhead, but also improve the convergence speed and final performance of the model. This approach is particularly valuable in resource-constrained edge computing scenarios, and provides new possibilities for the widespread application of federated learning.

[0136] Example 2

[0137] Reference Figure 2 - Figure 3 , which is the second embodiment of the present invention, and this embodiment provides an efficient communication federated learning model training method with residual adaptive compression. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0138] To verify the effectiveness of the proposed efficient communication federated learning model training method with residual adaptive compression, an experiment was conducted in an IoT system consisting of 10,000 smart factory devices. Each device acts as a client, collects its own operating data and participates in federated learning, with the goal of building an accurate equipment failure prediction model. The experimental environment uses Python 3.8 and PyTorch 1.9.0 frameworks and runs on a server equipped with an Intel Xeon E5-2680 v4 CPU and 128 GB RAM.

[0139] First, a simple three-layer fully connected neural network is initialized as the global model. The input features include 10 parameters such as device temperature, vibration, and current, and the output is the probability of failure within the next 24 hours. The local dataset of each client contains 1000 historical records. The experiment compares three methods: (1) traditional federated averaging (FedAvg), (2) fixed threshold sparsification (FS), and (3) the residual adaptive compression (RAC) method of the present invention.

[0140] In the RAC method, the client first uses the stochastic gradient descent algorithm to update the local model. Then, the local model memory set (which stores the historical weights of the last five rounds) is used to predict the current model weights and calculate the residual with the actual updated weights. The residual is adaptively sparsely compressed, where the sparse compression rate τ is selected as:

[0141] τ=0.1*(1-acc)+0.9*(1-t / T)

[0142] Here acc is the current model accuracy, t is the current iteration round, and T is the total rounds (set to 100).

[0143] On the server side, reliable clients are first screened based on the packet error rate (PER), with the PER threshold set to 0.1. The compressed residuals of the selected clients are then decompressed, combined with the prediction weights to obtain the complete local model weights, and finally weighted averaged to obtain the updated global model.

[0144] The experiment lasted for 100 rounds of training, with 1,000 clients randomly selected to participate in each round. During the entire process, indicators such as model accuracy, communication overhead, and training time were recorded. The results are shown in the following table:

[0145]

[0146] By analyzing the data in the above table, we can draw the following conclusions:

[0147] In terms of accuracy: the RAC method proposed in this paper finally achieved an accuracy of 93.7%, which is 1.2 percentage points higher than the traditional FedAvg method and 1.9 percentage points higher than the fixed threshold sparsification (FS) method. This shows that the RAC method not only does not lose model performance due to compression, but also improves the generalization ability of the model through an adaptive mechanism.

[0148] Communication efficiency: The average communication volume per round of the RAC method is only 38MB, which is 85.2% less than that of the FedAvg method and 40.6% less than that of the FS method. This significant reduction in communication volume directly reflects the advantages of the present invention in improving communication efficiency, and is particularly suitable for application in bandwidth-constrained IoT environments.

[0149] Training time: Thanks to the significant reduction in communication volume, the RAC method shortens the total training time to 12 hours, saving 75% of the time compared to the FedAvg method and 33.3% of the time compared to the FS method. This efficiency improvement enables the model to be deployed to the production environment faster, improving the system's responsiveness.

[0150] Convergence speed: The RAC method reached convergence in only 62 rounds, 23 rounds earlier than the FedAvg method and 16 rounds earlier than the FS method. This shows that the adaptive compression strategy of the present invention not only does not affect the convergence of the model, but accelerates the convergence process through effective information extraction.

[0151] Further analysis revealed that the advantages of the RAC method mainly come from two innovations:

[0152] First, the residual-based compression strategy captures the key information of the model update. In the experiment, it was observed that as the training progressed, the amplitude of the model update gradually decreased. The RAC method can automatically adjust the compression rate, retaining more information in the early stage of training and performing more aggressive compression in the later stage. This dynamic adjustment mechanism not only ensures the learning effect of the model, but also maximizes the communication efficiency.

[0153] Secondly, the client selection mechanism based on communication reliability significantly improves the robustness of the model. During the experiment, 10% of the clients were simulated to have unstable network connections. The RAC method effectively avoids the interference of these unstable factors on the global model by screening reliable clients, thereby maintaining high performance even in complex practical environments.

[0154] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An efficient communication federated learning model training method with residual adaptive compression, characterized by: include, The client obtains the global model of federated learning and uses the local dataset to update the model; The client uses the local model memory set and updates the model weights to obtain the local model residual; The client performs adaptive sparse compression on the local model residual, transmits it to the server via the uplink, and generates historical model weights and saves them in the local model memory set; The server selects a reliable client to make local model weight predictions, saves them in the global model memory set, and completes the federated learning model aggregation; The server broadcasts the aggregated updated global model to each client through the downlink, and repeats the above steps multiple times until each client model converges to the optimal global model.

2. The efficient communication federated learning model training method with residual adaptive compression as claimed in claim 1, characterized in that: The client obtains the global model of federated learning and uses the local data set to update the model, including the following steps: Each client updates the local model based on the local data set. On any client, the stochastic gradient descent algorithm is used to reduce the loss function of the local model. The local model is updated using the following formula: Among them, w t,m With w t-1,m Respectively represent the client U m The local model weights updated in round t and round t-1 training, D t,m Indicates the update of w t,m The training data set is provided by the client U m The local dataset D m Randomly generated, η t is the learning rate of the tth iteration; is the loss function F(w t-1,m ,D t,m )About w t-1,m gradient.

3. The efficient communication federated learning model training method with residual adaptive compression as claimed in claim 2, characterized in that: The client obtains the local model residual by using the local model memory set and updating the model weights, including the following steps: The client inputs the K local historical model weights in the local model memory set into the model prediction module to obtain the prediction model weight. The prediction module calculates the model weight of the output prediction according to the following formula: in, Indicates that client U is m The prediction model weights, Indicates that the client U is saved in the local model memory during the tkth round of training m The historical model weight, λ k is the weight coefficient, and The client uses the prediction model weight and the update model weight to calculate the local model residual. The local model residual is calculated using the following formula: in, represents the prediction model weight; w t,m Indicates updating model weights; r t,m Indicates client U m The local model residual obtained in the tth round of training.

4. The efficient communication federated learning model training method with residual adaptive compression as claimed in claim 3, characterized in that: The client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates historical model weights and stores them in the local model memory set, including the following steps: Sparse compression representation converts the original vector into a sparse vector with a small number of non-zero elements by removing some unimportant elements in the original vector; The sparse compression method reduces the bandwidth occupied by communication by sending elements in the local model residual that are larger than a certain threshold to the server. Different sparsification thresholds will affect the communication overhead and model accuracy. The sparsification threshold needs to be set dynamically, and the optimal sparsification threshold is selected based on the model accuracy and iteration rounds. To achieve adaptive selection of the sparseness threshold, the sparse compression ratio needs to be set dynamically: Among them, ρ t,m and acc t,m Respectively represent the client U m The sparse compression rate and model accuracy in the tth round of training, T represents the total number of training rounds, α and β are weight coefficients; The sparsification threshold is determined based on the sparsity compression ratio and the local model residual: Among them, θ t,m Indicates client U m The sparsification threshold at the tth round of training, J t，m =ρ t，m d represents the number of remaining elements after model residual compression, and d represents the local model residual r t,m The number of elements of Indicates that |r t,m |After sorting in descending order, take the order in the Jth t,m The parameter value corresponding to each position, |·| means taking the absolute value; The elements in the local model residual are calculated according to the following formula to obtain the compressed model residual: in, and r t,m,i Represent the compression model residuals and the local model residual r t,m The i-th element of m Compress the model residual Sent to the server via uplink; The client uses the compressed model residual and the predicted model weight to obtain the historical model weight: in, Indicates that the client U is saved in the local model memory during the tth round of training m The historical model weights.

5. The efficient communication federated learning model training method with residual adaptive compression as claimed in claim 4, characterized in that: The server selects a reliable client to perform local model weight prediction, saves it in the global model memory set, and completes the federated learning model aggregation including the following steps: The server evaluates the client's communication reliability based on the packet error rate between the client and the server; The server inputs the received local model residual and K global historical model weights into the model prediction module to obtain the predicted local model weights; The server aggregates the prediction model weights of each client using a federated learning model; Due to the unreliability of the wireless uplink, the client will generate errors when transmitting the model residual. The server needs to select reliable clients to participate in the model aggregation of federated learning. The server evaluates the client's communication reliability based on the packet error rate between the client and the server. The packet error rate between the client and the server is calculated using the following formula: Among them, q m It is client U m Packet error rate between the server and the server, h m It is client U m Channel gain to the server, Yes m The mathematical expectation of B UL is the client uplink transmission bandwidth, N0 is the noise power spectral density, P m It is client U m The transmit power of the client and the server is τ, and τ is the inverse of the coding gain for data transmission between the client and the server.

6. The efficient communication federated learning model training method with residual adaptive compression as claimed in claim 5, characterized in that: The server inputs the received local model residual and K global historical model weights into the model prediction module to obtain the predicted local model weight, which is expressed as: in, Represents the client U generated by the server prediction m The prediction model weight at iteration t, Indicates that the client U is saved in the global model memory set at the tkth iteration m The historical model weight, λ k is the weight coefficient, and 7. The efficient communication federated learning model training method with residual adaptive compression as claimed in claim 6, characterized in that: The server aggregates the prediction model weights of each client using a federated learning model: Among them, w t,F represents the model weight obtained by server aggregation in the tth round of training, N m Indicates that the server gets the client U m The size of the local dataset; Indicates client U m Whether to participate in the tth round of federated learning training, Indicates client U m Participate in this round of federated learning training, otherwise Indicates client U m Whether there is an error when transmitting the model residual to the server; Indicates client U m No error occurred when transmitting the model residual to the server. The probability of no error is 1-q m ,otherwise 8. A system for training a high-efficiency communication federated learning model with residual adaptive compression, based on the method for training a high-efficiency communication federated learning model with residual adaptive compression according to any one of claims 1 to 7, characterized in that: It also includes,an acquisition module, where the client obtains the global model of federated learning and uses the local dataset to update the model; Update module, the client uses the local model memory set and updates the model weights to obtain the local model residual; In the transmission module, the client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates historical model weights and saves them in the local model memory set; In the prediction module, the server selects a reliable client to make local model weight predictions, saves them in the global model memory set, and completes the federated learning model aggregation; In the convergence module, the server broadcasts the aggregated updated global model to each client through the downlink, and repeats the above steps multiple times until each client model converges to the optimal global model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the efficient communication federated learning model training method for residual adaptive compression described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the efficient communication federated learning model training method for residual adaptive compression described in any one of claims 1 to 7 are implemented.

Images