Method and system for efficient communication federated learning model training with residual adaptive compression

Abstract

The present application relates to the technical field of distributed learning and wireless communication, and particularly relates to a residual adaptive compression efficient communication federated learning model training method and system. A client obtains a global model and updates; a local model residual is obtained by using a local model memory set and updating a model weight; the residual is adaptively sparsely compressed and transmitted to a server; the server selects a reliable client to perform model weight prediction and aggregation; the updated global model is broadcast to the client, and the above steps are repeated until convergence. The present application significantly reduces communication overhead and improves model convergence speed and final performance through adaptive compression and reliable client selection. It is particularly suitable for resource-constrained edge computing scenarios and effectively solves key problems such as communication efficiency, model performance and privacy protection in federated learning, thereby providing new possibilities for the wide application of federated learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed learning and wireless communication, and particularly relates to a residual adaptive compression efficient communication federated learning model training method and system. BACKGROUND

[0002] Federated learning is an efficient distributed machine learning framework that trains various models using distributed data on client devices connected by a network without uploading raw data. Federated learning can protect user privacy while using distributed network data for federated model training, achieving or approaching the performance of centralized training. During model training, user clients do not need to transmit raw data, but transmit model weights. As machine learning model sizes continue to expand, existing mobile communication infrastructure is not always able to meet the requirements of federated learning in terms of bandwidth and latency, which limits the widespread deployment of federated learning. Communication efficiency is a key factor in deploying federated learning, especially in wireless networks.

[0003] Existing federated learning compression methods mostly directly compress and quantize the current model weights or gradients or discard part of the parameters, and send the compressed model weights or gradients to the server for aggregation. These compression methods use a uniform compression rate for all clients and do not consider communication heterogeneity in actual applications. Therefore, under the condition that the communication resources of the clients are limited, the transmission parameters of the clients need to be adaptively compressed to reduce the communication overhead of the uplink and improve the efficiency of global model training. SUMMARY

[0004] In view of the problems in the prior art, the present application is proposed.

[0005] Therefore, the problem to be solved by the present application is that existing federated learning compression methods mostly directly compress and quantize the current model weights or gradients or discard part of the parameters, and send the compressed model weights or gradients to the server for aggregation. These compression methods use a uniform compression rate for all clients and do not consider communication heterogeneity in actual applications. Therefore, under the condition that the communication resources of the clients are limited, the transmission parameters of the clients need to be adaptively compressed to reduce the communication overhead of the uplink and improve the efficiency of global model training.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the embodiments of the present application provide a residual adaptive compression efficient communication federated learning model training method, which comprises that a client obtains a federated learning global model and updates the model using a local data set.

[0008] The client obtains the local model residual by using the local model memory set and updating the model weight;

[0009] The client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates historical model weights which are saved in the local model memory set;

[0010] The server selects reliable clients to perform local model weight prediction, saves them in the global model memory set, and completes federated learning model aggregation;

[0011] The server broadcasts the aggregated updated global model to each client through the downlink, and repeats the above steps multiple times until each client model converges to the optimal global model.

[0012] As a preferred scheme of the efficient communication federated learning model training method with adaptive residual compression, wherein the client obtains the federated learning global model and updates the model using the local data set, including the following steps:

[0013] Each client updates the local model according to the local data set, and at any client, uses the stochastic gradient descent algorithm to reduce the loss function of the local model, and the local model is updated using the following formula:

[0014]

[0015] Where, w t,m and w t-1,m represent the local model weight of the client U m updated in the tth round and the (t-1) th round, D t,m represents the training data set used to update w t,m , which is randomly generated by the local data set D m of the client U m , and η t is the learning rate of the tth iteration; is the gradient of the loss function F(w t-1,m , D t,m ) with respect to w t-1,m .

[0016] As a preferred scheme of the efficient communication federated learning model training method with adaptive residual compression, wherein the client obtains the local model residual by using the local model memory set and updating the model weight, including the following steps:

[0017] The client inputs the K local historical model weights in the local model memory set into the model prediction module to obtain the predicted model weight, and the prediction module calculates the predicted model weight according to the following formula:

[0018]

[0019] wherein, denotes the prediction model weight of the client U m at the t-th round of training, denotes the historical model weight of the client U m at the t-k-th round of training saved in the local model memory set, and λ k is a weight coefficient, and

[0020] The client calculates the local model residual using the prediction model weight and the update model weight. The local model residual is calculated using the following formula:

[0021]

[0022] wherein, denotes the prediction model weight; w t,m denotes the update model weight; r t,m denotes the local model residual obtained by the client U m at the t-th round of training.

[0023] As a preferred scheme of the efficient communication federated learning model training method with residual adaptive compression according to the application, wherein: the client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates the historical model weight saved in the local model memory set, comprising the following steps:

[0024] Sparse compression means converting an original vector into a sparse vector with a small number of non-zero elements by removing some unimportant elements in the original vector;

[0025] The sparse compression method sends elements greater than a certain threshold in the local model residual to the server to reduce the bandwidth occupied by communication. Different sparsification thresholds will affect the communication overhead and model accuracy. The sparsification threshold needs to be dynamically set, and the best sparsification threshold is selected according to the model accuracy and the number of iterations. To achieve adaptive selection of the sparsification threshold, the sparse compression rate needs to be dynamically set:

[0026]

[0027] wherein, ρ t,m and acc t,m respectively denote the sparse compression rate and the model accuracy of the client U m at the t-th round of training, T denotes the total number of training rounds, and α and β are weight coefficients;

[0028] The sparsification threshold is determined according to the sparse compression rate and the local model residual:

[0029]

[0030] wherein, θ t,m represents the client U m the sparsification threshold at the t-th round of training, J t,m = ρ t,m · d represents the number of remaining elements after the model residual compression, d represents the number of elements of the local model residual r t,m , represents the parameter value corresponding to the J t,m -th position after the |r t,m | is sorted in descending order, |·| represents taking the absolute value;

[0031] The elements in the local model residual are calculated according to the following formula to obtain the compressed model residual:

[0032]

[0033] wherein, and r t,m, i represent the i-th element of the compressed model residual and the local model residual r t,m , respectively; the client U m sends the compressed model residual to the server through the uplink;

[0034] The client obtains the historical model weight using the compressed model residual and the predicted model weight:

[0035]

[0036] wherein, represents the historical model weight of the client U m at the t-th round of training saved in the local model memory set.

[0037] As a preferred scheme of the efficient communication federated learning model training method with residual adaptive compression according to the present application, wherein: the server selects a reliable client to perform local model weight prediction, saves it in the global model memory set, and completes federated learning model aggregation, including the following steps:

[0038] The server evaluates the communication reliability of the client according to the packet error rate between the client and the server;

[0039] The server inputs the received local model residual and K global historical model weights into the model prediction module to obtain the predicted local model weight;

[0040] The server aggregates the predicted model weights of each client through federated learning model aggregation.

[0041] Due to the unreliability of the wireless uplink, the client will generate errors when transmitting the model residual, and the server needs to select reliable clients to participate in the model aggregation of federated learning,

[0042] The server evaluates the communication reliability of the client according to the packet error rate between the client and the server, and the packet error rate between the client and the server is calculated by the following formula:

[0043]

[0044] Wherein, q m is the packet error rate between the client U m and the server, h m is the channel gain between the client U m and the server, is the mathematical expectation of h m , B UL is the uplink transmission bandwidth of the client, N0 is the noise power spectral density, P m is the transmission power of the client U m , and τ is the inverse of the coding gain of the data transmission between the client and the server.

[0045] As a preferred scheme of the efficient communication federated learning model training method of residual adaptive compression according to the application, wherein: the server inputs the received local model residual and K global historical model weights into the model prediction module to obtain the predicted local model weight, which is expressed by the formula:

[0046]

[0047] Wherein, indicates the predicted model weight of the client U m at the tth iteration predicted by the server, indicates the historical model weight of the client U m at the t-kth iteration saved in the global model memory, λ k is a weight coefficient, and

[0048] As a preferred scheme of the efficient communication federated learning model training method of residual adaptive compression according to the application, wherein: the server aggregates the predicted model weights of each client through federated learning model aggregation:

[0049]

[0050] Wherein, w t,F denotes the model weight aggregated by the server in the t-th round of training, N m denotes the local data set size of the client U m .

[0051] denotes whether the client U m participates in the t-th round of federated learning training, denotes that the client U m participates in the federated learning training of the current round, otherwise denotes that the client U m transmits the model residual to the server, and whether an error occurs;

[0052] denotes that the client U m transmits the model residual to the server without error, and the probability of no error is 1-q m , otherwise

[0053] In a second aspect, the embodiments of the present application provide an efficient communication federated learning model training system with residual adaptive compression, which comprises an acquisition module, a client acquires a global federated learning model, and uses a local data set to update the model;

[0054] An update module, the client obtains a local model residual by using a local model memory set and updating the model weight;

[0055] A transmission module, the client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates a historical model weight saved in the local model memory set;

[0056] A prediction module, the server selects reliable clients to predict the local model weight, saves it in the global model memory set, and completes federated learning model aggregation;

[0057] A convergence module, the server broadcasts the updated global model to each client through the downlink, and repeats the above steps multiple times until the model of each client converges to the optimal global model.

[0058] In a third aspect, the embodiments of the present application provide a computer device, comprising a memory and a processor, and the memory stores a computer program, wherein: the computer program instructions are executed by the processor to realize the steps of the residual adaptive compression efficient communication federated learning model training method according to the first aspect of the present application.

[0059] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium having stored thereon a computer program, wherein the computer program instructs a processor to implement the steps of the residual adaptive compression high-efficiency communication federated learning model training method according to the first aspect of the present application.

[0060] The present application has the beneficial effect that the present application can significantly reduce the uplink communication overhead of federated learning and improve the training efficiency of federated learning by adaptively compressing the model residuals of the clients during model training.

[0061] The present application can effectively improve the convergence speed of the federated learning model and reduce the adverse effects of unreliable users on model training by selecting reliable clients during federated learning model aggregation. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0063] Figure 1 Flowchart of the residual adaptive compression high-efficiency communication federated learning model training method;

[0064] Figure 2 Computer device diagram of the residual adaptive compression high-efficiency communication federated learning model training method;

[0065] Figure 3 Model training diagram of the residual adaptive compression high-efficiency communication federated learning model training method. DETAILED DESCRIPTION

[0066] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings.

[0067] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from the description, and those skilled in the art can make similar generalizations without departing from the scope of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0068] Second, the "one embodiment" or "an embodiment" referred to herein means a specific feature, structure, characteristic, or combination of features and characteristics described herein that can be included in at least one implementation of the application. The various appearances of "in one embodiment" or "in an embodiment" in the specification do not all refer to the same embodiment, although they can.

[0069] Embodiment 1

[0070] With reference to Figures 1-2 For the first embodiment of the application, the embodiment provides a high-efficiency communication federated learning model training method with residual adaptive compression, comprising,

[0071] S100: The client obtains a global federated learning model and updates the model using a local data set;

[0072] S101: The client obtains a global federated learning model and updates the model using a local data set, comprising the following steps:

[0073] Each client updates the local model according to the local data set. At any one client, the stochastic gradient descent algorithm is used to reduce the loss function of the local model. The local model is updated using the following formula:

[0074]

[0075] where w t,m and w t-1,m represent the local model weights of the client U m updated in the tth round and the (t-1)th round, D t,m represents the training data set used to update w t,m , which is randomly generated by the local data set D m of the client U m , and η t is the learning rate of the tth iteration; is the gradient of the loss function F(w t-1,m , D t,m ) with respect to w t-1,m .

[0076] S200: The client obtains the local model residual by using the local model memory set and updating the model weight;

[0077] S201: The client obtains the local model residual by using the local model memory set and updating the model weight, comprising the following steps:

[0078] The client inputs the K local historical model weights in the local model memory set to the model prediction module to obtain the predicted model weight. The prediction module calculates the predicted model weight according to the following formula:

[0079]

[0080] wherein, denotes the prediction model weight of the client U m at the t-th round of training, denotes the historical model weight of the client U m at the t-k-th round of training saved in the local model memory set, λ k is a weight coefficient, and

[0081] The client calculates the local model residual using the prediction model weight and the update model weight. The local model residual is calculated using the following formula:

[0082]

[0083] wherein, denotes the prediction model weight; w t,m denotes the update model weight; r t,m denotes the local model residual obtained by the client U m at the t-th round of training.

[0084] S300: The client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates the historical model weight saved in the local model memory set;

[0085] S301: The client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates the historical model weight saved in the local model memory set, including the following steps:

[0086] Sparse compression represents the conversion of an original vector into a sparse vector with a small number of non-zero elements by removing some unimportant elements in the original vector;

[0087] The sparse compression method sends the elements in the local model residual that are greater than a certain threshold to the server to reduce the bandwidth occupied by communication. Different sparsification thresholds will affect the communication overhead and model accuracy. The sparsification threshold needs to be dynamically set, and the best sparsification threshold is selected according to the model accuracy and the number of iterations. To achieve adaptive selection of the sparsification threshold, the sparse compression rate needs to be dynamically set:

[0088]

[0089] wherein, ρ t,m and acc t,m respectively denote the sparse compression rate and the model accuracy of the client U m at the t-th round of training, T denotes the total number of training rounds, and α and β are weight coefficients;

[0090] The sparsification threshold is determined according to the sparse compression rate and the local model residual:

[0091]

[0092] where θ t,m represents the client U m The sparsification threshold at the t-th round of training, J t,m = ρ t,m · d represents the number of remaining elements after the model residual compression, d represents the number of elements of the local model residual r t,m , represents the parameter value corresponding to the J t,m th position after the |r t,m | is sorted in descending order, and |·| represents taking the absolute value;

[0093] The elements in the local model residual are calculated according to the following formula to obtain the compressed model residual:

[0094]

[0095] wherein, and r t,m,i represent the i-th element of the compressed model residual and the local model residual r t,m , respectively, and U m The compressed model residual is sent to the server through the uplink;

[0096] The client obtains the historical model weight using the compressed model residual and the predicted model weight:

[0097]

[0098] wherein, represents the historical model weight of the client U m at the t-th round of training saved in the local model memory set.

[0099] S400: The server selects reliable clients for local model weight prediction, saves them in the global model memory set, and completes federated learning model aggregation;

[0100] S401: The server selects reliable clients for local model weight prediction, saves them in the global model memory set, and completes federated learning model aggregation, including the following steps:

[0101] The server evaluates the communication reliability of the client according to the packet error rate between the client and the server;

[0102] The server inputs the received local model residual and K global historical model weights into a model prediction module to obtain predicted local model weights;

[0103] The server performs federated learning model aggregation on the predicted model weights of each client;

[0104] Due to the unreliability of the wireless uplink, the client will generate errors when transmitting the model residual, and the server needs to select reliable clients to participate in the model aggregation of federated learning,

[0105] The server evaluates the communication reliability of the client according to the packet error rate between the client and the server, and the packet error rate between the client and the server is calculated by the following formula:

[0106]

[0107] Wherein, q m is the packet error rate between the client U m and the server, h m is the channel gain between the client U m and the server, is the mathematical expectation of h m , B UL is the uplink transmission bandwidth of the client, N0 is the noise power spectral density, P m is the transmission power of the client U m , and τ is the inverse of the coding gain of the data transmission between the client and the server.

[0108] S402: The server inputs the received local model residual and K global historical model weights into a model prediction module to obtain predicted local model weights, which is expressed by the formula:

[0109]

[0110] Wherein, represents the predicted model weight of the client U m predicted by the server at the tth iteration, represents the historical model weight of the client U m saved in the global model memory at the t-kth iteration, λ k is a weight coefficient, and

[0111] S403: The server performs federated learning model aggregation on the predicted model weights of each client:

[0112]

[0113] Wherein, w t,Fdenotes the model weight aggregated by the server in the t-th round of training, N m denotes the local dataset size of the client U m .

[0114] denotes whether the client U m participates in the t-th round of federated learning training, denotes the client U m participates in the federated learning training of the current round, otherwise denotes whether the client U m has an error when transmitting the model residual to the server;

[0115] denotes that the client U m has no error when transmitting the model residual to the server, and the probability of no error is 1-q m , otherwise

[0116] S500: The server broadcasts the updated global model to each client through the downlink, and repeats the above steps multiple times until each client model converges to the optimal global model.

[0117] Model residual: refers to the difference between the local model and the prediction model, reflecting the information of model update.

[0118] Adaptive sparse compression: a compression method that dynamically adjusts the sparsification threshold according to the model accuracy and the number of iterations.

[0119] Adaptive sparse compression step: by dynamically adjusting the sparsification threshold, the communication overhead is minimized while ensuring the performance of the model.

[0120] Reliable client selection step: by evaluating the communication reliability of the client, the negative impact of the client on the model training is avoided.

[0121] Further, the embodiment also provides a residual adaptive compression efficient communication federated learning model training system, comprising,

[0122] An acquisition module, the client acquires a federated learning global model and updates the model using a local dataset;

[0123] An update module, the client obtains a local model residual using a local model memory set and an updated model weight;

[0124] A transmission module, the client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates a historical model weight saved in the local model memory set;

[0125] A prediction module, the server selects a reliable client to perform local model weight prediction, saves it in a global model memory set, and completes federated learning model aggregation;

[0126] A convergence module, the server broadcasts the aggregated updated global model to each client through a downlink, and repeats the above steps multiple times until each client model converges to an optimal global model.

[0127] The embodiment also provides a computer device suitable for the efficient communication federated learning model training method with residual adaptive compression, which comprises a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the efficient communication federated learning model training method with residual adaptive compression proposed in the above embodiment.

[0128] The computer device can be a terminal, which comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device. In addition, the input device can be an external keyboard, touchpad or mouse, etc.

[0129] The embodiment also provides a storage medium having a computer program stored thereon, which is executed by a processor to realize the efficient communication federated learning model training method with residual adaptive compression proposed in the above embodiment.

[0130] In summary, through the steps of obtaining a federated learning global model by a client and updating the model using a local data set, model training using distributed data is realized under the premise of protecting data privacy. This method avoids direct transmission of original data, effectively reduces the risk of data leakage, and also fully utilizes a large amount of data resources scattered in various clients, finally achieving the beneficial effects of protecting privacy while improving model training effect.

[0131] The step of obtaining the local model residual by the client using the local model memory set and updating the model weight realizes the extraction and compression of the model update information. This residual-based method can effectively capture the key information of model changes, greatly reduce the amount of data that needs to be transmitted, thereby significantly reducing the communication overhead, and ultimately achieving the beneficial effect of improving the efficiency of federated learning under limited bandwidth conditions.

[0132] The step of adaptively sparsely compressing the local model residual by the client realizes the dynamic balance between communication efficiency and model performance. This adaptive method can automatically adjust the compression rate according to the current model training state and communication conditions, while ensuring model convergence, and maximally reducing data transmission, ultimately achieving the beneficial effect of optimizing federated learning performance in resource-limited scenarios.

[0133] The step of selecting reliable clients by the server to predict the local model weight realizes the adaptation to unstable communication environment. This communication reliability-based screening mechanism can effectively reduce the negative impact of unreliable clients on the global model, improving the robustness of federated learning in complex network environments, and ultimately achieving the beneficial effect of improving model training quality and convergence speed.

[0134] The step of broadcasting the aggregated global model by the server to each client through the downlink and performing multiple rounds of iteration realizes the collaborative optimization of the distributed learning process. This iterative learning method can continuously integrate the knowledge of each client, gradually improving the performance of the global model, and ultimately achieving the beneficial effect of realizing high-quality model training under the premise of protecting privacy.

[0135] In summary, the present scheme ingeniously solves multiple key problems in federated learning, such as communication efficiency, model performance, and privacy protection. In particular, the adaptive sparse compression and reliable client selection steps bring unexpected technical effects: they not only greatly reduce communication overhead, but also improve model convergence speed and final performance. This method is particularly valuable in resource-limited edge computing scenarios, providing new possibilities for the widespread application of federated learning.

[0136] Embodiment 2

[0137] Reference Figure 2 - Figure 3 As the second embodiment of the present application, this embodiment provides an efficient communication federated learning model training method with residual adaptive compression. In order to verify the beneficial effects of the present application, economic benefit calculation and simulation experiments are used for scientific demonstration.

[0138] To verify the effectiveness of the residual adaptive compression-based efficient communication federated learning model training method proposed in the application, experiments were conducted in an Internet of Things system composed of 10,000 intelligent factory devices. Each device acts as a client, collects its own operation data and participates in federated learning, aiming to build an accurate device fault prediction model. The experimental environment uses Python3.8 and PyTorch1.9.0 framework, and runs on a server equipped with Intel Xeon E5-2680v4 CPU and 128GB RAM.

[0139] First, a simple three-layer fully connected neural network is initialized as the global model, and the input features include device temperature, vibration, current, etc. 10 parameters, and the output is the probability of failure within the next 24 hours. Each client's local data set contains 1000 historical records. The experiment compares three methods: (1) traditional federated average (FedAvg), (2) fixed threshold sparsification (FixedSparsification, FS), and (3) the residual adaptive compression (ResidualAdaptiveCompression, RAC) method of the application.

[0140] In the RAC method, the client first updates the local model using the stochastic gradient descent algorithm. Then, the local model memory set (which saves the historical weights of the last 5 rounds) is used to predict the current model weights, and the residual with the actual updated weights is calculated. The residual is adaptively sparsely compressed, and the selection formula of the sparse compression rate τ is:

[0141] τ = 0.1 * (1-acc) + 0.9 * (1-t / T)

[0142] Here acc is the current model accuracy, t is the current iteration round, and T is the total number of rounds (set to 100).

[0143] On the server side, first, the reliable clients are selected according to the packet error rate (PER), and the PER threshold is set to 0.1. Then, the compressed residuals of the selected clients are decompressed, and the complete local model weights are obtained by combining the predicted weights, and finally the updated global model is obtained by weighted averaging.

[0144] The experiment lasted for 100 rounds of training, and 1000 clients were randomly selected to participate in each round. During the entire process, the model accuracy, communication overhead and training time were recorded. The results are shown in the following table:

[0145]

[0146] Through the analysis of the data in the above table, the following conclusions can be drawn:

[0147] In terms of accuracy, the RAC method proposed in this invention achieved an accuracy of 93.7%, which is 1.2 percentage points higher than the traditional FedAvg method and 1.9 percentage points higher than the fixed threshold sparsity (FS) method. This shows that the RAC method not only did not suffer from model performance loss due to compression, but also improved the model's generalization ability through an adaptive mechanism.

[0148] Communication efficiency: The average communication volume per round of the RAC method is only 38MB, which is 85.2% less than the FedAvg method and 40.6% less than the FS method. This significant reduction in communication volume directly demonstrates the advantages of this invention in improving communication efficiency, making it particularly suitable for application in bandwidth-constrained IoT environments.

[0149] Training time: Thanks to a significant reduction in communication, the RAC method shortens the total training time to 12 hours, saving 75% of the time compared to the FedAvg method and 33.3% of the time compared to the FS method. This efficiency improvement allows the model to be deployed to the production environment more quickly, improving the system's responsiveness.

[0150] Convergence speed: The RAC method achieved convergence in only 62 rounds, 23 rounds earlier than the FedAvg method and 16 rounds earlier than the FS method. This shows that the adaptive compression strategy of this invention not only did not affect the convergence of the model, but also accelerated the convergence process through effective information extraction.

[0151] Further analysis revealed that the advantages of the RAC method mainly stem from two innovative aspects:

[0152] First, the residual-based compression strategy captures key information from model updates. Experiments show that as training progresses and the magnitude of model updates gradually decreases, the RAC method automatically adjusts the compression ratio, preserving more information in the early stages of training and applying more aggressive compression in later stages. This dynamic adjustment mechanism ensures both the model's learning effectiveness and maximizes communication efficiency.

[0153] Secondly, the client selection mechanism based on communication reliability significantly improves the robustness of the model. During the experiment, 10% of clients were simulated to have unstable network connections. The RAC method effectively avoids the interference of these unstable factors on the global model by selecting reliable clients, thus maintaining high performance even in complex real-world environments.

[0154] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A residual adaptive compression efficient communication federated learning model training method, characterized in that: The application relates to a federated learning method and device. The client obtains a federated learning global model and updates the model using a local data set; The client obtains a local model residual by using a local model memory set and an updated model weight; The client performs adaptive sparse compression on the local model residual, transmits the local model residual to a server through an uplink, and generates a historical model weight which is saved in the local model memory set; The client performs adaptive sparse compression on the local model residual, transmits the local model residual to a server through an uplink, and generates a historical model weight which is saved in the local model memory set, and the method comprises the following steps: Sparse compression is a method of converting an original vector into a sparse vector with a small number of non-zero elements by removing some unimportant elements in the original vector; The sparse compression method transmits elements greater than a certain threshold in the local model residual to the server to reduce the bandwidth occupied by communication. Different sparse thresholds affect the communication overhead and model accuracy. Therefore, the sparse threshold needs to be dynamically set, and the optimal sparse threshold is selected according to the model accuracy and the iteration round. To achieve adaptive selection of the sparse threshold, the sparse compression rate needs to be dynamically set: ； wherein, and respectively represent the client The sparse compression rate and model accuracy at the i-th training round, T represents the total number of training rounds, and is a weight coefficient; The sparse threshold is determined according to the sparse compression rate and the local model residual: ； wherein, represents the client In the first Sparse threshold in the training of the round, represents the number of remaining elements after the compression of the model residual, represents the number of elements of the local model residual , represents the parameter value corresponding to the position of the ranking in descending order , represents the absolute value; The elements in the local model residual are calculated to obtain a compressed model residual according to the following formula: ； wherein, and denote the first element of the compressed model residual and the local model residual respectively; the client sends the compressed model residual to the server over the uplink; The client obtains a historical model weight by using the compressed model residual and a predicted model weight: ； wherein, represents the historical model weights of the client at the time of the round of training; and is saved in the local model memory set. The server selects reliable clients to predict the local model weight, saves the local model weight in a global model memory set, and completes federated learning model aggregation; The server selects reliable clients to predict the local model weight, saves the local model weight in a global model memory set, and completes federated learning model aggregation, and the method comprises the following steps: The server evaluates the communication reliability of the clients according to the packet error rate between the clients and the server; The server inputs the received local model residual and a global historical model weight into a model prediction module to obtain a predicted local model weight; The server aggregates the predicted model weights of the clients according to the federated learning model; Due to the unreliability of the wireless uplink, the clients will generate errors when transmitting the model residual, and the server needs to select reliable clients to participate in the model aggregation of the federated learning, The server evaluates the communication reliability of the clients according to the packet error rate between the clients and the server, and the packet error rate between the clients and the server is calculated according to the following formula: ； wherein is the packet error rate between the client and the server, is the channel gain between the client and the server, is the mathematical expectation of , is the client uplink transmission bandwidth, is the noise power spectral density, is the transmission power of the client , is the inverse of the coding gain of the data transmission between the client and the server; The server broadcasts the aggregated updated global model to each client through a downlink, and repeats the above steps multiple times until the model of each client converges to an optimal global model.

2. The method of claim 1, wherein the method further comprises: The client obtains a federated learning global model and updates the model using a local data set, and the method comprises the following steps: Each client updates the local model according to the local data set. In any one client, the loss function of the local model is reduced by using a stochastic gradient descent algorithm. The local model is updated according to the following formula: ； wherein, with denote the client In the first round and the round of training updates the local model weights, denote the training data set used to update by the client local data set is randomly generated, is the learning rate for the iteration; is the gradient of the loss function with respect to . 3.The method of claim 2, wherein: The client obtains a local model residual by using a local model memory set and an updated model weight, and the method comprises the following steps: The client inputs the local model weight in memory set into the model prediction module to obtain the predicted model weight, and the prediction module calculates the output predicted model weight according to the following formula: ； wherein, represents the prediction model weight of the client at the i-th training round, represents the historical model weight of the client saved in the local model memory set at the i-th training round, represents the prediction model weight of the client at the i-th training round, is a weight coefficient, and ; The client calculates the local model residual by using a predicted model weight and an updated model weight. The local model residual is calculated according to the following formula: ； wherein, denotes the prediction model weight; denotes the updated model weight; denotes the client In the first local model residual obtained in the first round of training.

4. The method of claim 3, wherein the method further comprises: The server inputs the received local model residual and a global historical model weight into a model prediction module to obtain a predicted local model weight, which is expressed by a formula as follows: ； wherein, represents a client predicted by the server at the iteration, represents the historical model weight of the client at the iteration saved in the global model memory, is a weight coefficient, and .

5. The method of claim 4, wherein the method further comprises: The server aggregates the predicted model weights of the clients according to the federated learning model. ； wherein, represents the first training in the server aggregation of the model weights, represents the server obtained client size of the local data set; represents the client whether to participate in the federated learning training of the round, represents the client participates in the federated learning training of the current round, otherwise , represents the client whether an error occurs when the client transmits the model residual to the server; representing the client transmitting the model residual to the server without error, the probability of transmitting without error being , otherwise .

6. A residual adaptive compression efficient communication federated learning model training system based on the residual adaptive compression efficient communication federated learning model training method of any one of claims 1-5. Also include, the acquisition module, the client acquires the global model of federated learning, and uses the local data set to update the model; The update module, the client obtains the local model residual by using the local model memory set and updating the model weight; The transmission module, the client performs adaptive sparse compression on the local model residual, transmits it to the server through the uplink, and generates the historical model weight and saves it in the local model memory set; The prediction module, the server selects reliable clients to predict the local model weight, saves it in the global model memory set, and completes the federated learning model aggregation; The convergence module, the server broadcasts the updated global model to each client through the downlink, and repeats the above steps multiple times until each client model converges to the optimal global model. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The processor executes the computer program to realize the steps of the residual adaptive compression efficient communication federated learning model training method of any one of claims 1-5.

8. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to realize the steps of the residual adaptive compression efficient communication federated learning model training method of any one of claims 1-5.

Images