Federated learning spatiotemporal compression joint optimization method, device and system
By jointly optimizing the globally optimal communication frequency and gradient compression rate in federated learning, the problem of high system complexity in existing methods is solved, achieving high training efficiency and adaptive response, adapting to dynamic changes in device computing power and network bandwidth.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING MIANBI INTELLIGENT TECH CO LTD
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-09
AI Technical Summary
Existing federated learning methods optimize only a single compression dimension of time or space during gradient transmission, resulting in high system complexity and difficulty in adapting to scenarios with heterogeneous device computing power and dynamic fluctuations in network bandwidth.
The system obtains initial configuration information and environmental data of worker nodes from the server, determines the globally optimal communication frequency and gradient compression rate, and broadcasts them to each worker node to achieve spatiotemporal compression joint optimization. It constructs a univariate convex function to simplify the optimization process and builds a mapping relationship between communication frequency and gradient compression rate by combining a preset convergence factor.
It significantly improves training efficiency, reduces computational complexity, and can capture network fluctuations and changes in node computing power in real time, enabling adaptive response and improving training efficiency.
Smart Images
Figure CN122175039A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of federated learning technology, and more specifically, to a spatiotemporal compression joint optimization method, apparatus, and system for federated learning. Background Technology
[0002] Federated Learning (FL) trains models locally on the data and sends model updates (such as gradients or model parameters) to a central server for aggregation, rather than directly transmitting the raw data. This approach protects data privacy while enabling data value extraction. With the explosive growth in model size, gradient transmission has become a core communication bottleneck during training. Existing work often employs two independent strategies to mitigate this problem: temporal compression (such as the FedAVG algorithm, which reduces aggregation frequency by increasing the number of local iterations) and spatial gradient compression (such as reducing the amount of data transmitted per transmission by only transmitting a subset of parameters).
[0003] However, existing work often optimizes only a single compression dimension independently, lacking a sufficient understanding of the synergistic mechanism between temporal and spatial compression. Related studies frequently assume that their effects on model convergence are independent, leading to either the introduction of excessive hyperparameters or the design of more complex adaptive algorithms. Furthermore, these adaptive algorithms only independently regulate the compression strategy for a single dimension. This significantly increases system complexity, making it difficult to achieve optimal end-to-end training efficiency in scenarios with heterogeneous device computing power and dynamically fluctuating network bandwidth. Summary of the Invention
[0004] This application provides a spatiotemporal compression joint optimization method, apparatus and system for federated learning, which can solve the problems caused by a single compression dimension.
[0005] The specific technical solution is as follows: In a first aspect, embodiments of this application provide a spatiotemporal compression joint optimization method for federated learning, the method being applied to a server, the method comprising: Obtain the initial configuration information of the server and the environmental data information reported by each worker node. The initial configuration information includes a preset convergence factor and / or a search interval. The search interval is a feasible interval of the communication frequency. The communication frequency is the number of consecutive training steps executed locally by the worker node between two adjacent global aggregations on the server. For each communication frequency within the search interval, based on the environmental data information of the working node and the initialization configuration information, calculate the communication frequency evaluation value corresponding to each working node, and determine the maximum value among all communication frequency evaluation values of all working nodes under the communication frequency. Select the minimum value from the maximum values of all communication frequency evaluation values within the search interval, and determine the communication frequency corresponding to the minimum value as the globally optimal communication frequency. The globally optimal gradient compression is determined based on the globally optimal communication frequency and the preset convergence factor. The globally optimal communication frequency and the globally optimal gradient compression rate are broadcast to each working node.
[0006] In one possible implementation, the environmental data information includes at least one of the following: local computation time, network latency, compression factor, real-time bandwidth, and number of model parameters.
[0007] In one possible implementation, for each communication frequency within the search interval, based on the environmental data information of the working node and the initialization configuration information, a communication frequency evaluation value corresponding to each working node is calculated, including: For each communication frequency within the search interval, a communication frequency evaluation value corresponding to each working node is calculated according to a preset evaluation function. The preset evaluation function includes: ; Among them, the Indicates the communication frequency, the Indicates communication frequency The corresponding communication frequency evaluation value, the Indicates the network latency, the This represents the compression coefficient. Represents the preset convergence factor, the The model parameter quantity is represented by the following. Indicates the real-time bandwidth, the This indicates the time taken for the local computation.
[0008] In one possible implementation, determining the globally optimal gradient compression rate based on the globally optimal communication frequency and the preset convergence factor includes: The globally optimal gradient compression rate is determined based on a preset correlation relationship; The preset association relationships include: ; Among them, the Indicates the communication frequency, the Represents the gradient compression ratio, the This represents the preset convergence factor.
[0009] In one possible implementation, the initialization configuration information further includes a learning rate, and the method further includes: Obtain the compressed cumulative gradient uploaded by each worker node, wherein the compressed cumulative gradient is the cumulative gradient obtained by the worker node through spatiotemporal compression based on the globally optimal communication frequency and the globally optimal gradient compression rate; Based on the compressed cumulative gradient of each working node, the learning rate, and the number of working nodes, the global model parameters obtained in the previous round of global aggregation are updated to obtain the global model parameters obtained in the current round of global aggregation. The global model parameters obtained from this round of global aggregation are broadcast to each worker node.
[0010] Secondly, embodiments of this application provide a spatiotemporal compression joint optimization method for federated learning, the method being applied to worker nodes, the method comprising: Obtain the globally optimal communication frequency and globally optimal gradient compression ratio sent by the server. The globally optimal communication frequency and globally optimal gradient compression ratio are determined by the server based on its own initialization configuration information and the environmental data information reported by each worker node. The initialization configuration information includes a preset convergence factor and / or a search interval. The search interval is a feasible interval for the communication frequency. The communication frequency is the number of consecutive training steps executed locally by the worker node between two adjacent global aggregations on the server. Based on the globally optimal communication frequency, perform gradient iteration calculations for the corresponding number of steps locally to obtain the cumulative gradient; According to the global optimal gradient compression rate, the sum of the cumulative gradient and the historical compression error cache is spatially compressed to obtain the compressed cumulative gradient; The compressed cumulative gradient is uploaded to the server so that the server can update the global model parameters based on the compressed cumulative gradient uploaded by each worker node.
[0011] In one possible implementation, the method further includes: The historical compression error cache is updated based on the accumulated gradient and the compressed accumulated gradient to obtain the updated compression error cache, so as to perform the next round of error compensation based on the updated compression error cache.
[0012] Thirdly, embodiments of this application provide a spatiotemporal compression joint optimization apparatus for federated learning, the apparatus being applied to a server, the apparatus comprising: The acquisition unit is used to acquire the initial configuration information of the server and the environmental data information reported by each worker node. The initial configuration information includes a preset convergence factor and / or a search interval. The search interval is a feasible interval of the communication frequency. The communication frequency is the number of consecutive training steps executed locally by the worker node between two adjacent global aggregations on the server. The calculation unit is used to calculate the communication frequency evaluation value corresponding to each working node for each communication frequency within the search interval, based on the environmental data information of the working node and the initialization configuration information. The first determining unit is used to determine the maximum value among the communication frequency evaluation values of all working nodes under the communication frequency. The selection unit is used to select the minimum value from the maximum values of all communication frequency evaluation values in the search interval, and to determine the communication frequency corresponding to the minimum value as the global optimal communication frequency. The second determining unit is used to determine the global optimal gradient compression rate based on the global optimal communication frequency and the preset convergence factor. The broadcast unit is used to broadcast the globally optimal communication frequency and the globally optimal gradient compression rate to each working node.
[0013] In one possible implementation, the environmental data information includes at least one of the following: local computation time, network latency, compression factor, real-time bandwidth, and number of model parameters.
[0014] In one possible implementation, the computing unit is used to calculate the communication frequency evaluation value corresponding to each working node for each communication frequency within the search interval, based on a preset evaluation function. The preset evaluation function includes: ; Among them, the Indicates the communication frequency, the Indicates communication frequency The corresponding communication frequency evaluation value, the Indicates the network latency, the This represents the compression coefficient. Represents the preset convergence factor, the The model parameter quantity is represented by the following. Indicates the real-time bandwidth, the This indicates the time taken for the local computation.
[0015] In one possible implementation, the second determining unit is configured to determine the globally optimal gradient compression rate based on a preset correlation relationship; The preset association relationships include: ; Among them, the Indicates the communication frequency, the Represents the gradient compression ratio, the This represents the preset convergence factor.
[0016] In one possible implementation, the initialization configuration information also includes a learning rate; The acquisition unit is further configured to acquire the compressed cumulative gradient uploaded by each working node, wherein the compressed cumulative gradient is the cumulative gradient obtained by the working node through spatiotemporal compression based on the global optimal communication frequency and the global optimal gradient compression rate. The device further includes: The update unit is used to update the global model parameters obtained in the previous round of global aggregation based on the compressed cumulative gradient of each working node, the learning rate, and the number of working nodes, so as to obtain the global model parameters obtained in the current round of global aggregation. The broadcast unit is used to broadcast the global model parameters obtained from the current round of global aggregation to each working node.
[0017] Fourthly, embodiments of this application provide a spatiotemporal compression joint optimization apparatus for federated learning, the apparatus being applied to a worker node, the apparatus comprising: The acquisition unit is used to acquire the globally optimal communication frequency and globally optimal gradient compression ratio sent by the server. The globally optimal communication frequency and globally optimal gradient compression ratio are determined by the server based on its own initialization configuration information and the environmental data information reported by each worker node. The initialization configuration information includes a preset convergence factor and / or a search interval. The search interval is a feasible interval for the communication frequency. The communication frequency is the number of consecutive training steps executed locally by the worker node between two adjacent global aggregations on the server. The calculation unit is used to perform gradient iteration calculations locally based on the globally optimal communication frequency, and obtain the cumulative gradient. A compression unit is used to spatially compress the sum of the cumulative gradient and the historical compression error cache according to the global optimal gradient compression rate to obtain the compressed cumulative gradient. An upload unit is used to upload the compressed cumulative gradient to the server so that the server can update the global model parameters based on the compressed cumulative gradient uploaded by each worker node.
[0018] In one possible implementation, the device further includes: An update unit is used to update the historical compression error cache according to the accumulated gradient and the compressed accumulated gradient to obtain an updated compression error cache, so as to perform the next round of error compensation based on the updated compression error cache.
[0019] Fifthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any possible implementation of the first aspect, or implements the method as described in any possible implementation of the second aspect.
[0020] Sixthly, embodiments of this application provide an electronic device, which includes: One or more processors; The processor is coupled to a storage device for storing one or more programs; When one or more programs are executed by one or more processors, the electronic device performs the method as described in any possible implementation of the first aspect, or performs the method as described in any possible implementation of the second aspect.
[0021] In a seventh aspect, embodiments of this application provide a spatiotemporal compression joint optimization system for federated learning, the system comprising a server and multiple worker nodes; The server is configured to execute the method described in any possible implementation of the first aspect; The working node is used to execute the method described in any possible implementation of the second aspect.
[0022] Eighthly, embodiments of this application provide a computer program product containing instructions that, when executed on a computer or processor, cause the computer or processor to perform the method described in any possible implementation of the first aspect, or to perform the method described in any possible implementation of the second aspect.
[0023] Based on the above scheme, the spatiotemporal compression joint optimization method, apparatus, and system for federated learning provided in this application embodiment can first obtain initialization configuration information (including preset convergence factors and / or search intervals) and environmental data information of each working node by the server. Then, it determines the globally optimal communication frequency using this initialization configuration information and environmental data information, and finally determines the globally optimal gradient compression rate by combining it with the preset convergence factor and broadcasting it to each working node. This application embodiment breaks through the limitations of traditional methods that orthogonally optimize communication frequency and gradient compression rate, achieving deep parameter coordination. This not only significantly improves training efficiency but also has low computational complexity, can capture network fluctuations and changes in node computing power in real time, and exhibits extremely fast adaptive response.
[0024] Furthermore, the technical effects achievable by the embodiments of this application also include: 1. A univariate convex function (i.e., a preset evaluation function) is constructed using communication frequency as the independent variable and the corresponding communication frequency evaluation value as the dependent variable. This simplifies complex optimization into a univariate convex function search, further reducing computational complexity and enabling real-time capture of network fluctuations and node computing power changes. The preset evaluation function includes known quantities such as network latency, compression coefficient, preset convergence factor, number of model parameters, real-time bandwidth, and local computation time.
[0025] 2. By combining a preset convergence factor to pre-construct the mapping relationship between communication frequency and gradient compression rate, the synchronous optimization of two variables is realized, transforming the two complex variable optimization problem into the simplest hyperparameter setting. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0027] Figure 1 A flowchart illustrating a spatiotemporal compression joint optimization method for federated learning provided in an embodiment of this application; Figure 2 A flowchart illustrating another spatiotemporal compression joint optimization method for federated learning provided in this application embodiment; Figure 3 A block diagram of a spatiotemporal compression joint optimization device for federated learning provided in this application embodiment; Figure 4 Block diagram of another spatiotemporal compression joint optimization device for federated learning provided in this application embodiment Figure 5 This is a schematic diagram of the structure of a spatiotemporal compression joint optimization system for federated learning, provided as an embodiment of this application. Detailed Implementation
[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.
[0030] In federated learning, gradient transport is a core communication bottleneck during training. Traditional methods only independently optimize one compression dimension, either time or space, lacking a sufficient understanding of their collaborative mechanism. This results in high system complexity and difficulty adapting to scenarios with heterogeneous device computing power and dynamically fluctuating network bandwidth. Therefore, this application provides a spatiotemporal compression joint optimization method for federated learning, which can be applied to servers, such as... Figure 1 As shown, the method includes: S110: Obtain the server's initial configuration information and the environmental data information reported by each worker node. The initial configuration information includes the preset convergence factor and / or search interval.
[0031] In this embodiment, the search interval is a feasible interval of the communication frequency, where the communication frequency is the number of consecutive training steps executed locally by the worker node between two adjacent global aggregations on the server, for example, 500. The preset convergence factor can be determined based on the preset communication frequency and the preset gradient compression rate. For example, the formula for calculating the preset convergence factor may include: ; in, This indicates the preset convergence factor. This indicates the preset communication frequency. This indicates the preset gradient compression ratio. The specific values for the preset communication frequency and the preset gradient compression ratio can be determined based on practical experience, such as... , .
[0032] Environmental data includes at least one of the following: local computation time, network latency, compression factor, real-time bandwidth, and number of model parameters. Local computation time refers to the time required for one round of iterative model training on the local worker node; the compression factor characterizes the overhead of compression computation; and the number of model parameters refers to the number of model parameters.
[0033] In addition, the server's initialization configuration information may also include the model parameters for initializing the global model, the optimization frequency, and the learning rate. The optimization frequency refers to the frequency at which the spatiotemporal compression joint optimization process is initiated during federated learning. The learning rate is used by the server to perform global aggregation operations. Each worker node, upon initialization, configures the model parameters and compression error cache for its local model. .
[0034] S120: For each communication frequency within the search interval, based on the environmental data information and initial configuration information of the working nodes, calculate the communication frequency evaluation value corresponding to each working node, and determine the maximum value among the communication frequency evaluation values of all working nodes under that communication frequency.
[0035] Specifically, the server can calculate the communication frequency evaluation value corresponding to each working node for each communication frequency within the search interval, based on a preset evaluation function. The preset evaluation functions include: ; in, Indicates the communication frequency. Indicates communication frequency The corresponding communication frequency evaluation value, Indicates network latency. Indicates the compression factor. This indicates the preset convergence factor. Indicates the number of model parameters. Indicates real-time bandwidth. This indicates the time taken for local computation.
[0036] S130: Select the minimum value among the maximum values of all communication frequency evaluation values within the search interval, and determine the communication frequency corresponding to the minimum value as the global optimal communication frequency.
[0037] To improve the efficiency of obtaining the minimum value, a binary search method can be used to find the minimum value. That is, the binary search method achieves the effect of "for each communication frequency in the search interval, based on the environmental data information and initial configuration information of the working node, calculate the communication frequency evaluation value corresponding to each working node, determine the maximum value among all communication frequency evaluation values of all working nodes under the communication frequency, and select the minimum value among the maximum values of all communication frequency evaluation values in the search interval".
[0038] S140: Determine the global optimal gradient compression rate based on the global optimal communication frequency and the preset convergence factor.
[0039] Specifically, the server can determine the globally optimal gradient compression rate based on preset association relationships; The preset association relationships include: ; in, Indicates the communication frequency. Indicates the gradient compression rate. This indicates the preset convergence factor.
[0040] After obtaining the globally optimal communication frequency and the preset convergence factor, the globally optimal communication frequency and the preset convergence factor can be substituted into the formula to solve for the unknown gradient compression ratio, and the solved gradient compression ratio can be used as the globally optimal gradient compression ratio.
[0041] S150: Broadcasts the globally optimal communication frequency and the globally optimal gradient compression rate to each working node.
[0042] After obtaining the globally optimal communication frequency and the globally optimal gradient compression rate, the server can broadcast these two parameters to each worker node in the distributed architecture, so that each worker node can perform spatiotemporal compression based on the globally optimal communication frequency and the globally optimal gradient compression rate to obtain the compressed cumulative gradient.
[0043] The spatiotemporal compression joint optimization method for federated learning provided in this application allows the server to first obtain initialization configuration information (including a preset convergence factor and / or search interval) and environmental data information of each worker node. The server then determines the globally optimal communication frequency using this initialization configuration information and environmental data information, and finally determines the globally optimal gradient compression rate by combining it with the preset convergence factor and broadcasting it to each worker node. This application overcomes the limitations of traditional methods that orthogonally optimize communication frequency and gradient compression rate, achieving deep parameter synergy. This not only significantly improves training efficiency but also has low computational complexity, enabling real-time capture of network fluctuations and node computing power changes, and providing extremely fast adaptive response. Furthermore, a univariate convex function (i.e., the preset evaluation function) is constructed using communication frequency as the independent variable and the corresponding communication frequency evaluation value as the dependent variable. This simplifies complex optimization to a univariate convex function search, further reducing computational complexity and enabling real-time capture of network fluctuations and node computing power changes. The preset evaluation function includes known quantities such as network latency, compression coefficient, preset convergence factor, number of model parameters, real-time bandwidth, and local computation time. By combining a preset convergence factor to pre-construct the mapping relationship between communication frequency and gradient compression rate, synchronous optimization of two variables is achieved, transforming the two-variable optimization problem into the simplest hyperparameter setting.
[0044] In one possible implementation, the initialization configuration information also includes the learning rate. After each worker node obtains the globally optimal communication frequency and the globally optimal gradient compression rate, it performs spatiotemporal compression based on the globally optimal communication frequency and the globally optimal gradient compression rate to obtain the compressed cumulative gradient, and reports its compressed cumulative gradient to the server. After receiving the compressed cumulative gradient reported by each worker node, the server can perform a global aggregation operation.
[0045] Specifically, the server obtains the compressed cumulative gradient uploaded by each worker node. The compressed cumulative gradient is the cumulative gradient obtained by the worker node through spatiotemporal compression based on the globally optimal communication frequency and the globally optimal gradient compression rate. Based on the compressed cumulative gradient, learning rate, and number of worker nodes, the server updates the global model parameters obtained in the previous round of global aggregation to obtain the global model parameters obtained in the current round of global aggregation. The server then broadcasts the global model parameters obtained in the current round of global aggregation to each worker node.
[0046] The methods for global aggregation include: ; In this formula, This represents the model parameters after global aggregation in round T+1. This represents the model parameters after global aggregation in round T. Indicates the learning rate. Indicates the number of worker nodes. This represents the compressed cumulative gradient of the i-th working node.
[0047] After completing this round of global aggregation, the server can broadcast the global model parameters obtained from this round of global aggregation to each worker node, so that each worker node can train its local model based on the new global model parameters.
[0048] Based on the above method embodiments, another embodiment of this application provides a spatiotemporal compression joint optimization method for federated learning, which is applied to worker nodes, such as... Figure 2 As shown, the method includes: S210: Obtain the globally optimal communication frequency and globally optimal gradient compression rate sent by the server.
[0049] Among them, the global optimal communication frequency and the global optimal gradient compression rate are determined by the server based on its own initialization configuration information and the environmental data information reported by each worker node. The initialization configuration information includes a preset convergence factor and / or search interval. The search interval is the feasible interval of the communication frequency. The communication frequency is the number of consecutive training steps that the worker node executes locally between two adjacent global aggregations on the server.
[0050] The calculation methods for the globally optimal communication frequency and the globally optimal gradient compression ratio are detailed in the server-side implementation method, and will not be repeated here.
[0051] S220: Based on the globally optimal communication frequency, perform gradient iteration calculations for the corresponding number of steps locally to obtain the cumulative gradient.
[0052] When the globally optimal communication frequency is used When indicating, the i-th worker node executes locally. The gradient is calculated iteratively step by step to obtain the cumulative gradient. Gradient iteration calculation methods include, but are not limited to, SGD (Stochastic Gradient Descent).
[0053] S230: Based on the globally optimal gradient compression rate, spatially compress the sum of the cumulative gradient and the historical compression error cache to obtain the compressed cumulative gradient.
[0054] Among them, the method of spatial compression based on the sum of the accumulated gradient and the historical compression error cache according to the globally optimal gradient compression rate can be Top-k sparsity compression or other spatial compression methods.
[0055] In one possible implementation, after obtaining the compressed cumulative gradient, the working node can also update the historical compression error cache based on the cumulative gradient and the compressed cumulative gradient to obtain the updated compression error cache, so as to perform the next round of error compensation based on the updated compression error cache.
[0056] Methods for updating the historical compression error cache include: ; in, This represents the updated compression error cache obtained by the i-th working node after the T-th round of compensation. This represents the historical compression error cache of the i-th working node during the T-th round of compensation. This represents the cumulative gradient obtained by the i-th working node during the T-th round of compensation. It represents the compressed cumulative gradient of the i-th working node during the T-th round of compensation.
[0057] S240: Upload the compressed cumulative gradient to the server so that the server can update the global model parameters based on the compressed cumulative gradient uploaded by each worker node.
[0058] The method by which the server updates the global model parameters based on the compressed cumulative gradients uploaded by each worker node is detailed in the server-side method implementation, and will not be repeated here.
[0059] The spatiotemporal compression joint optimization method for federated learning provided in this application allows each worker node to obtain the globally optimal communication frequency and globally optimal gradient compression rate determined by the server based on its own initialization configuration information and environmental data reported by each worker node. Based on this globally optimal communication frequency and globally optimal gradient compression rate, local training, determination and uploading of the spatiotemporally compressed cumulative gradient are achieved, and the server performs global aggregation based on the compressed cumulative gradient. This application overcomes the limitations of traditional methods that orthogonally optimize communication frequency and gradient compression rate, achieving deep parameter collaboration. It not only significantly improves training efficiency but also has low computational complexity, can capture network fluctuations and changes in node computing power in real time, and exhibits extremely fast adaptive response.
[0060] Algorithms for parameter updates in federated learning include FedAVG (Federated Averaging), PASGD (Parallel Asynchronous Stochastic Gradient Descent), and γ-FedHT (γ-Federated Hard Thresholding). The following section uses these three algorithms as examples and compares them with the algorithm provided in this application across multiple dimensions to demonstrate the advantages of the algorithm provided in this application. Table 1 shows the differences and commonalities between the algorithm provided in this application and other algorithms, and Table 2 presents the experimental results. The data in the tables represent the ratio of the convergence speed of the algorithms on the specified task, with the FedAVG convergence speed as the baseline.
[0061] ; ; Furthermore, the spatiotemporal compression joint optimization method for federated learning provided in this application can be applied to any federated learning scenario. Two such scenarios are illustrated below: Scenario 1: Large-scale model pre-training / fine-tuning across geographically distributed data centers. In wide area network environments between different data centers, bandwidth is limited and fluctuates frequently. The embodiments of this application can automatically balance local computational load and transmission compression rate, significantly shortening the overall training cycle.
[0062] Scenario 2: Edge computing and mobile internet applications. The computing power of the mobile phones or IoT devices participating in training varies greatly, and the network environment (Wi-Fi / 5G) is unstable. The embodiments of this application can, to some extent, alleviate the obstacles to training posed by some extremely slow nodes, thereby achieving efficient federated learning.
[0063] Based on the above method embodiments, another embodiment of this application provides a spatiotemporal compression joint optimization apparatus for federated learning, wherein the apparatus is applied to a server, such as... Figure 3 As shown, the device includes: The acquisition unit 310 is used to acquire the initial configuration information of the server and the environmental data information reported by each working node. The initial configuration information includes a preset convergence factor and / or a search interval. The search interval is a feasible interval of the communication frequency. The communication frequency is the number of consecutive training steps executed locally by the working node between two adjacent global aggregations on the server. The calculation unit 320 is used to calculate the communication frequency evaluation value corresponding to each working node for each communication frequency within the search interval, based on the environmental data information of the working node and the initialization configuration information. The first determining unit 330 is used to determine the maximum value among the communication frequency evaluation values of all working nodes under the communication frequency. The selection unit 340 is used to select the minimum value from the maximum values of all communication frequency evaluation values in the search interval, and to determine the communication frequency corresponding to the minimum value as the global optimal communication frequency. The second determining unit 350 is used to determine the global optimal gradient compression rate based on the global optimal communication frequency and the preset convergence factor. The broadcast unit 360 is used to broadcast the globally optimal communication frequency and the globally optimal gradient compression rate to each working node.
[0064] In one possible implementation, the environmental data information includes at least one of the following: local computation time, network latency, compression factor, real-time bandwidth, and number of model parameters.
[0065] In one possible implementation, the computing unit 320 is used to calculate the communication frequency evaluation value corresponding to each working node for each communication frequency within the search interval, according to a preset evaluation function. The preset evaluation function includes: ; Among them, the Indicates the communication frequency, the Indicates communication frequency The corresponding communication frequency evaluation value, the Indicates the network latency, the This represents the compression coefficient. Represents the preset convergence factor, the The model parameter quantity is represented by the following. Indicates the real-time bandwidth, the This indicates the time taken for the local computation.
[0066] In one possible implementation, the second determining unit 350 is configured to determine the global optimal gradient compression rate based on a preset correlation relationship; The preset association relationships include: ; Among them, the Indicates the communication frequency, the Represents the gradient compression ratio, the This represents the preset convergence factor.
[0067] In one possible implementation, the initialization configuration information also includes a learning rate; The acquisition unit 310 is further configured to acquire the compressed cumulative gradient uploaded by each working node, wherein the compressed cumulative gradient is the cumulative gradient obtained by the working node through spatiotemporal compression based on the global optimal communication frequency and the global optimal gradient compression rate. The device further includes: The update unit is used to update the global model parameters obtained in the previous round of global aggregation based on the compressed cumulative gradient of each working node, the learning rate, and the number of working nodes, so as to obtain the global model parameters obtained in the current round of global aggregation. The broadcast unit 360 is used to broadcast the global model parameters obtained from the current round of global aggregation to each working node.
[0068] The spatiotemporal compression joint optimization device for federated learning provided in this application embodiment can first obtain initialization configuration information (including preset convergence factors and / or search intervals) and environmental data information of each working node from the server. Then, it determines the globally optimal communication frequency using this initialization configuration information and environmental data information, and finally determines the globally optimal gradient compression ratio by combining it with the preset convergence factor and broadcasting it to each working node. This application embodiment breaks through the limitations of traditional methods that orthogonally optimize communication frequency and gradient compression ratio, achieving deep parameter coordination. This not only significantly improves training efficiency but also has low computational complexity, can capture network fluctuations and node computing power changes in real time, and has extremely fast adaptive response. Furthermore, a univariate convex function (i.e., the preset evaluation function) is constructed using communication frequency as the independent variable and the corresponding communication frequency evaluation value as the dependent variable. This simplifies complex optimization to a univariate convex function search, further reducing computational complexity and enabling real-time capture of network fluctuations and node computing power changes. The known quantities in the preset evaluation function include network latency, compression coefficient, preset convergence factor, number of model parameters, real-time bandwidth, and local computation time. By combining a preset convergence factor to pre-construct the mapping relationship between communication frequency and gradient compression rate, synchronous optimization of two variables is achieved, transforming the two-variable optimization problem into the simplest hyperparameter setting.
[0069] Based on the above method embodiments, another embodiment of this application provides a spatiotemporal compression joint optimization device for federated learning, wherein the device is applied to a worker node, such as... Figure 4 As shown, the device includes: The acquisition unit 410 is used to acquire the globally optimal communication frequency and the globally optimal gradient compression ratio sent by the server. The globally optimal communication frequency and the globally optimal gradient compression ratio are determined by the server based on its own initialization configuration information and the environmental data information reported by each working node. The initialization configuration information includes a preset convergence factor and / or a search interval. The search interval is a feasible interval for the communication frequency. The communication frequency is the number of consecutive training steps executed locally by the working node between two adjacent global aggregations on the server. The calculation unit 420 is used to perform gradient iteration calculations locally based on the global optimal communication frequency to obtain the cumulative gradient; Compression unit 430 is used to spatially compress the sum of the cumulative gradient and the historical compression error cache according to the global optimal gradient compression rate to obtain the compressed cumulative gradient. Upload unit 440 is used to upload the compressed cumulative gradient to the server so that the server can update the global model parameters based on the compressed cumulative gradient uploaded by each worker node.
[0070] In one possible implementation, the device further includes: An update unit is used to update the historical compression error cache according to the accumulated gradient and the compressed accumulated gradient to obtain an updated compression error cache, so as to perform the next round of error compensation based on the updated compression error cache.
[0071] The spatiotemporal compression joint optimization device for federated learning provided in this application embodiment enables each worker node to obtain the globally optimal communication frequency and globally optimal gradient compression rate determined by the server based on its own initialization configuration information and environmental data reported by each worker node. Based on this globally optimal communication frequency and globally optimal gradient compression rate, local training, determination and uploading of the spatiotemporally compressed cumulative gradient are achieved, and the server performs global aggregation based on the compressed cumulative gradient. This application embodiment overcomes the limitations of traditional methods that orthogonally optimize communication frequency and gradient compression rate, achieving deep parameter collaboration. It not only significantly improves training efficiency but also has low computational complexity, can capture network fluctuations and changes in node computing power in real time, and exhibits extremely fast adaptive response.
[0072] Based on the above method embodiments, another embodiment of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method as described in any of the above embodiments. Any of the above embodiments includes any of the methods in which a server is the execution subject, and also includes any of the methods in which a worker node is the execution subject.
[0073] Based on the above method embodiments, another embodiment of this application provides an electronic device or computer device, including: One or more processors; The processor is coupled to a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the electronic device or computer device performs the method described in any of the above embodiments. Each of the above embodiments includes any embodiment of the method in which the server is the executing entity, and also includes any embodiment of the method in which the worker node is the executing entity.
[0074] Based on the above method embodiments, another embodiment of this application provides a spatiotemporal compression joint optimization system for federated learning, the system including a server and multiple worker nodes; The server is used to execute the method described in any embodiment where the server is the execution subject; The working node is used to execute the method described in any embodiment where the working node is the execution subject.
[0075] Specifically, such as Figure 5As shown, the server (referred to as global server 510 in the figure) includes a search parameter module 511, an update collection module 512 and a global aggregation module 513. Each working node 520 (taking 3 working nodes as an example in the figure) includes an environment monitoring module 521, a local training module 522 and a communication compression module 523.
[0076] The environmental monitoring module 521 is used to monitor the local environmental data information of the working node and report the environmental data information to the server. The environmental data information includes at least one of the following: local computing time, network latency, compression coefficient, real-time bandwidth, and model parameter quantity.
[0077] The search parameter module 511 is used to obtain the initialization configuration information of the server and the environmental data information reported by each worker node. The initialization configuration information includes a preset convergence factor and / or a search interval, where the search interval is a feasible interval for communication frequencies, and the communication frequency is the number of consecutive training steps executed locally by the worker node between two adjacent global aggregations on the server. For each communication frequency within the search interval, based on the environmental data information and the initialization configuration information of the worker node, a communication frequency evaluation value corresponding to each worker node is calculated, and the maximum value among the communication frequency evaluation values of all worker nodes at that communication frequency is determined. The minimum value is selected from the maximum values of all communication frequency evaluation values within the search interval, and the communication frequency corresponding to the minimum value is determined as the globally optimal communication frequency. Based on the globally optimal communication frequency and the preset convergence factor, the globally optimal gradient compression ratio is determined. The globally optimal communication frequency and the globally optimal gradient compression ratio are broadcast to each worker node.
[0078] Local training module 522 is used to iteratively train local modules based on local raw data; The communication compression module 523 is used to obtain the globally optimal communication frequency and the globally optimal gradient compression ratio sent by the server; perform gradient iteration calculations for the corresponding number of steps based on the globally optimal communication frequency to obtain the cumulative gradient; perform spatial compression on the sum of the cumulative gradient and the historical compression error cache according to the globally optimal gradient compression ratio to obtain the compressed cumulative gradient; and upload the compressed cumulative gradient to the server so that the server can update the global model parameters based on the compressed cumulative gradient uploaded by each working node.
[0079] Update collection module 512 to collect the compressed cumulative gradient uploaded by each worker node.
[0080] The global aggregation module 513 is used to update the global model parameters obtained in the previous round of global aggregation based on the compressed cumulative gradient of each working node and the number of working nodes, so as to obtain the global model parameters obtained in the current round of global aggregation; and broadcast the global model parameters obtained in the current round of global aggregation to each working node.
[0081] The spatiotemporal compression joint optimization system for federated learning provided in this application embodiment can first obtain initialization configuration information (including preset convergence factors and / or search intervals) and environmental data information of each working node from the server. Then, it determines the globally optimal communication frequency using this initialization configuration information and environmental data information, and finally determines the globally optimal gradient compression ratio by combining it with the preset convergence factor and broadcasting it to each working node. This application embodiment breaks through the limitations of traditional methods that orthogonally optimize communication frequency and gradient compression ratio, achieving deep parameter synergy. This not only significantly improves training efficiency but also has low computational complexity, can capture network fluctuations and node computing power changes in real time, and has extremely fast adaptive response. Furthermore, a univariate convex function (i.e., the preset evaluation function) is constructed using communication frequency as the independent variable and the corresponding communication frequency evaluation value as the dependent variable. This simplifies complex optimization to a univariate convex function search, further reducing computational complexity and enabling real-time capture of network fluctuations and node computing power changes. The known quantities in the preset evaluation function include network latency, compression coefficient, preset convergence factor, number of model parameters, real-time bandwidth, and local computation time. By combining a preset convergence factor to pre-construct the mapping relationship between communication frequency and gradient compression rate, synchronous optimization of two variables is achieved, transforming the two-variable optimization problem into the simplest hyperparameter setting.
[0082] Based on the above embodiments, another embodiment of this application provides a computer program product, which includes instructions that, when executed on a computer or processor, cause the computer or processor to perform the method described in any of the above embodiments.
[0083] The above-described device and system embodiments correspond to the method embodiments and have the same technical effects. For detailed descriptions, please refer to the method embodiments. The device embodiments are derived from the method embodiments; detailed descriptions can be found in the method embodiments section, and will not be repeated here. Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.
Claims
1. A spatiotemporal compression joint optimization method for federated learning, characterized in that, The method is applied to a server, and the method includes: Obtain the initial configuration information of the server and the environmental data information reported by each worker node. The initial configuration information includes a preset convergence factor and / or a search interval. The search interval is a feasible interval of the communication frequency. The communication frequency is the number of consecutive training steps executed locally by the worker node between two adjacent global aggregations on the server. For each communication frequency within the search interval, based on the environmental data information of the working node and the initialization configuration information, calculate the communication frequency evaluation value corresponding to each working node, and determine the maximum value among all communication frequency evaluation values of all working nodes under the communication frequency. Select the minimum value from the maximum values of all communication frequency evaluation values within the search interval, and determine the communication frequency corresponding to the minimum value as the globally optimal communication frequency. The global optimal gradient compression rate is determined based on the global optimal communication frequency and the preset convergence factor. The globally optimal communication frequency and the globally optimal gradient compression rate are broadcast to each working node.
2. The method according to claim 1, characterized in that, The environmental data information includes at least one of the following: local computation time, network latency, compression factor, real-time bandwidth, and number of model parameters.
3. The method according to claim 2, characterized in that, For each communication frequency within the search interval, based on the environmental data information of the working node and the initialization configuration information, calculate the communication frequency evaluation value corresponding to each working node, including: For each communication frequency within the search interval, a communication frequency evaluation value corresponding to each working node is calculated according to a preset evaluation function. The preset evaluation function includes: ； Among them, the Indicates the communication frequency, the Indicates communication frequency The corresponding communication frequency evaluation value, the Indicates the network latency, the This represents the compression coefficient. Represents the preset convergence factor, the The model parameter quantity is represented by the following. Indicates the real-time bandwidth, the This indicates the time taken for the local computation.
4. The method according to claim 2, characterized in that, Determining the globally optimal gradient compression rate based on the globally optimal communication frequency and the preset convergence factor includes: The globally optimal gradient compression rate is determined based on a preset correlation relationship; The preset association relationships include: ； Among them, the Indicates the communication frequency, the Represents the gradient compression ratio, the This represents the preset convergence factor.
5. The method according to any one of claims 1-4, characterized in that, The initialization configuration information also includes a learning rate, and the method further includes: Obtain the compressed cumulative gradient uploaded by each worker node, wherein the compressed cumulative gradient is the cumulative gradient obtained by the worker node through spatiotemporal compression based on the globally optimal communication frequency and the globally optimal gradient compression rate; Based on the compressed cumulative gradient of each working node, the learning rate, and the number of working nodes, the global model parameters obtained in the previous round of global aggregation are updated to obtain the global model parameters obtained in the current round of global aggregation. The global model parameters obtained from this round of global aggregation are broadcast to each worker node.
6. A spatiotemporal compression joint optimization method for federated learning, characterized in that, The method is applied to a working node, and the method includes: Obtain the globally optimal communication frequency and globally optimal gradient compression ratio sent by the server. The globally optimal communication frequency and globally optimal gradient compression ratio are determined by the server based on its own initialization configuration information and the environmental data information reported by each worker node. The initialization configuration information includes a preset convergence factor and / or a search interval. The search interval is a feasible interval for the communication frequency. The communication frequency is the number of consecutive training steps executed locally by the worker node between two adjacent global aggregations on the server. Based on the globally optimal communication frequency, perform gradient iteration calculations for the corresponding number of steps locally to obtain the cumulative gradient; According to the global optimal gradient compression rate, the sum of the cumulative gradient and the historical compression error cache is spatially compressed to obtain the compressed cumulative gradient; The compressed cumulative gradient is uploaded to the server so that the server can update the global model parameters based on the compressed cumulative gradient uploaded by each worker node.
7. The method according to claim 6, characterized in that, The method further includes: The historical compression error cache is updated based on the accumulated gradient and the compressed accumulated gradient to obtain the updated compression error cache, so as to perform the next round of error compensation based on the updated compression error cache.
8. A spatiotemporal compression joint optimization device for federated learning, characterized in that, The device is used in a server, and the device includes: The acquisition unit is used to acquire the initial configuration information of the server and the environmental data information reported by each worker node. The initial configuration information includes a preset convergence factor and / or a search interval. The search interval is a feasible interval of the communication frequency. The communication frequency is the number of consecutive training steps executed locally by the worker node between two adjacent global aggregations on the server. The calculation unit is used to calculate the communication frequency evaluation value corresponding to each working node for each communication frequency within the search interval, based on the environmental data information of the working node and the initialization configuration information. The first determining unit is used to determine the maximum value among the communication frequency evaluation values of all working nodes under the communication frequency. The selection unit is used to select the minimum value from the maximum values of all communication frequency evaluation values in the search interval, and to determine the communication frequency corresponding to the minimum value as the global optimal communication frequency. The second determining unit is used to determine the global optimal gradient compression rate based on the global optimal communication frequency and the preset convergence factor. The broadcast unit is used to broadcast the globally optimal communication frequency and the globally optimal gradient compression rate to each working node.
9. A spatiotemporal compression joint optimization device for federated learning, characterized in that, The device is applied to a working node, and the device includes: The acquisition unit is used to acquire the globally optimal communication frequency and globally optimal gradient compression ratio sent by the server. The globally optimal communication frequency and globally optimal gradient compression ratio are determined by the server based on its own initialization configuration information and the environmental data information reported by each worker node. The initialization configuration information includes a preset convergence factor and / or a search interval. The search interval is a feasible interval for the communication frequency. The communication frequency is the number of consecutive training steps executed locally by the worker node between two adjacent global aggregations on the server. The calculation unit is used to perform gradient iteration calculations locally based on the globally optimal communication frequency, and obtain the cumulative gradient. A compression unit is used to spatially compress the sum of the cumulative gradient and the historical compression error cache according to the global optimal gradient compression rate to obtain the compressed cumulative gradient. An upload unit is used to upload the compressed cumulative gradient to the server so that the server can update the global model parameters based on the compressed cumulative gradient uploaded by each worker node.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method as described in any one of claims 1-5, or when the program is executed by a processor, it implements the method as described in any one of claims 6-7.
11. An electronic device, characterized in that, The electronic device includes: One or more processors; The processor is coupled to a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the electronic device performs the method as described in any one of claims 1-5, or the method as described in any one of claims 6-7.
12. A spatiotemporal compression joint optimization system for federated learning, characterized in that, The system includes a server and multiple worker nodes; The server is configured to perform the method according to any one of claims 1-5; The working node is used to execute the method of any one of claims 6-7.