Federated learning method, electronic device and storage medium based on communication data compression
By constructing and uploading the re-arranged update gradient number list, the problem of low transmission efficiency of model parameters in federated learning is solved, and lossless compression and efficient training are achieved.
Patent Information
- Application Number
- CN202410811913.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2044-06-21
AI Technical Summary
In the large-scale model training scenario of federated learning, the network transmission of model parameters is large, resulting in communication becoming a bottleneck in training speed. The lossy compression method of the existing technology will affect the convergence speed of the model.
A federated learning method based on lossless compression is proposed. By obtaining model parameters and updating the sequence number of the gradient, a third ordered list is constructed for data rearrangement and uploading, ensuring the efficiency and accuracy of network transmission.
Lossless compression of model parameters is realized, the amount of data transmitted by the network is reduced and the training efficiency of federated learning models is improved.
Smart Images
Figure CN118485138B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of federated learning, and in particular to a federated learning method based on communication data compression, an electronic device and a corresponding storage medium. Background Art
[0002] At present, with the rapid development of artificial intelligence technology, the application of artificial intelligence technology is becoming more and more extensive. It has been applied to many fields such as image analysis, speech recognition, text processing, intelligent recommendation, security detection, etc. Federated learning and large model technology have become the forefront of artificial intelligence technology. In the large model training scenario based on federated learning, the data of model parameters transmitted in a single time between the parameter server and the distributed training terminal is relatively large. The latest GPT4 model has about A single node needs to transmit 6.4TB of parameters through the network at a time, and network communication becomes a bottleneck affecting the training speed.
[0003] Although there are some existing methods for parameter compression in federated learning, such as using pooling operators to compress communication data, although the amount of communication data can be reduced, the pooling operator is a lossy compression and will reduce the convergence speed of the model. For example, some existing technologies use top-k parameter sparsification, parameter quantization, and historical gradient reuse to compress the gradient of network transmission, but this technology is essentially a lossy compression technology, which will affect the convergence speed of the model. Summary of the invention
[0004] The purpose of the embodiments of the present invention is to provide a federated learning method, electronic device and storage medium based on communication data compression, which reduces the time consumption of network communication by losslessly compressing model parameters transmitted over the network, so as to at least solve some of the problems in the background technology.
[0005] In order to achieve the above object, the present invention provides a federated learning method based on communication data compression, which is applied to a client node. The method includes:
[0006] Obtain model parameters and a first ordered list, and update the local model according to the model parameters; the first ordered list includes serial numbers of several components; after training the updated local model with a local training data set, an update gradient is obtained; the serial numbers of several components with the largest absolute values in the update gradient are written into a second ordered list; calculate the common serial numbers that belong to both the first ordered list and the second ordered list, and write the components corresponding to the common serial numbers in the update gradient into a third ordered list; write the common serial numbers in the first ordered list that are adjacent to the non-common serial numbers into the third ordered list; write the serial numbers that belong to the second ordered list but not to the first ordered list and the components corresponding to the serial numbers in the update gradient into the third ordered list; upload the obtained third ordered list.
[0007] Optionally, the number of components corresponding to the sequence numbers in the first ordered list and the second ordered list are both .
[0008] Optionally, the first ordered list includes a plurality of components whose serial numbers are the ones with the largest absolute values in the update momentum of the global model and are arranged in descending order of absolute values. The sequence number of the component.
[0009] Optionally, if the sequence number of the component in the updated gradient and the component are both written into the third ordered list, the position of the sequence number of the component is the previous position of the component.
[0010] Optionally, the method includes a model initialization step; the model initialization step includes: setting initialization parameters and setting counting variables.
[0011] Optionally, obtaining the model parameters includes: a server node randomly selecting a number of client nodes from a whole set consisting of all client nodes of the federated learning system; and receiving the model parameters sent by the server node to the selected client nodes.
[0012] Optionally, calculating the common sequence number that belongs to both the first ordered list and the second ordered list, and writing the component corresponding to the common sequence number in the updated gradient into a third ordered list; and writing the common sequence number that is located after the non-common sequence number in the first ordered list into the third ordered list, including: initializing the loop control parameter p 1. Position control parameters q is 1 and the third ordered list is empty; from the first ordered list p If the sequence number starts from the second ordered list and , put the component corresponding to the serial number in the update gradient into the first q Locations, p andq If the sequence number is in the second ordered list and is based on the first ordered list, -1 sequence number is also in the second ordered list, then the component corresponding to the sequence number in the update gradient is placed in the first q Locations, p and q If the sequence number is in the second ordered list, but is based on the first ordered list, -1 sequence number is not in the second ordered list, then the sequence number and the component corresponding to the sequence number in the update gradient are placed in the first q and q +1 position, p Plus one, q Add two; if the number is not in the second ordered list, p plus one; when If it is greater than the total number of sequence numbers in the first ordered list, the execution ends; otherwise, based on p A new sequence number is determined in the first ordered list, and the cycle is repeated.
[0013] Optionally, the sequence number that belongs to the second ordered list but not to the first ordered list and the component corresponding to the sequence number in the updated gradient are written into the third ordered list, including: for each sequence number in the second ordered list, if the sequence number is not in the first ordered list, writing the sequence number to the end of the third ordered list, and then writing the component corresponding to the sequence number in the updated gradient to the end of the third ordered list.
[0014] The present application also provides a federated learning method based on communication data compression, which is applied to a server node. The method includes: sending model parameters and a first ordered list to a client node, wherein the model parameters are used to update a local model of the client node; the first ordered list includes serial numbers of several components; receiving a third ordered list uploaded by the client node; generating an update gradient corresponding to the client node according to the serial numbers and components in the third ordered list; aggregating the generated update gradients to generate a global model update gradient, and updating the global model in combination with the update momentum and model parameters.
[0015] Optionally, sending the model parameters and the first ordered list to the client node includes: after sending the model parameters to the client node, generating the first ordered list and sending the first ordered list to the client node.
[0016] Optionally, an update gradient corresponding to the client node is generated according to the sequence number and components in the third ordered list, including: processing the elements in the third ordered list in sequence as follows: if the element is a sequence number and the sequence number is in the first ordered list, processing all components between the element and the next sequence number, and copying all the components in sequence to the update gradient; the positions where all components are copied to the update gradient are determined based on the sequence number and the required number of sequence numbers immediately following the sequence number in the first ordered list; if the element is a sequence number and the sequence number is not in the first ordered list, processing the next component of the element, and copying the next component to the update gradient; the position where the component is copied to the update gradient is determined based on the sequence number; if the element is a component and is the first element in the third ordered list, copying the element as the first component of the update gradient.
[0017] Optionally, the generated update gradients are aggregated to generate a global model update gradient, including: determining a weight value based on local sample data of the client node corresponding to each update gradient; and obtaining a global model update gradient by performing a weighted average calculation based on the update gradient and the corresponding weight value.
[0018] Optionally, the global model is updated in combination with the update momentum and model parameters, including: obtaining the update momentum of this round by weighted summation of the update momentum of the previous round and the update gradient of the global model; calculating the correction amount according to the update momentum of this round and the learning rate, correcting the model parameters with the correction amount, and then updating the global model.
[0019] Optionally, the method is configured to be executed in a loop when the following preset conditions are met; the preset conditions include: after updating the global model in combination with updating the momentum and model parameters, determining that the current state of the federated learning system does not meet the training stop condition of the federated learning.
[0020] The present invention also provides a federated learning device based on communication data compression, which is applied to a client node, and the device includes: a data acquisition module, which is used to acquire model parameters and a first ordered list, and update the local model according to the model parameters; the first ordered list includes the serial numbers of several components; a gradient processing module, which is used to train the updated local model with a local training data set to obtain an updated gradient; write the serial numbers of several components with the largest absolute values in the updated gradient into a second ordered list; a first rearrangement module, which is used to calculate the common serial numbers that belong to both the first ordered list and the second ordered list, and write the components corresponding to the common serial numbers in the updated gradient into a third ordered list; write the common serial numbers in the first ordered list that are adjacent to the non-common serial numbers into the third ordered list; a second rearrangement module, which is used to write the serial numbers that belong to the second ordered list but not to the first ordered list and the components corresponding to the serial numbers in the updated gradient into the third ordered list; and a data upload module, which is used to upload the obtained third ordered list.
[0021] The present invention also provides a federated learning device based on communication data compression, which is applied to a server node. The device includes: a data sending module, which is used to send model parameters and a first ordered list to a client node, and the model parameters are used to update the local model of the client node; the first ordered list includes the serial numbers of several components; a data receiving module, which is used to receive a third ordered list uploaded by the client node; a gradient parsing module, which is used to generate an update gradient corresponding to the client node according to the serial number and component in the third ordered list; and a model updating module, which is used to aggregate the generated update gradients to generate a global model update gradient, and update the global model in combination with the update momentum and model parameters.
[0022] The present invention also provides an electronic device, comprising: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the aforementioned federated learning method based on communication data compression by executing the instructions stored in the memory.
[0023] The present invention also provides a machine-readable storage medium on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the aforementioned federated learning method based on communication data compression.
[0024] The present invention also provides a computer program product, including a computer program, which implements the aforementioned federated learning method based on communication data compression when executed by a processor.
[0025] The above technical solution has the following beneficial effects:
[0026] The Top-k sparsification technology of model parameters was improved, and a new lossless compression method of model parameter serial numbers was proposed, which can reduce the amount of network transmission data and network communication delay in federated learning scenarios and improve the training efficiency of federated learning models.
[0027] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:
[0029] Figure 1 Schematically showing a schematic diagram of steps of a federated learning method based on communication data compression applied to a client node according to an embodiment of the present invention;
[0030] Figure 2 A schematic diagram of the steps of a federated learning method based on communication data compression applied to a server node according to an embodiment of the present invention is shown;
[0031] Figure 3 A schematic diagram of the implementation process of the federated learning method based on communication data compression according to an embodiment of the present invention is schematically shown;
[0032] Figure 4 A schematic diagram of the structure of a federated learning device based on communication data compression according to an embodiment of the present invention is shown;
[0033] Figure 5 The figure schematically shows the internal structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0034] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.
[0035] Figure 1 The following schematically shows the steps of a federated learning method based on communication data compression applied to a client node according to an embodiment of the present invention. Figure 1 As shown, a federated learning method based on communication data compression, the method comprising:
[0036] S01, obtaining model parameters and a first ordered list, and updating a local model according to the model parameters; the first ordered list includes serial numbers of several components;
[0037] S02, after training the updated local model using the local training data set, obtain an updated gradient; write the sequence numbers of several components with the largest absolute values in the updated gradient into a second ordered list;
[0038] S03, calculating the common sequence numbers belonging to both the first ordered list and the second ordered list, and writing the components corresponding to the common sequence numbers in the update gradient into a third ordered list; and writing the common sequence numbers adjacent to the non-common sequence numbers in the first ordered list into the third ordered list;
[0039] S04, write the sequence number that belongs to the second ordered list but does not belong to the first ordered list and the component corresponding to the sequence number in the update gradient into the third ordered list;
[0040] S05. Upload the obtained third ordered list.
[0041] Through the above implementation methods, a parameter selection and rearrangement algorithm for the local model parameter update gradient is proposed, which can efficiently construct the local model training update data required to be uploaded by the client node, realize lossless compression of the model parameter sequence number, and reduce the amount of data transmitted over the network and the network communication delay in the federated learning scenario, thereby improving the training efficiency of the federated learning model.
[0042] In some embodiments of the present application, the number of components corresponding to the serial numbers in the first ordered list and the second ordered list are both In the aforementioned implementation, the number of components corresponding to the serial numbers in the first ordered list and the second ordered list is not limited. In the present implementation, the number of components corresponding to the serial numbers in the first ordered list and the second ordered list is limited to be equal, that is, both are When the number of sequence numbers in the first list and the second list is equal, the efficiency of the communication data compression algorithm in this embodiment has an optimal value.
[0043] In some embodiments of the present application, the first ordered list includes several components whose serial numbers are the largest absolute values in the update momentum of the global model and are arranged in descending order of absolute value. The parameter server S is in the After the round model parameters are sent, extract the global model update momentum The absolute value of the largest components, and place the above components in descending order according to their absolute values. Copy the sequence number in the ordered list , the first ordered list; and It is sent to all selected federated learning client nodes. yes No. elements, Represents the client node The updated gradient of the local model parameters No. The amount in The serial number in , In this implementation manner, the client node dynamically adjusts the data structure of the uploaded data based on the parameter sequence number list sent by the parameter server, which can reduce the amount of data that the client node needs to transmit.
[0044] In some embodiments of the present application, if the sequence number of the component in the update gradient and the component are both written into the third ordered list, the position of the sequence number of the component is in the previous position of the component. This embodiment provides a specification for parsing the third ordered list on the server node by limiting the order of the sequence number and the component.
[0045] In some embodiments of the present application, the method includes a model initialization step; the model initialization step includes: setting initialization parameters and setting count variables. If the federated learning system is in a state before the first round of training, it may be necessary to initialize the model of the federated learning system. In the federated learning system, it is assumed that the federated learning parameter server is S, and the parameter server belongs to the server node in the federated learning system. All federated learning client nodes constitute the set , the total number of all federated learning client nodes is The specific steps of model initialization are: the parameter server S initializes the parameters of the adopted deep neural network model, and the initialization parameters are ,in , is the total number of parameter dimensions of the global model. Set the count variable ,initialization . At this time, there is no previous round of model training, and many parameters or variables will be set to initial values. However, it should be noted that the model initialization is not necessary. For example, after the federated learning system starts the second round and subsequent training, the aforementioned federated learning method based on communication data compression can be applied, but it is no longer necessary to perform the initialization before the federated learning method based on communication data compression.
[0046] In some embodiments of the present application, obtaining model parameters includes: a server node randomly selects a number of client nodes from the entire set of all client nodes in the federated learning system; and receives the model parameters sent by the server node to the selected client nodes. This embodiment provides the execution steps of sending model parameters: When the round model parameters are sent, the parameter server S Randomly select from client nodes Client nodes, 0< ≤ , the selected federated learning client node is , where i is the client node number, 0< ≤ . Set the model global model parameters Sent to the selected federated learning client node, starting the Round of local model training and model parameter update. , and are all positive integers. For example, the parameter server S is Randomly select from client nodes client nodes.
[0047] In the aforementioned step S02, During the local model training, each selected federated learning client node Using global model parameters Update the local model, use the local training data set to train the local model using the stochastic gradient descent algorithm, and calculate the update gradient of the local model parameters , .right Sort the d components of by their absolute values, and place the components with the largest absolute values in The serial numbers in the list form an ordered list , with the number of components equal to k For example, , which is the second ordered list.
[0048] In some implementations of the present application, a common sequence number belonging to both the first ordered list and the second ordered list is calculated, and a component corresponding to the common sequence number in the update gradient is written into a third ordered list; and a common sequence number located after a non-common sequence number in the first ordered list is written into the third ordered list, which can be implemented according to the following logic:
[0049] Initialize loop control parameters p 1. Position control parameters q is 1 and the third ordered list is empty; from the first ordered list p The sequence number starts from At the beginning, this number corresponds to the first If the sequence number is in the second ordered list and , put the component corresponding to the serial number in the update gradient into the first q Locations, p and q If the sequence number is in the second ordered list and is based on the first ordered list, -1 sequence number is also in the second ordered list, then the component corresponding to the sequence number in the update gradient is placed in the first q Locations, p and q If the sequence number is in the second ordered list, but is based on the first ordered list, -1 sequence number is not in the second ordered list, then the sequence number and the component corresponding to the sequence number in the update gradient are placed in the first q and q +1 position, p Plus one, q Add two; if the number is not in the second ordered list, p plus one; when If it is greater than the total number of sequence numbers in the first ordered list, the execution ends; otherwise, based on p A new sequence number is determined in the first ordered list, and the cycle is repeated.
[0050] Specifically, set the count variable and and an empty ordered list , that is, the third ordered list; initialize and The steps for rearranging hit parameters are as follows: During the local model training, if No. The sequence number of the components in the ordered list Zhongqie , then No. Copy the components to No. positions, then and The count of is increased by 1; if No. The sequence number of the components in the ordered list Zhongqie No. The sequence number of the components in the ordered list In No. Copy the components to No. positions, then and The count of is increased by 1; if No. The sequence number of the components in the ordered list Zhongqie No. The sequence number of the component is not in the ordered list In Copy to volume No. position, No. Copy the components to No. positions, then The count of is increased by 1, The count of is increased by 2; if No. The sequence number of the component is not in the ordered list In The count of is increased by 1. The aforementioned hit parameter rearrangement step is executed cyclically. hour, k For an ordered list The total number of serial numbers in , and then terminate this cycle. This implementation provides a specific execution logic, which has the advantage of being simple and easy to implement.
[0051] In some embodiments of the present application, writing the sequence number that belongs to the second ordered list but not to the first ordered list and the component corresponding to the update gradient of the sequence number into the third ordered list includes: for each sequence number in the second ordered list, if the sequence number is not in the first ordered list, writing the sequence number to the end of the third ordered list, and then writing the component corresponding to the update gradient of the sequence number to the end of the third ordered list. During the local model training, The If the sequence number is not in the ordered list , then copy the serial number to No. Position, the serial number in The corresponding components in are copied to No. positions, then The count of is increased by 1, The count is increased by 2; if the sequence number is in the ordered list In The count is increased by 1. The above steps are repeated. , k For an ordered list When the total number of sequence numbers in is reached, the loop process is terminated. Based on the previous embodiment continuous.
[0052] Optionally, in step S05, the client node Will include An ordered list of elements Send to parameter server .
[0053] The present application also provides a federated learning method based on communication data compression, which is applied to a server node and implemented in conjunction with the aforementioned method applied to a client node. Figure 2 The following schematically shows the steps of a federated learning method based on communication data compression applied to a server node according to an embodiment of the present invention. Figure 2 As shown, the method includes:
[0054] S01. Sending model parameters and a first ordered list to a client node, wherein the model parameters are used to update a local model of the client node; the first ordered list includes serial numbers of several components;
[0055] S02, receiving a third ordered list uploaded by the client node;
[0056] S03, generating an update gradient corresponding to the client node according to the sequence number and component in the third ordered list;
[0057] S04. Aggregate the generated update gradients to generate global model update gradients, and update the global model in combination with the update momentum and model parameters.
[0058] In this implementation, the server node, i.e., the parameter server If the steps are numbered consecutively based on the federated learning method based on communication data compression implemented by the client node, step S01 in this embodiment should be S00, that is, before the client obtains the model parameters and the first ordered list, and steps S02 to S04 in this embodiment should be S06 to S08, that is, after uploading the obtained third ordered list. When the round model parameters are updated, the parameter server Receive all selected federated learning client nodes Ordered list of uploads , and based on this ordered list Generate updated gradients and proceed to the subsequent federated learning steps. The generation method of the first ordered list is as mentioned above and will not be repeated here.
[0059] In some embodiments of the present application, sending model parameters and a first ordered list to a client node includes: after sending the model parameters to the client node, generating the first ordered list and sending the first ordered list to the client node. This embodiment proposes a method for transmitting a parameter sequence number list using the network's idle bandwidth. After the parameter server sends the model parameters, the network bandwidth will be idle. The parameter server uses this time to send the parameter sequence number list, and the client node uses the idle network bandwidth and network card to receive the parameter sequence number list. The advantage of this embodiment is that it does not bring additional delays to the overall progress of model training.
[0060] In some implementations of the present application, generating an update gradient corresponding to the client node according to the sequence number and component in the third ordered list includes: sequentially performing the following processing on the elements in the third ordered list:
[0061] If the element is a sequence number and the sequence number is in the first ordered list, then process all components between the element and the next sequence number, and copy all the components to the updated gradient in sequence; the positions where all components are copied to the updated gradient are determined based on the sequence number and the required number of sequence numbers immediately following the sequence number in the first ordered list; if the element is a sequence number and the sequence number is not in the first ordered list, then process the next component of the element and copy the next component to the updated gradient; the position where the component is copied to the updated gradient is determined based on the sequence number; if the element is a component and is the first element in the third ordered list, copy the element as the first component of the updated gradient. Specifically, the above scheme can be implemented by the following logical execution steps: Set a counting variable , count variables , count variables and vector , ,initialization , , , All component values of are initialized to 0, for The sum of the sequence numbers and the number of vector components in . If No. elements are ordinal and exist is equal to the sequence number, then let , ,Will No. Elements are copied as No. Quantity, then The count of is increased by 2; if No. Elements are ordinal numbers, but are not in the ordered list In the middle, let Equal to the sequence number, No. Elements are copied as No. Quantity, then The count of is increased by 2; if No. If the elements are vector components, then The count is increased by 1. ,Will No. Elements are copied as No. Quantity, then The count is increased by 1. The above steps are executed repeatedly. Terminate this cycle process.
[0062] In some embodiments of the present application, the generated update gradients are aggregated to generate a global model update gradient, including: determining a weight value based on the local sample data of the client node corresponding to each update gradient; and obtaining the global model update gradient by performing a weighted average calculation based on the update gradient and the corresponding weight value. When the round model parameters are updated, the parameter server Calculate global model update gradients , global model update momentum and the updated parameters of the global model .in For client nodes A locally stored sample dataset, For collection Contains the number of elements. It is the sum of the number of samples of all selected client nodes. is the learning rate, , is the weight coefficient, . Global model update gradient Calculated by the following formula: ;
[0063] The update momentum of the previous round and the global model update gradient are weighted and calculated to get the update momentum of this round, that is: .
[0064] The correction amount is calculated according to the update momentum and learning rate of this round. After the model parameters are corrected by the correction amount, the global model is updated, that is: . This achieves the update of the global model of this round.
[0065] It can be seen that the federated learning method based on communication data compression applied to server nodes proposes a parameter parsing and restoration algorithm for local model parameter update gradients, which can efficiently parse local model training update data from compressed uploaded data.
[0066] In some embodiments of the present application, the method is configured to be executed in a loop when the following preset conditions are met; the preset conditions include: after updating the global model in combination with the updated momentum and model parameters, it is determined that the current state of the federated learning system does not meet the training stop condition of the federated learning. The training stop condition includes: the parameters of the global model Convergence or If the parameters of the global model Convergence or , then terminate the training process, otherwise The value of is increased by 1, and the steps except model initialization are repeated to start the next round of model parameter distribution, local model training, third ordered list upload, third ordered list reception and parsing, global parameter update and other steps. is the preset training cycle threshold constant, is a positive integer greater than 1.
[0067] Through the above implementation methods, the amount of data transmitted over the network and the network communication delay in the federated learning scenario can be reduced, and the training efficiency of the federated learning model can be improved.
[0068] Figure 3 The following schematically shows the implementation process of the federated learning method based on communication data compression according to an embodiment of the present invention. Figure 3 As shown in Figure 1, it includes the following implementation steps: The parameter server initializes the model parameters. The parameter server randomly selects client nodes and sends model parameters. The parameter server updates the momentum according to the global model to build a sequence number list. , which is the first ordered list. The client node receives the model parameters and ordered list sent by the parameter server , train the model based on local sample data and generate local model update gradients, construct an ordered list , that is, the third ordered list and upload it. The parameter server receives The global model parameters are updated and the training termination conditions are determined. When the training termination conditions are met, the training process is terminated. When the training termination conditions are not met, the process jumps to the parameter server to randomly select client nodes and issue model parameters to start the next round of execution.
[0069] Through the above implementation methods, a parameter selection and rearrangement algorithm for the local model parameter update gradient is proposed at the client node, which can efficiently construct the local model training update data required to be uploaded by the client node. And adapted thereto, a parameter parsing and restoration algorithm for the local model parameter update gradient is provided at the server node, which can efficiently parse the compressed uploaded data to obtain the local model training update data.
[0070] Based on the same inventive concept, the present invention also provides a federated learning device based on communication data compression. Figure 4 The structure diagram of the federated learning device based on communication data compression according to the embodiment of the present invention is schematically shown. Figure 4 As shown, applied to a client node, the device includes: a data acquisition module, used to acquire model parameters and a first ordered list, and update a local model according to the model parameters; the first ordered list includes the serial numbers of several components; a gradient processing module, used to train the updated local model with a local training data set to obtain an updated gradient; write the serial numbers of several components with the largest absolute values in the updated gradient into a second ordered list; a first rearrangement module, used to calculate the common serial numbers that belong to both the first ordered list and the second ordered list, and write the components corresponding to the common serial numbers in the updated gradient into a third ordered list; write the common serial numbers in the first ordered list that are adjacent to the non-common serial numbers into the third ordered list; a second rearrangement module, used to write the serial numbers that belong to the second ordered list but not to the first ordered list and the components corresponding to the serial numbers in the updated gradient into the third ordered list; and a data upload module, used to upload the obtained third ordered list.
[0071] In some optional implementations of the present application, the number of components corresponding to the serial numbers in the first ordered list and the second ordered list are both .
[0072] In some optional embodiments of the present application, the first ordered list includes several components whose serial numbers are the largest absolute values in the update momentum of the global model and are arranged in descending order of absolute value. The sequence number of the component.
[0073] In some optional implementations of the present application, if the serial number of the component in the updated gradient and the component are both written into the third ordered list, the position of the serial number of the component is the previous position of the component.
[0074] In some optional implementations of the present application, the method includes a model initialization step; the model initialization step includes: setting initialization parameters and setting counting variables.
[0075] In some optional embodiments of the present application, obtaining model parameters includes: a server node randomly selects several client nodes from the entire set consisting of all client nodes of the federated learning system; and receiving the model parameters sent by the server node to the selected client nodes.
[0076] In some optional implementations of the present application, calculating the common sequence number that belongs to both the first ordered list and the second ordered list, writing the component corresponding to the common sequence number in the updated gradient into a third ordered list; writing the common sequence number that is located after the non-common sequence number in the first ordered list into the third ordered list, includes: initializing the loop control parameter p 1. Position control parameters q is 1 and the third ordered list is empty; from the first ordered list p If the sequence number starts from the second ordered list and , put the component corresponding to the serial number in the update gradient into the first q Locations, p and q If the sequence number is in the second ordered list and is based on the first ordered list, -1 sequence number is also in the second ordered list, then the component corresponding to the sequence number in the update gradient is placed in the first q Locations, p and q If the sequence number is in the second ordered list, but is based on the first ordered list, -1 sequence number is not in the second ordered list, then the sequence number and the component corresponding to the sequence number in the update gradient are placed in the first q and q +1 position, p Plus one, q Add two; if the number is not in the second ordered list, p plus one; when If it is greater than the total number of sequence numbers in the first ordered list, the execution ends; otherwise, based on p A new sequence number is determined in the first ordered list, and the cycle is repeated.
[0077] In some optional embodiments of the present application, the sequence number that belongs to the second ordered list but not to the first ordered list and the component corresponding to the sequence number in the updated gradient are written into the third ordered list, including: for each sequence number in the second ordered list, if the sequence number is not in the first ordered list, the sequence number is written to the end of the third ordered list, and then the component corresponding to the sequence number in the updated gradient is written to the end of the third ordered list.
[0078] The present application also provides a federated learning device based on communication data compression, which is applied to a server node, and the device includes: a data sending module, which is used to send model parameters and a first ordered list to a client node, and the model parameters are used to update the local model of the client node; the first ordered list includes the serial numbers of several components; a data receiving module, which is used to receive a third ordered list uploaded by the client node; a gradient parsing module, which is used to generate an update gradient corresponding to the client node according to the serial number and component in the third ordered list; and a model updating module, which is used to aggregate the generated update gradients to generate a global model update gradient, and update the global model in combination with the update momentum and model parameters.
[0079] In some optional implementations of the present application, sending the model parameters and the first ordered list to the client node includes: after sending the model parameters to the client node, generating the first ordered list and sending the first ordered list to the client node.
[0080] In some optional embodiments of the present application, an update gradient corresponding to the client node is generated according to the sequence number and components in the third ordered list, including: processing the elements in the third ordered list in sequence as follows: if the element is a sequence number and the sequence number is in the first ordered list, processing all components between the element and the next sequence number, and copying all the components in sequence to the update gradient; the positions where all components are copied to the update gradient are determined based on the sequence number and the required number of sequence numbers immediately following the sequence number in the first ordered list; if the element is a sequence number and the sequence number is not in the first ordered list, processing the next component of the element, and copying the next component to the update gradient; the position where the component is copied to the update gradient is determined based on the sequence number; if the element is a component and is the first element in the third ordered list, copying the element as the first component of the update gradient.
[0081] In some optional embodiments of the present application, the generated update gradients are aggregated to generate a global model update gradient, including: determining a weight value based on the local sample data of the client node corresponding to each update gradient; and obtaining a global model update gradient by performing a weighted average calculation based on the update gradient and the corresponding weight value.
[0082] In some optional embodiments of the present application, the global model is updated in combination with the update momentum and model parameters, including: obtaining the update momentum of this round by performing a weighted sum calculation on the update momentum of the previous round and the update gradient of the global model; calculating a correction amount based on the update momentum of this round and the learning rate, correcting the model parameters with the correction amount, and then updating the global model.
[0083] In some optional embodiments of the present application, the method is configured to be executed in a loop when the following preset conditions are met; the preset conditions include: after updating the global model in combination with updating the momentum and model parameters, determining that the current state of the federated learning system does not meet the training stop conditions of the federated learning.
[0084] The specific definition of each functional module in the above-mentioned federated learning device based on communication data compression can be found in the above-mentioned definition of the federated learning method based on communication data compression, which will not be repeated here. Each module in the above-mentioned system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory in the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It also achieves the advantages of reducing the amount of data transmitted over the network and the network communication delay in the federated learning scenario, and improving the training efficiency of the federated learning model.
[0085] In some embodiments of the present invention, an electronic device is also provided, comprising: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the aforementioned federated learning method based on communication data compression. Its internal structure diagram can be as follows Figure 5 shown. Figure 5 The internal structure diagram of an electronic device according to an embodiment of the present invention is schematically shown. The electronic device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The network interface A02 of the electronic device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a federated learning method based on communication data compression is implemented. Based on the application of the executed federated learning method based on communication data compression to a client node or a server node, the electronic device is embodied as one of the client node or the server node.
[0086] Those skilled in the art will understand that Figure 5The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0087] In one embodiment of the present invention, a machine-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the aforementioned federated learning method based on communication data compression.
[0088] In one embodiment of the present invention, a computer program product is provided, including a computer program, which implements the aforementioned federated learning method based on communication data compression when executed by a processor.
[0089] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0091] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0093] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0094] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0095] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0096] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0097] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A federated learning method based on communication data compression, characterized in that: Applied to a client node, the method comprises: Obtain model parameters and a first ordered list, and update the local model according to the model parameters; the first ordered list includes serial numbers of several components; After training the updated local model with the local training data set, an updated gradient is obtained; the serial numbers of several components with the largest absolute values in the updated gradient are written into a second ordered list; Calculate the common sequence numbers that belong to both the first ordered list and the second ordered list, write the components corresponding to the common sequence numbers in the update gradient into a third ordered list; write the common sequence numbers that are adjacent to the non-common sequence numbers in the first ordered list into the third ordered list; Writing the sequence number that belongs to the second ordered list but does not belong to the first ordered list and the component corresponding to the update gradient of the sequence number into the third ordered list, including: for each sequence number in the second ordered list, if the sequence number is not in the first ordered list, writing the sequence number to the end of the third ordered list, and then writing the component corresponding to the update gradient of the sequence number to the end of the third ordered list; Upload the resulting third ordered list.
2. The method according to claim 1, characterized in that: The number of components corresponding to the serial numbers in the first ordered list and the second ordered list are both .
3. The method according to claim 1 or 2, characterized in that: The first ordered list includes several components whose serial numbers are the largest absolute values in the update momentum of the global model and are arranged in descending order of absolute value. The sequence number of the component.
4. The method according to claim 1, characterized in that: If the sequence number of the component in the updated gradient and the component are both written into the third ordered list, the position of the sequence number of the component is the previous position of the component.
5. The method according to claim 1, characterized in that The method comprises a model initialization step; the model initialization step comprises: setting initialization parameters and setting counting variables.
6. The method according to claim 1, characterized in that Get model parameters, including: The server node randomly selects a number of client nodes from the entire set of all client nodes in the federated learning system; The model parameters sent by the server node to the selected client node are received.
7. The method according to claim 1, characterized in that Calculating common sequence numbers belonging to both the first ordered list and the second ordered list, writing components corresponding to the common sequence numbers in the update gradient into a third ordered list; and writing common sequence numbers in the first ordered list that are located after non-common sequence numbers into the third ordered list, comprising: Initialize loop control parameters p 1. Position control parameters q is 1 and the third ordered list is empty; From the first ordered list p Start with a serial number, If the sequence number is in the second ordered list and , put the component corresponding to the serial number in the update gradient into the first q Locations, p and q Add one to each; If the sequence number is in the second ordered list and is based on the first ordered list -1 sequence number is also in the second ordered list, then the component corresponding to the sequence number in the update gradient is placed in the first q Locations, p and q Add one to each; If the sequence number is in the second ordered list, but based on the first ordered list -1 sequence number is not in the second ordered list, then the sequence number and the component corresponding to the sequence number in the update gradient are placed in the first q and q +1 position, p Plus one, q plus two; If the sequence number is not in the second ordered list, p plus one; when If it is greater than the total number of sequence numbers in the first ordered list, the execution ends; otherwise, based on p A new sequence number is determined in the first ordered list, and the cycle is repeated.
8. A federated learning method based on communication data compression, characterized in that: Applied to a server node, the method comprises: Sending model parameters and a first ordered list to a client node, wherein the model parameters are used to update a local model of the client node; the first ordered list includes serial numbers of several components, and the client is configured to execute the federated learning method based on communication data compression as described in any one of claims 1 to 7; receiving a third ordered list uploaded by a client node; Generate an update gradient corresponding to the client node according to the sequence number and component in the third ordered list; The generated update gradients are aggregated to generate global model update gradients, and the global model is updated by combining the update momentum and model parameters.
9. The method according to claim 8, characterized in that Sending the model parameters and the first ordered list to the client node includes: after sending the model parameters to the client node, generating the first ordered list and sending the first ordered list to the client node.
10. The method according to claim 8, characterized in that Generating an update gradient corresponding to the client node according to the sequence number and the component in the third ordered list, comprising: The elements in the third ordered list are processed in sequence as follows: If the element is a sequence number and the sequence number is in the first ordered list, all components between the element and the next sequence number are processed and all the components are copied to the updated gradient in sequence; the positions where all the components are copied to the updated gradient are determined based on the sequence number and the required number of sequence numbers immediately following the sequence number in the first ordered list; If the element is a sequence number and the sequence number is not in the first ordered list, processing the next component of the element and copying the next component to the updated gradient; the position where the component is copied to the updated gradient is determined based on the sequence number; If the element is a component and is the first element in the third ordered list, copy the element as the first component of the updated gradient.
11. The method according to claim 8, characterized in that The generated update gradients are aggregated to generate the global model update gradients, including: Determine a weight value based on local sample data of the client node corresponding to each updated gradient; The global model update gradient is obtained by weighted average calculation based on the update gradient and the corresponding weight value.
12. The method according to claim 11, characterized in that The global model is updated by combining the updated momentum and model parameters, including: The update momentum of the previous round and the global model update gradient are weighted and calculated to get the update momentum of this round; The correction amount is calculated according to the update momentum and learning rate of this round, and the model parameters are corrected with the correction amount, and then the global model is updated.
13. The method according to claim 8, characterized in that The method is configured to be executed cyclically when the following preset conditions are met; The preset condition includes: after updating the global model in combination with the updated momentum and the model parameters, determining that the current state of the federated learning system does not meet the training stop condition of the federated learning.
14. An electronic device, characterized in that: include: at least one processor; a memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the steps of the federated learning method based on communication data compression as described in any one of claims 1 to 7 or any one of claims 8 to 13 by executing the instructions stored in the memory.
15. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the federated learning method based on communication data compression as described in any one of claims 1 to 13 are implemented.
16. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the federated learning method based on communication data compression as described in any one of claims 1 to 13 are implemented.