A Low-Power Federated Learning Method Based on Tensor Neural Networks

Abstract

This invention discloses a low-power federated learning method based on tensor neural networks. A central server represents the parameters of the original deep neural network (DNN) into a tensor neural network (TNN) using tensor decomposition. Each user downloads the global TNN model from the central server, updates the TNN model parameters using their own training data and the gradient descent (SGD) optimization operator, and sends their respective loss values ​​and updated private head parameters to the central server. The central server updates the set shared base parameters and distributes the updated shared base parameters to each user. Each user continues iterative training using the updated shared base parameters until the TNN model converges. This method effectively reduces user computational load, thereby reducing user computational power consumption and transmission power consumption, and thus effectively reducing the power consumption of federated learning (FL).

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a low-power federated learning method based on tensor neural networks. Background Technology

[0002] Federated Learning (FL) is a distributed machine learning method that trains a model across multiple nodes with local data (such as mobile devices, sensors, or edge devices). It constructs a global model based on virtual fused data by exchanging model parameters or intermediate results between nodes and a server, without needing to exchange local node sample data. This achieves a balance between data privacy and shared computation. To address the complexity of training data (large number of training samples and high feature dimensionality), complex neural network models have become a mainstream technique. However, since user nodes participating in FL are typically power-constrained edge mobile devices, the training of complex neural networks and the uploading of model parameters can lead to excessively high user-side computational and transmission power consumption.

[0003] To reduce the power consumption of federated learning (FL), existing technologies primarily rely on compressing model parameters and reducing the frequency of model interactions. The main idea behind model parameter compression is to encode the model parameters uploaded by the user to the server using data compression techniques, ensuring that the number of bits in the encoded data is less than the number of bits in the original unencoded data. This reduces the amount of model parameter information exchanged between the user and the server, thereby lowering transmission power consumption. Common data compression methods can be categorized into lossy and lossless compression. However, most existing methods used for FL model compression are lossy, such as quantization and sparsification. These methods inevitably lead to performance degradation in the FL model. Furthermore, existing methods do not consider the computational power consumption of user devices, leaving room for optimization. Therefore, how to reduce user transmission power consumption while maintaining FL model performance, and simultaneously optimize the computational power consumption of FL user devices, is a pressing issue that current technologies must address. Summary of the Invention

[0004] The purpose of this invention is to provide a low-power federated learning method based on tensor neural networks. This method can effectively reduce the amount of computation for users, thereby reducing user computational power consumption and transmission power consumption, and thus effectively reducing the power consumption of federated learning (FL).

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A low-power federated learning method based on tensor neural networks, the method comprising:

[0007] Step 1: First, the central server represents the parameters of the original deep neural network (DNN) as a tensor neural network (TNN) based on the tensor decomposition method. The tensor neural network (TNN) is then used to initialize the local models of each user. Specifically, the last two layers of the tensor neural network (TNN) are set as the user's private header parameters, and the remaining front layer parameters are the user's shared base layer parameters.

[0008] Step 2: Each user downloads the global TNN model from the central server, updates the parameters of the tensor neural network TNN model using their own training data and the gradient descent optimization operator SGD, and sends their respective loss values ​​and updated private head parameters to the central server.

[0009] Step 3: The central server updates the set shared base parameters based on the loss values ​​received from each user and the updated private header parameters, and then sends the updated shared base parameters to each user.

[0010] Step 4: Each user continues iterative training using the updated shared base parameters until the TNN model converges.

[0011] As can be seen from the technical solution provided by the present invention, the above method can effectively reduce the amount of user computation, thereby reducing user computation power consumption and transmission power consumption, and thus effectively reducing the power consumption of federated learning (FL). Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic diagram of the low-power federated learning method based on tensor neural networks provided in an embodiment of the present invention;

[0014] Figure 2 This is a schematic diagram showing the average test accuracy and number of training rounds in the examples given in this invention.

[0015] Figure 3 This diagram illustrates the amount of model parameters that need to be transmitted and calculated, as well as the average test accuracy, for different methods described in the examples of this invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, and do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0017] like Figure 1 The diagram shows a flowchart of a low-power federated learning method based on tensor neural networks provided in an embodiment of the present invention. The method includes:

[0018] Step 1: First, the central server uses the tensor decomposition method to represent the parameters of the original deep neural network (DNN) as a tensor neural network (TNN), and then uses the tensor neural network (TNN) to initialize the local models of each user.

[0019] In this case, the last two layers of the Tensor Neural Network (TNN) are set to be the user's private header parameters, while the remaining first layer parameters are the user's shared base layer parameters.

[0020] In this step, taking Tensor Ring (TR) decomposition as an example, the central server uses layer-by-layer tensor decomposition to decompose the original deep neural network (DNN) matrix. Expressed in the form of a tensor neural network (TNN) according to formula (1):

[0021]

[0022] Where i1, i2, ..., i n The index of the DNN parameter matrix W in high-dimensional tensor form; r n-1 i n r n Represents the three-dimensional tensor kernel The index; {R0, R1, ..., R n} represents the rank of the tensor decomposition; It is a three-dimensional tensor kernel, and R0 = R n And set the parameters of the last two layers of the Tensor Neural Network (TNN) to the user's private header parameters. The remaining front-layer parameters are shared base-layer parameters for users.

[0023] From formula (1), we can deduce that: assuming each dimension I... i =d, the rank of the tensor decomposition is set to r, using the symbol... To represent complexity, the original deep neural network (DNN) has the following number of parameters: The Tensor Neural Network (TNN) contains the following number of parameters: Therefore, by controlling the rank of tensor decomposition, the number of model parameters can be significantly reduced, thereby reducing user transmission power consumption and computational power consumption generated during model training.

[0024] Tensor decomposition, mentioned in this step, refers to decomposing a high-dimensional tensor into multiple low-dimensional sub-tensors to reduce the dimensionality and complexity of the data while preserving the important features of the original data. Common tensor decomposition methods include CP decomposition, Tucker decomposition, TT decomposition, and TR decomposition, etc.

[0025] Tensor Neural Networks (TNNs) are a type of neural network model that uses tensor decomposition to represent the input, output, and hidden layers of a neural network in tensor form. Compared to traditional neural network models, TNNs are better suited for processing high-dimensional and complex structured data.

[0026] Step 2: Each user downloads the global TNN model from the central server, updates the parameters of the tensor neural network TNN model using their own training data and the gradient descent optimization operator SGD (Stochastic Gradient Descent), and sends their respective loss values ​​and updated private head parameters to the central server.

[0027] In this step, during the update of TNN model parameters, each user's TNN model parameters are divided into two parts, including private header parameters. Update according to formula (2):

[0028]

[0029] Here, Indicates private header parameters The k-th tensor kernel in the model; η represents the learning step size; Let be the local loss function value for the i-th user, representing the difference between the model's predicted output and the sample label, which can often be calculated using cross-entropy;

[0030] After the update is complete, each user will and updated private header parameters Upload to the central server.

[0031] Step 3: The central server updates the set shared base parameters based on the loss values ​​received from each user and the updated private header parameters, and then sends the updated shared base parameters to each user.

[0032] In this step, the central server determines the loss function values ​​received from each user. and updated private header parameters For the set shared base parameters Update according to the following formula (3):

[0033]

[0034] Here, Indicates shared base level parameters The k-th tensor kernel in the model; η represents the learning step size; This represents a shared loss value, consisting of the loss values ​​of N users. The average value can be obtained;

[0035] After the update is complete, the server will update the shared base parameters. Distribute to each user.

[0036] The following explains how formulas (2) and (3) utilize the loss function to adjust the TNN tensor kernel. The principle behind parameter updates will be explained:

[0037] Assuming the input of any hidden layer in the original deep neural network (DNN) is h1, the parameter is W, and the bias vector is b, then the output h2 of the hidden layer is expressed as shown in formula (4):

[0038] h2=σ(W T h1+b) (4)

[0039] Here, σ represents the nonlinear activation function of the neural network, such as the ReLU activation function and the softmax activation function; the parameters W of the original deep neural network (DNN) are represented as a series of sub-tensor kernels after tensor decomposition. make The cost function of the original deep neural network (DNN) for samples (x, y) can be represented as follows: The objective to be optimized by each user can be equivalently expressed as follows:

[0040]

[0041] During the local training process, each user can utilize the gradient descent optimization operator SGD to directly optimize a series of sub-tensor kernels of the tensor neural network (TNN). The parameters are updated according to the specific rules shown in formulas (6) and (7):

[0042]

[0043]

[0044] Where, o2 = W T h1+b represents the output of the hidden layer with h1 as input before activation by the nonlinear function σ.

[0045] Step 4: Each user continues iterative training using the updated shared base parameters until the TNN model converges.

[0046] In this step, the model is considered to have converged when the change in the loss function value of the TNN model is less than a set threshold after multiple consecutive training epochs, or when the model training reaches the set maximum number of training epochs.

[0047] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0048] The method described in this embodiment of the invention will be explained in detail below with a specific example. This example applies the method to a classification task of handwritten digit recognition (MNIST dataset). Specifically, two users and one central server are set up. To simulate the common Non-IID scenario in federated learning (FL), one user's local training and test sets contain only odd numbers (1, 3, 5, 7, 9), while the other user's local training and test sets contain only even numbers (0, 2, 4, 6, 8). To simulate the scenario of insufficient user training data, each user only extracts 10% of the training data for model training. During the testing phase, each user tests the model based on their own test set (which is not repeated with the training set). The original DNN model uses an MLP with four fully connected layers, and a TR-Rank of 10 is selected to perform TR decomposition on the DNN to obtain a Tensor Neural Network (TNN). The specific information of the model is shown in Table 1 below. During training, the learning step size η = 0.001, batch size = 100, and the maximum number of iterations is 100 epochs.

[0049] Table 1 MLP Model Structure

[0050]

[0051]

[0052] The method described in this embodiment (FL-TNN) is compared with other typical FedAvg methods (FL-DNN), single-user DNN training (STL-DNN), and single-user TNN training (STL-TNN). The average test accuracy of each method and the number of model parameters required for each user to achieve a specific accuracy are recorded. The average test accuracy is used to evaluate the model's performance, such as... Figure 2 The diagram shows the average test accuracy and number of training rounds for this example; the number of model parameters required for transmission and computation per user is used to measure the transmission and computation power consumption on the user side, such as... Figure 3The diagram illustrates the number of model parameters required for transmission and computation, as well as the average test accuracy, for different methods in this example. From... Figure 2 and 3 The results show that, compared with the existing methods, the method described in this embodiment of the invention achieves higher model accuracy while having lower energy consumption requirements on the FL user side.

[0053] In summary, the method described in this embodiment of the invention uses a TNN to represent the original DNN, and the calculation process for updating the parameters of the TNN is directly based on the sub-tensor kernel of the TNN, which can effectively reduce the amount of computation for the user, thereby reducing computational power consumption. Secondly, this method splits the TNN model parameters into private header parameters and shared base parameters. The user only needs to update the private header parameters, which can also effectively reduce the user's computational power consumption. Finally, after the user completes the local parameter update, they only need to upload their own loss value and private header parameters (a part of the complete model) to the server. Compared with uploading the complete model parameters, this can effectively reduce the amount of data transmitted, thereby reducing transmission power consumption. Through the above three aspects, the present invention can effectively reduce FL power consumption.

[0054] In addition, the method described in this embodiment of the invention combines a private header and a shared base layer to construct a personalized model for each user, which can improve the model performance of each user compared to a unified global model.

[0055] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0056] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A low-power federated learning method based on tensor neural networks, characterized in that, The method includes: Step 1: First, the central server represents the parameters of the original deep neural network (DNN) as a tensor neural network (TNN) based on the tensor decomposition method. The tensor neural network (TNN) is then used to initialize the local models of each user. Specifically, the last two layers of the tensor neural network (TNN) are set as the user's private header parameters, and the remaining front layer parameters are the user's shared base layer parameters. Step 2: Each user downloads the global TNN model from the central server, updates the parameters of the tensor neural network TNN model using their own training data and the gradient descent optimization operator SGD, and sends their respective loss values ​​and updated private head parameters to the central server. In step 2, during the TNN model parameter update process, each user's TNN model parameters are divided into two parts, including private header parameters. Update according to formula (2): Here, Indicates private header parameters The k-th tensor kernel in the model; η represents the learning step size; Let be the local loss function value for the i-th user, representing the difference between the model's predicted output and the sample label; After the update is complete, each user will and updated private header parameters Upload to the central server; Step 3: The central server updates the set shared base parameters based on the loss values ​​received from each user and the updated private header parameters, and then sends the updated shared base parameters to each user. In step 3, the central server calculates the loss function values ​​received from each user. and updated private header parameters For the set shared base parameters Update according to the following formula (3): Here, Indicates shared base level parameters The k-th tensor kernel in the model; η represents the learning step size; This represents a shared loss value, consisting of the loss values ​​of N users. The average value can be obtained; After the update is complete, the server will share the updated basic parameters. Distribute to each user; Step 4: Each user continues iterative training using the updated shared base parameters until the TNN model converges.

2. The low-power federated learning method based on tensor neural networks according to claim 1, characterized in that, In step 1, the central server uses layer-by-layer tensor decomposition to decompose the original deep neural network (DNN) matrix. Expressed in the form of a tensor neural network (TNN) according to formula (1): Where i1, i2, ..., i n The index of the DNN parameter matrix W in high-dimensional tensor form; r n-1 i n r n Represents the three-dimensional tensor kernel The index; {R0, R1, ..., R n } represents the rank of the tensor decomposition; It is a three-dimensional tensor kernel, and R0 = R n And set the parameters of the last two layers of the Tensor Neural Network (TNN) to the user's private header parameters. The remaining front-layer parameters are shared base-layer parameters for users. From formula (1), we can deduce that: assuming each dimension I... i =d, the rank of the tensor decomposition is set to r, using the symbol... To represent complexity, the original deep neural network (DNN) has the following number of parameters: The Tensor Neural Network (TNN) contains the following number of parameters: Therefore, by controlling the rank of the tensor decomposition, the number of parameters in the model can be significantly reduced.

3. The low-power federated learning method based on tensor neural networks according to claim 1, characterized in that, In step 4, the model is considered to have converged when the change in the loss function value of the TNN model is less than a set threshold after multiple consecutive training epochs or when the model training reaches the set maximum number of training epochs.

Images