Gradient-free efficient longitudinal federated learning method and system based on layer-by-layer optimization

Abstract

The invention discloses a gradient-free efficient longitudinal federated learning method and system based on layer-by-layer optimization. The method comprises the following steps: carrying out initial configuration on a longitudinal federated learning model, obtaining feature data of each participant and setting model parameters; considering the obtained model parameters and the calculation resource limitation of each participant, a Moore-Penrose generalized inverse matrix is adopted to replace the traditional gradient, and the communication overhead is reduced and the model training efficiency is improved through layer-by-layer optimization. The whole training process is executed layer by layer through forward propagation and back propagation until the model reaches a convergence state on all participants. The invention provides the efficient gradient-free longitudinal federated learning method and system, the resource utilization rate and the training efficiency can be remarkably improved, and the method and the system have wide application prospects.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology, and in particular to a gradient-free and efficient vertical federated learning method and system based on layer-by-layer optimization. Background Art

[0002] With the continuous development of machine learning technology, federated learning (FL), as a distributed machine learning framework, has been widely used in recent years. Federated learning achieves effective utilization of data resources by jointly training machine learning models through the collaboration of multiple participants without sharing their local data. Unlike traditional centralized machine learning methods, federated learning can train models on multiple distributed nodes, effectively protecting the privacy of data and promoting the collaborative use of multi-party data. According to the distribution of data among participants, federated learning can be divided into horizontal federated learning (HFL) and vertical federated learning (VFL). Among them, horizontal federated learning is suitable for scenarios with the same data features but different samples, while vertical federated learning is suitable for scenarios where each participant holds the same sample but different feature sets. In the fields of finance, medicine, etc., different institutions can effectively improve the performance of prediction models through vertical federated learning to meet actual business needs, while avoiding privacy and compliance issues caused by direct data exchange.

[0003] In the traditional vertical federated learning framework, the optimization method based on gradient descent (SGD) is widely used to update model parameters. This method calculates local features by each participant and passes the generated embedding vector to the central server, which trains and updates the global model. This gradient-based optimization method has significant limitations in practical applications. With the expansion of data scale and the increase of model complexity, the frequency and intensity of gradient calculations have increased significantly, resulting in a sharp increase in the consumption of computing and communication resources. This frequent gradient exchange not only brings huge computing and communication burdens, but also may cause delays in the training process and degradation of system performance, especially when dealing with complex models. This problem is particularly prominent. In addition, the frequent transmission of gradient information may make the design and implementation of the system more complicated, increasing the difficulty of maintenance and expansion. These problems are particularly prominent in the deployment and practical application of federated learning systems, because the computing and communication capabilities of the participants often vary greatly, further exacerbating this contradiction.

[0004] To alleviate these problems, researchers have proposed gradient-free optimization methods, such as zero-order optimization (ZOO). The ZOO method estimates the gradient by evaluating the change in the value of the objective function at multiple sampling points, avoiding the high cost required for traditional gradient calculations. However, the ZOO method itself also has significant limitations. Since the ZOO method requires multiple evaluations of the objective function, its convergence speed is usually slower than the gradient descent method, and the computational overhead is large, especially when the model complexity is high. These problems are more prominent. Although the ZOO method reduces the need for explicit gradient transmission to a certain extent, it still requires a large number of function evaluations in essence, so the computational complexity has not been fundamentally reduced. In addition, the ZOO method also has certain deficiencies in its adaptability when dealing with different data features and model architectures, making it less effective than expected in practical applications. Therefore, although the ZOO method provides an alternative to the traditional gradient method, its practicality in a wide range of applications still needs further verification and optimization.

[0005] Existing vertical federated learning methods, whether traditional gradient-based optimization methods or innovative methods based on gradient-free learning, face challenges in resource overhead and efficiency in practice. The high consumption of computing and communication resources restricts the speed and efficiency of model training, which in turn affects the actual application effect of federated learning. Frequent gradient exchanges and function evaluations not only increase the complexity of the system, but may also lead to uncertainty in actual operations. Therefore, how to reduce computing and communication overhead and improve training efficiency while ensuring model performance remains an important issue to be solved in the field of vertical federated learning. With the continuous expansion of federated learning applications, developing optimization algorithms that can operate efficiently in a variety of practical scenarios has become a research focus and development direction in this field. Summary of the invention

[0006] Purpose of the invention: In order to overcome the defects and shortcomings in the prior art, the present invention proposes a gradient-free and efficient vertical federated learning method and system based on layer-by-layer optimization, which reduces communication overhead and improves model training efficiency by adopting the Moore-Penrose generalized inverse matrix to replace the traditional gradient calculation, while effectively reducing the communication requirements and computational complexity between the participants.

[0007] In order to achieve the above invention object, the technical solution of the present invention is as follows:

[0008] In the first aspect, a gradient-free and efficient vertical federated learning method based on layer-by-layer optimization comprises the following steps:

[0009] After each participant's local neural network model is initialized, it independently performs forward propagation locally according to the set model parameters and the configured weights and biases of each layer, calculates the intermediate embedding representation of each layer in turn, and transmits the results to the server after calculation;

[0010] After receiving the intermediate embedding representations transmitted by each participant, the server integrates them into a global feature vector, performs forward propagation of the global model, calculates the output results, and initializes the feedback matrix according to the label data. It then performs backpropagation to update the feedback matrix layer by layer until the first layer, and then splits the feedback matrix and sends it back to the corresponding participant.

[0011] After receiving the feedback matrix, each participant performs back propagation to independently update the weights and biases of the local model layer by layer, and sends the updated intermediate embedding representation to the server. Each participant propagates the feedback matrix forward from the last layer. The current layer receives the feedback matrix from the next layer, updates it using pseudo-inverse matrix calculation, calculates the intermediate embedding representation of this layer, and sends the calculation result to the next layer without participating in subsequent calculations.

[0012] The server aggregates the updated intermediate embedding representations of each participant to form a global feature, and continues to perform backpropagation to update the weights and biases of the global model layer by layer. The server uses the pseudo-inverse matrix to calculate layer by layer and adjusts the weights after each layer.

[0013] Repeat the forward propagation and back propagation process, and the server continues to receive and process the update information of each participant until each layer of the local model of the participant and the global model is effectively updated and convergence is achieved.

[0014] Furthermore, after receiving the feedback matrix, each participant updates the weights and biases of each layer through the Moore-Penrose pseudo-inverse matrix of each layer, and uses the feedback matrix to guide the adjustment of weights in each layer; in this process, each participant uses the pseudo-inverse matrix to update the feedback matrix layer by layer, and updates the model parameters layer by layer according to the feedback matrix. The updated weights and biases are applied to the forward propagation of the next layer to generate a new intermediate representation, until the model parameters of all layers are updated, and the updated intermediate representation is sent to the server.

[0015] Furthermore, the calculation method of the server to perform back propagation layer by layer to update the feedback matrix is ​​F s (i-1) = W s + (i)×(F s (i)-b s (i)), where W s + (i) is the pseudo-inverse matrix of the weight matrix of the server layer i, b s (i) is the bias of the layer, Fs (i) is the feedback matrix of this layer.

[0016] Furthermore, on the client side of each participant, the layer-by-layer optimization method specifically includes:

[0017] For the i-th layer, when the layer is a neural network layer, the client first processes the input features, and the client q represents the current input as X q (i) By expanding through concatenation, the bias term is introduced into the input features so that each layer can correctly consider the influence of the bias during the forward propagation process;

[0018] The client sends the intermediate representation after the forward propagation to the server. The server integrates the intermediate embedding representations of all clients into a global feature vector X, inputs it into the global model, and performs forward propagation layer by layer until the final output is generated. The feedback matrix is ​​initialized according to the label data, and the global model is back-propagated. After segmentation, the feedback matrix F adapted to the client is generated. q (i) and transmit the feedback matrix back to each client to guide subsequent back-propagation;

[0019] Back propagation starts from the last layer, and the client updates the parameters of each layer in turn through the feedback matrix. The process is as follows: First, the Moore-Penrose pseudo-inverse matrix Update the feedback matrix F q (i) and update the weight W of the current layer based on the updated feedback matrix q (i) and bias b q (i) Recalculate the intermediate representation X q (i),After the intermediate representation is calculated, it will be used as the input of the next layer;

[0020] If the current layer is an activated layer, the client's feedback matrix F q (i) Apply the corresponding inverse operation of the activation function so that the adjusted feedback information is correctly transmitted during the back-propagation process;

[0021] Until all layers are updated, the client sends the calculated intermediate embedding to the server for aggregation. Further, the update of the feedback matrix on the client is calculated as follows: in is the pseudo-inverse matrix of the weight matrix of the client layer i, b q (i) is the bias of the layer, F q (i) is the feedback matrix of this layer;

[0022] The calculation for parameter updates on the client is in It is the pseudo-inverse matrix of the forward calculation result after the previous layer is updated.

[0023] In a second aspect, a gradient-free efficient vertical federated learning method based on layer-by-layer optimization is provided, which is executed by a client and includes the following steps:

[0024] After initializing the local neural network model, forward propagation is performed independently locally according to the set model parameters and the configured weights and biases of each layer, and the intermediate embedding representation of each layer is calculated in turn. After calculation, the results are transmitted to the server;

[0025] Receive the feedback matrix from the server, perform back propagation to independently update the weights and biases of the local model layer by layer, and send the updated intermediate embedding representation to the server, wherein the back propagation starts from the last layer and propagates the feedback matrix forward. The current layer receives the feedback matrix from the next layer, and after updating it using the pseudo-inverse matrix calculation, calculates the intermediate embedding representation of this layer and sends the calculation result to the next layer without participating in subsequent calculations; the feedback matrix is ​​initialized by the server according to the label data, and after back propagation update, it is sent back to the client after segmentation;

[0026] Repeat the forward propagation and back propagation process until each layer of the local model is effectively updated and convergence is achieved.

[0027] In the third aspect, a gradient-free efficient vertical federated learning method based on layer-by-layer optimization is executed by the server, including the following steps:

[0028] After receiving the intermediate embedding representations transmitted by each participant, it is integrated into a global feature vector, forward propagation of the global model is performed, the output result is calculated, and the feedback matrix is ​​initialized according to the label data. Backward propagation of the global model is performed until the first layer, and the feedback matrix is ​​split and sent back to the corresponding participant;

[0029] Receive the updated intermediate embedding representations of each participant, integrate them to form global features, perform back propagation to update the global model layer by layer, wherein the server uses the pseudo-inverse matrix to calculate layer by layer, and adjusts the weights after each layer; the updated intermediate embedding representation is obtained by each participant performing back propagation to independently update the weights and biases of the local model layer by layer after receiving the feedback matrix, and calculating according to the updated weights and biases, wherein each participant propagates the feedback matrix forward from the last layer, and the current layer receives the feedback matrix from the next layer, and after updating it using the pseudo-inverse matrix calculation, calculates the intermediate embedding representation of this layer and sends the calculation result to the next layer, and does not participate in subsequent calculations;

[0030] Repeat the forward propagation and back propagation process until each layer of the global model is effectively updated and convergence is achieved.

[0031] In a fourth aspect, the present invention also provides a computer device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the gradient-free efficient vertical federated learning method based on layer-by-layer optimization as described in the second aspect or the third aspect are implemented.

[0032] In a fifth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the gradient-free efficient vertical federated learning method based on layer-by-layer optimization as described in the second aspect or the third aspect.

[0033] In the sixth aspect, the present invention also provides a gradient-free and efficient vertical federated learning system based on layer-by-layer optimization, comprising a server and several participant devices, wherein the participant devices and the server complete their respective model optimization based on the gradient-free and efficient vertical federated learning method based on layer-by-layer optimization described in the first aspect.

[0034] Beneficial effects: The present invention proposes an efficient vertical federated learning training method based on layer-by-layer optimization and gradient-free methods. By using the Moore-Penrose pseudo-inverse matrix to back-propagate the feedback matrix and model update, the present invention significantly reduces the communication overhead caused by global gradient exchange in traditional methods. At the same time, through the layer-by-layer optimization strategy, the present invention can effectively reduce the computational burden in model training and reduce the resource consumption required for multiple rounds of iterations. Compared with the prior art, the present invention greatly improves the overall efficiency and resource utilization of the system while ensuring model accuracy and convergence speed, and is particularly suitable for distributed computing environments with strict resource constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of a gradient-free efficient vertical federated learning method based on layer-by-layer optimization according to the present invention;

[0036] Figure 2 is a schematic diagram of a client local model training process according to the present invention;

[0037] Figure 3 is a schematic diagram of a server global model training process according to the present invention;

[0038] Figure 4 Schematic diagram of complete model training according to the present invention. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0040] The present invention proposes a gradient-free and efficient vertical federated learning method based on layer-by-layer optimization. Different from the traditional gradient descent method, the present invention adopts the Moore-Penrose pseudo-inverse matrix for parameter update, avoiding frequent gradient exchange, thereby significantly reducing communication and computing costs. Through the layer-by-layer optimization strategy, the client performs forward propagation calculation of intermediate embedding locally and sends the result to the server. The server then integrates the embedding of each client for global forward propagation and generates a feedback matrix to guide the client's back propagation and model update. The client and server repeatedly iterate this process until the model converges, achieving efficient and privacy-preserving federated learning.

[0041] In order to have a clearer understanding of the technical solution and beneficial effects of the present invention, the principle of the present invention is first explained. In the vertical federated learning system, each participant calculates local features separately and transmits the generated embedding vectors to the central server, which trains and updates the global model. The pursuit of efficient computing and communication resources can be modeled as the following optimization problem:

[0042]

[0043] Where N represents the number of local data samples of the participant, C q (m) represents the computational overhead of participant q at layer m, which is mainly determined by the neural network structure, feature dimension, and model parameter scale of this layer (where q ranges from 1 to Q, indicating that there are a total of Q clients participating). q (m) represents the communication demand of this layer, which refers to the amount of data exchanged between the participants and the server at this layer, B is the available bandwidth, and τ q is the communication time allocated to party q.

[0044] Through this optimization model, computing and communication resources can be allocated during the layer-by-layer training process, reducing the overall system resource consumption. q (m) is associated with the forward and backward propagation operations of each layer, while the communication overhead It indicates the bandwidth time required for data transmission over the network.

[0045] According to an embodiment of the present invention, in the layer-by-layer optimization method, each participant independently performs forward propagation locally to calculate the intermediate representation of each layer. q(m) is affected by the neural network structure, feature dimension and model parameter scale of this layer. In order to ensure computational efficiency, the present invention combines the objective function in the optimization problem, optimizes the use of computing resources in the forward propagation process of each layer, and avoids unnecessary computing consumption. The server receives the intermediate representation transmitted by each participant and integrates it into a global feature vector, and calculates the feedback matrix. In the reverse propagation process, each participant updates the parameters layer by layer according to the feedback matrix, and the update of each layer is accurately adjusted by the information in the feedback matrix to ensure that the communication requirement T q (m) and the available bandwidth B, thereby minimizing the communication overhead.

[0046] In the embodiment of the present invention, the a9a dataset is used. The client model is a three-layer multi-layer perceptron (MLP), each layer contains 64, 32, and 2 neurons, and the server side contains an output layer, and the activation function is Softmax.

[0047] Reference Figure 1 , the method of the present invention specifically comprises the following steps:

[0048] Step (1), model initialization.

[0049] In the process of the present invention, during the model initialization process, each client (i.e., the participating device) and the server initialize their neural network models to ensure that the model can run efficiently in a multi-party collaborative environment. Each client q needs to initialize the local neural network model N q The client model consists of a multi-layer neural network, and the parameters of each layer include the weight matrix W q,i and the bias vector b q,i .W q,i represents the weight matrix of the i-th layer of the q-th client, which is used to connect the neurons of the previous layer to the current layer; and b q,i is the corresponding bias, which is used to adjust the output of each layer. During the initialization process, the weights and biases of the client model are usually initialized randomly or with zero values. Commonly used methods include Xavier initialization or He initialization, which can automatically adjust the distribution of initial weights according to the number of input and output neurons of the layer, so that the output of each layer remains within a reasonable numerical range, thereby effectively avoiding the occurrence of problems such as gradient vanishing or gradient exploding.

[0050] While initializing the model parameters, each client will also initialize the local data x qA series of preprocessing operations are performed, such as standardization, normalization, and missing value imputation. These data preprocessing steps are designed to improve the quality and consistency of the data, making the calculation results of each layer more stable and reliable during the forward propagation process.

[0051] The server initializes the global model N s , which is used to integrate the embedding representations from each client and perform global forward propagation and optimization. The weights and biases of the server model are represented by W s,i and b s,i Indicates that W s,i represents the weight matrix of the i-th layer, which is responsible for processing the embedded representation transmitted by the client; b s,i The server's initialization strategy is similar to that of the client, using random or zero initialization to ensure that the output of each layer is within a reasonable range.

[0052] In step (2), the client performs forward calculation based on local data and sends the calculated intermediate embedding to the server.

[0053] After the model is initialized, each client uses its local neural network model N q Start forward propagation calculation. Figure 2 As shown in ①③⑤, local forward propagation refers to the client using the preprocessed input data x q , calculate the embedding representation of each layer layer by layer, that is, feature extraction. The core of the forward propagation is that each layer of the neural network will pass the input data through the weight W q,i and bias b q,i The linear transformation is then processed through a nonlinear activation function (such as ReLU, Sigmoid, etc.) to obtain the output of the next layer. This process is progressive until the last layer, and finally generates the intermediate embedding representation h q,L , here we assume that the client local model has a total of L layers. This embedded representation contains the high-dimensional representation information of the client local features. The forward propagation calculation formula is:

[0054] h q,i =σ(W q,i ·h q,i-1 +b q,i )

[0055] Among them, h q,i represents the output of the i-th layer, σ is the activation function, and the initial input is the client's local data x qThis layer-by-layer calculation method can fully utilize the feature extraction capabilities of each layer and gradually transform the original data into high-level feature representations. For each client, this embedding calculation is completed completely locally, without the need to exchange original data with the server or other clients, thus effectively protecting data privacy.

[0056] In step (3), the server aggregates the intermediate embeddings sent by the client and uses them as input to complete the forward calculation of the global model.

[0057] like Figure 3 As shown in Figure 1, the server is responsible for aggregating the embeddings sent by each client into a global feature, and further forward propagating in the global model to generate preliminary prediction results. The server first receives the intermediate embeddings from each client, which represent the high-dimensional features of the client after multiple layers of processing. The server splices these intermediate embeddings into an aggregate embedding (global feature vector X), which contains the data feature information of all participants.

[0058] After completing feature aggregation, the server inputs the aggregated embedding into the global model N s In the server model, global forward propagation is performed. Similar to the client's forward propagation, the server model performs linear transformation and nonlinear activation on the input features layer by layer, and finally generates a predicted output y. The difference between the server's global model and the client model is that it not only processes data from a single source, but also integrates data from multiple sources. Therefore, the design of the global model is often more complex and requires the ability to process high-dimensional data and multi-feature fusion.

[0059] In step (4), the server initializes the label data as the feedback matrix and starts the global model back propagation until the first layer, splits the feedback matrix and sends it back to the corresponding client.

[0060] The real label data exists on the server, so the server directly initializes the label data as the feedback matrix F s , that is, F s =Y, and then start to perform the back propagation of the global model, advancing from the last layer to the first layer.

[0061] According to an embodiment of the present invention, the server completes the feedback matrix F s After initialization, back propagation starts from the last layer. The server updates the feedback matrix at each layer and transmits it to the previous layer until the first layer obtains the feedback matrix of the layer. The specific calculation process is F s (i-1) = W s + (i)×(F s (i)-b s (i)), where W s+ (i) is the pseudo-inverse matrix of the weight matrix of the current layer of the server, b s (i) is the bias of this layer. Note that the server does not perform parameter updates here. The server's parameter updates are performed after the client completes the update.

[0062] like Figure 3 As shown, the server divides the feedback matrix into parts adapted to each client model level according to the intermediate embedding vector dimensions transmitted by each client in step (3), forming the corresponding feedback matrix F q After the back propagation is completed, the server will divide these feedback matrices F q Distribute to each client to guide the client to perform layer-by-layer backpropagation and parameter update of the local model.

[0063] In step (5), the client performs back propagation to update the local model layer by layer according to the received feedback matrix, and sends the updated intermediate embedding to the server.

[0064] The client receives the feedback matrix F from the server q After that, the feedback matrix will be used for back propagation. Back propagation starts from the last layer, and the client updates the parameters of each layer in turn through the feedback matrix. The specific calculation of the client back propagation feedback matrix is The calculation for parameter update is in is the pseudo-inverse matrix of the weight matrix of the current layer of the client, b q (i) is the bias of the layer, It is the pseudo-inverse matrix of the forward calculation result after the previous layer is updated. The purpose of is to avoid traditional gradient calculations and to perform local optimization independently at each layer. This update method reduces the overhead of multiple rounds of global gradient exchanges and only adjusts the weights through the local feedback matrix. The client is based on the updated feedback matrix F q (i) Update the weight W of the current layer q (i) and bias b q (i) and recalculate the intermediate representation X q (i) After the intermediate representation is calculated, it will be used as the input of the next layer for forward propagation. If the layer is an activation layer, the client will calculate the feedback matrix F q (i) Apply the corresponding inverse operation of the activation function so that the adjusted feedback information is correctly transmitted during the back-propagation process. Until all layers have completed the update, the client sends the calculated intermediate embedding to the server for aggregation.

[0065] Figure 2Taking the client's 3-layer fully connected neural network model as an example, during the entire training process, only one network layer is updated each time through back propagation. Figure 2 The ②④⑥ in corresponds to three rounds of back propagation process, updating network layers 1, 2, and 3 respectively. Figure 2 As shown in back propagation ②, the client propagates the feedback matrix forward from the last layer until network layer 2 sends the feedback matrix to network layer 1, and then network layer 1 uses the feedback matrix to update. After the update, the latest forward result of the model is sent to network layer 2, and network layer 1 no longer participates in subsequent calculations. Figure 2 Back propagation ④ in the above process corresponds to the second round of back propagation. Similarly, the client propagates the feedback matrix forward from the last layer until network layer 3 sends the feedback matrix to network layer 2. Then network layer 2 uses the feedback matrix to update. After the update, the latest forward result of the model is sent to network layer 3. Network layer 2 no longer participates in subsequent calculations. Back propagation ⑥ continues the similar update process of ②④ until network layer 3 sends the latest forward result of the model to the server. Then network layer 3 no longer participates in subsequent calculations. This layer-by-layer, step-by-step adjustment method makes the update of each layer independent of other layers, so that the entire model can be fine-tuned at the local level. After the update is completed, the local model of the client recalculates the new intermediate embedding in the next round of iterations.

[0066] In step (6), the server aggregates the updated intermediate embeddings and performs back-propagation to update the global model layer by layer.

[0067] The server receives the updated embedding representation X from the client. q After that, these representations are concatenated to form the global feature X, and then back-propagation updates are continued layer by layer. The server will update the global model N based on the integrated global feature X. s The server will continue to use the updated embedding features to optimize the model parameters during the layer-by-layer update process. Similar to the client update, the server will use the pseudo-inverse matrix to calculate the feedback layer by layer and adjust the weights after each layer.

[0068] Step (7), make predictions based on the updated global model.

[0069] After completing all layer-by-layer optimization and feedback distribution, the server uses the optimized global model to make the final prediction. Global prediction is a forward propagation operation performed by the server based on the latest integrated and optimized global features, generating the final prediction result through the output layer. The complete interactive and iterative process can be found in Figure 4 , multiple rounds of calculation and communication of intermediate embedding and feedback matrices are performed between the client and the server, thereby achieving optimization of the global model.

[0070] In order to verify the effectiveness of the proposed gradient-free and efficient vertical federated learning method based on layer-by-layer optimization, we conducted experiments on the a9a dataset and compared it with other influential optimization methods in the field. The results are shown in Table 1.

[0071] Table 1a Comparison of test accuracy and communication cost (MB) on the 9a dataset

[0072]

[0073] By comparison, it can be seen that the present invention significantly reduces the computation and communication costs. Experimental results show that the present invention reduces the communication cost by up to 94% and improves the accuracy by up to 11.5%.

[0074] The present invention also provides a gradient-free and efficient vertical federated learning system based on layer-by-layer optimization, including a server and several participant devices, wherein the participant devices and the server complete their respective model optimization based on the above-mentioned gradient-free and efficient vertical federated learning method based on layer-by-layer optimization.

[0075] From the perspective of each participant, the present invention also provides a gradient-free and efficient vertical federated learning method based on layer-by-layer optimization, which is executed by the client and includes the following steps:

[0076] After initializing the local neural network model, forward propagation is performed independently locally according to the set model parameters and the configured weights and biases of each layer, and the intermediate embedding representation of each layer is calculated in turn. After calculation, the results are transmitted to the server;

[0077] Receive the feedback matrix from the server, perform back propagation to independently update the weights and biases of the local model layer by layer, and send the updated intermediate embedding representation to the server, wherein the back propagation starts from the last layer and propagates the feedback matrix forward. The current layer receives the feedback matrix from the next layer, and after updating it using the pseudo-inverse matrix calculation, calculates the intermediate embedding representation of this layer and sends the calculation result to the next layer without participating in subsequent calculations; the feedback matrix is ​​initialized by the server according to the label data, and after back propagation update, it is sent back to the client after segmentation;

[0078] Repeat the forward propagation and back propagation process until each layer of the local model is effectively updated and convergence is achieved.

[0079] From the perspective of the server, the present invention also provides a gradient-free and efficient vertical federated learning method based on layer-by-layer optimization, which is executed by the server and includes the following steps:

[0080] After receiving the intermediate embedding representations transmitted by each participant, it is integrated into a global feature vector, forward propagation of the global model is performed, the output result is calculated, and the feedback matrix is ​​initialized according to the label data. Backward propagation of the global model is performed until the first layer, and the feedback matrix is ​​split and sent back to the corresponding participant;

[0081] Receive the updated intermediate embedding representations of each participant, integrate them to form global features, perform back propagation to update the global model layer by layer, wherein the server uses the pseudo-inverse matrix to calculate layer by layer, and adjusts the weights after each layer; the updated intermediate embedding representation is obtained by each participant performing back propagation to independently update the weights and biases of the local model layer by layer after receiving the feedback matrix, and calculating according to the updated weights and biases, wherein each participant propagates the feedback matrix forward from the last layer, and the current layer receives the feedback matrix from the next layer, and after updating it using the pseudo-inverse matrix calculation, calculates the intermediate embedding representation of this layer and sends the calculation result to the next layer, and does not participate in subsequent calculations;

[0082] Repeat the forward propagation and back propagation process until each layer of the global model is effectively updated and convergence is achieved.

[0083] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the gradient-free efficient vertical federated learning method based on layer-by-layer optimization executed by the client or by the server as described above are implemented.

[0084] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the gradient-free efficient vertical federated learning method based on layer-by-layer optimization, which is executed by a client or a server as described above.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A gradient-free and efficient vertical federated learning method based on layer-by-layer optimization, characterized in that: The following steps are involved: After each participant's local neural network model is initialized, it independently performs forward propagation locally according to the set model parameters and the configured weights and biases of each layer, calculates the intermediate embedding representation of each layer in turn, and transmits the results to the server after calculation; After receiving the intermediate embedding representations transmitted by each participant, the server integrates them into a global feature vector, performs forward propagation of the global model, calculates the output results, and initializes the feedback matrix according to the label data. It then performs backpropagation to update the feedback matrix layer by layer until the first layer, and then splits the feedback matrix and sends it back to the corresponding participant. After receiving the feedback matrix, each participant performs back propagation to independently update the weights and biases of the local model layer by layer, and sends the updated intermediate embedding representation to the server. Each participant propagates the feedback matrix forward from the last layer. The current layer receives the feedback matrix from the next layer, updates it using pseudo-inverse matrix calculation, calculates the intermediate embedding representation of this layer, and sends the calculation result to the next layer without participating in subsequent calculations. The server aggregates the updated intermediate embedding representations of each participant to form a global feature, and continues to perform backpropagation to update the weights and biases of the global model layer by layer. The server uses the pseudo-inverse matrix to calculate layer by layer and adjusts the weights after each layer. Repeat the forward propagation and back propagation process, and the server continues to receive and process the update information of each participant until each layer of the local model of the participant and the global model is effectively updated and convergence is achieved.

2. The method according to claim 1, characterized in that After receiving the feedback matrix, each participant updates the weights and biases of each layer through the Moore-Penrose pseudo-inverse matrix of each layer, and uses the feedback matrix to guide the adjustment of weights in each layer; in this process, each participant uses the pseudo-inverse matrix to update the feedback matrix layer by layer, and updates the model parameters layer by layer according to the feedback matrix. The updated weights and biases are applied to the forward propagation of the next layer to generate a new intermediate representation, until the model parameters of all layers are updated, and the updated intermediate representation is sent to the server.

3. The method according to claim 1, characterized in that The calculation method of the server to perform back propagation and update the feedback matrix layer by layer is F s (i-1) = W s + (i)×(F s (i)-b s (i)), where W s + (i) is the pseudo-inverse matrix of the weight matrix of the server layer i, b s (i) is the bias of the layer, F s (i) is the feedback matrix of this layer.

4. The method according to claim 1, characterized in that: On the client side of each participant, the layer-by-layer optimization method specifically includes: For the i-th layer, when the layer is a neural network layer, the client first processes the input features, and the client q represents the current input as X q (i) By expanding through concatenation, the bias term is introduced into the input features so that each layer can correctly consider the influence of the bias during the forward propagation process; The client sends the intermediate representation after the forward propagation to the server. The server integrates the intermediate embedding representations of all clients into a global feature vector X, inputs it into the global model, and performs forward propagation layer by layer until the final output is generated. The feedback matrix is ​​initialized according to the label data, and the global model is back-propagated. After segmentation, the feedback matrix F adapted to the client is generated. q (i) and transmit the feedback matrix back to each client to guide subsequent back-propagation; Back propagation starts from the last layer, and the client updates the parameters of each layer in turn through the feedback matrix. The process is as follows: First, the Moore-Penrose pseudo-inverse matrix Update the feedback matrix F q (i) and update the weight W of the current layer based on the updated feedback matrix q (i) and bias b q (i) Recalculate the intermediate representation X q (i),After the intermediate representation is calculated, it will be used as the input of the next layer; If the current layer is an activated layer, the client's feedback matrix F q (i) Apply the corresponding inverse operation of the activation function so that the adjusted feedback information is correctly transmitted during the back-propagation process; Until all layers are updated, the client sends the calculated intermediate embeddings to the server for aggregation.

5. The method according to claim 4, characterized in that The specific calculation for updating the feedback matrix on the client is in is the pseudo-inverse matrix of the weight matrix of the client layer i, b q (i) is the bias of the layer, F q (i) is the feedback matrix of this layer; The calculation for parameter updates on the client is in It is the pseudo-inverse matrix of the forward calculation result after the previous layer is updated.

6. A gradient-free and efficient vertical federated learning method based on layer-by-layer optimization, executed by a client, characterized in that: The following steps are involved: After initializing the local neural network model, forward propagation is performed independently locally according to the set model parameters and the configured weights and biases of each layer, and the intermediate embedding representation of each layer is calculated in turn. After calculation, the results are transmitted to the server; Receive the feedback matrix from the server, perform back propagation to independently update the weights and biases of the local model layer by layer, and send the updated intermediate embedding representation to the server, wherein the back propagation starts from the last layer and propagates the feedback matrix forward. The current layer receives the feedback matrix from the next layer, and after updating it using the pseudo-inverse matrix calculation, calculates the intermediate embedding representation of this layer and sends the calculation result to the next layer without participating in subsequent calculations; the feedback matrix is ​​initialized by the server according to the label data, and after back propagation update, it is sent back to the client after segmentation; Repeat the forward propagation and back propagation process until each layer of the local model is effectively updated and convergence is achieved.

7. A gradient-free and efficient vertical federated learning method based on layer-by-layer optimization, executed by the server, characterized in that: The following steps are involved: After receiving the intermediate embedding representations transmitted by each participant, it is integrated into a global feature vector, forward propagation of the global model is performed, the output result is calculated, and the feedback matrix is ​​initialized according to the label data. Backward propagation of the global model is performed until the first layer, and the feedback matrix is ​​split and sent back to the corresponding participant; Receive the updated intermediate embedding representations of each participant, integrate them to form global features, perform back propagation to update the global model layer by layer, wherein the server uses the pseudo-inverse matrix to calculate layer by layer, and adjusts the weights after each layer; the updated intermediate embedding representation is obtained by each participant performing back propagation to independently update the weights and biases of the local model layer by layer after receiving the feedback matrix, and calculating according to the updated weights and biases, wherein each participant propagates the feedback matrix forward from the last layer, and the current layer receives the feedback matrix from the next layer, and after updating it using the pseudo-inverse matrix calculation, calculates the intermediate embedding representation of this layer and sends the calculation result to the next layer, and does not participate in subsequent calculations; Repeat the forward propagation and back propagation process until each layer of the global model is effectively updated and convergence is achieved.

8. A computer device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the gradient-free efficient vertical federated learning method based on layer-by-layer optimization as described in claim 6 or 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the gradient-free efficient vertical federated learning method based on layer-by-layer optimization as described in claim 6 or 7 are implemented.

10. A gradient-free and efficient vertical federated learning system based on layer-by-layer optimization, comprising a server and several participant devices, wherein the participant devices and the server complete their respective model optimization based on the gradient-free and efficient vertical federated learning method based on layer-by-layer optimization described in any one of claims 1-5.

Images