Accurate and secure vertical federated learning methods, systems, and products based on feature transformation

Abstract

The present invention discloses an accurate and secure longitudinal federated learning method, system, and product based on feature conversion. First, based on self-supervised task learning, aligned unlabeled samples are used to increase the longitudinal federated data source to break through the limitation of aligned labeled samples; then, each passive participant completes the feature conversion layer and local model training on the local private data set; then, each passive participant adds differential privacy noise perturbation during the forward propagation process and uploads it to the active participant; then, the active participant performs global aggregation and completes the model update, returning the corresponding gradient to the passive participant; finally, the passive participant completes the local model update and starts a new round of local model training. The present invention improves the accuracy of the model and accelerates the training of the model. This provides a possible solution for the deployment of longitudinal federated learning based on differential privacy in practical scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology and relates to a vertical federated learning method, system and product, and specifically to an accurate and secure vertical federated learning method, system and product based on feature conversion. Background Art

[0002] Vertical federated learning enables multiple parties to collaboratively train models, with the active party leveraging auxiliary features from the passive party to enhance model performance. Vertical federated learning typically stores data locally and only exchanges model updates. This has been successfully applied in fields such as banking, healthcare, and e-commerce. While vertical federated learning ensures privacy by keeping data local, privacy concerns still exist. For example, the active party can use feature reconstruction attacks to infer sensitive features of the passive party.

[0003] To address this issue, it is a common practice to integrate privacy protection mechanisms into vertical federated learning, which can be divided into two categories: cryptographic privacy protection mechanisms and non-cryptographic privacy protection mechanisms. Cryptographic privacy protection mechanisms use secure computing techniques (such as homomorphic encryption and secret sharing) to perform calculations on multi-party data, but they impose a huge burden on computing and communication, especially for deep neural networks. Therefore, for vertical federated learning applications with high efficiency requirements or complex computing tasks, non-cryptographic privacy protection mechanisms (such as differential privacy) are preferred. However, one challenge of applying differential privacy to vertical federated learning is the significant drop in model accuracy. Although it provides good privacy protection, such a severe loss in model accuracy may jeopardize the accuracy and reliability of the results, especially in sensitive fields such as medical diagnosis or finance, which may lead to some security risks. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides an accurate and secure longitudinal federated learning method, system and product based on feature conversion, which improves the accuracy of the model and accelerates the training of the model.

[0005] The technical solution adopted by the method of the present invention is: an accurate and secure longitudinal federated learning method based on feature transformation, involving entities including an active participant and K-1 passive participants, where K is a preset value;

[0006] The following steps are involved:

[0007] Step 1: The active participant and K-1 passive participants use aligned unlabeled samples to increase the longitudinal federated data source;

[0008] Step 2: Each passive participant completes the feature conversion layer and local model training using its private dataset;

[0009] Step 3: Each passive participant k∈[K-1] in H k Add noise disturbance to get And upload it to the active participant; among them, H k Indicates the forward propagation result obtained by the passive side training;

[0010] Step 4: The active participant performs global aggregation and completes the global model update, returning the corresponding gradient to the passive participant. The passive participant completes the local model update and starts a new round of local model training.

[0011] Preferably, the specific implementation of step 1 includes the following sub-steps:

[0012] Step 1.1: Each participant uses a deep neural network model to extract features from the aligned unlabeled samples, obtains a feature representation vector, and uploads it to the active participant;

[0013] Step 1.2: The active participant performs aggregation and L2 normalization to obtain the aggregated feature representation, and then clusters the feature representation to generate cluster labels y = {y i ,…,y K};

[0014] Step 1.3: Active participants make decisions based on the prediction results And the clustering result y i Calculate the cross entropy loss function Where N is the number of samples, K is the number of clusters, and y ij ∈y i represents the cluster label for sample i, Indicates the predicted category label for sample i;

[0015] Step 1.4: Each participant shall Perform local model updates, and use the local model parameters learned by each participant to obtain a global pre-trained model for subsequent federated learning tasks. After obtaining the cluster labels of aligned unlabeled samples, the aligned unlabeled samples and corresponding cluster labels are added to each participant's local private dataset to complete the dataset expansion.

[0016] Preferably, the specific implementation of step 2 includes the following sub-steps:

[0017] Step 2.1: Each participant k∈[K] randomly selects a small batch of samples x k ;

[0018] Step 2.2: Each participant k∈[K] in x k Train the feature conversion layer and local model to obtain the model output results Among them C k(·) indicates that the model parameter is ξ k The feature conversion layer, θ k are local model parameters, Represents the expanded local model after adding the feature conversion layer.

[0019] Preferably, in step 2, the feature conversion layer is a linear function or a nonlinear function; the nonlinear function includes y=ax 2 +b, ReLU(ax+b), Sigmoid(ax+b), where a and b are hyperparameters, x and y are the corresponding input and output, and ReLU and Sigmoid are nonlinear activation functions.

[0020] Preferably, in step 2, the local model is a feedforward neural network with multiple layers of interconnected nodes, including an input layer, one or more hidden layers, and an output layer; in each layer, a group of nodes performs linear or nonlinear transformation on the input signal and passes the processed information to the next layer;

[0021] The hidden layer includes one or more convolution layers, which perform dot product calculations on the convolution kernel and the input matrix of the layer. When the convolution kernel slides along the input matrix of the layer, the convolution operation generates a feature map, which is then contributed to the input of the next layer; the convolution layer is used to generate a set of parallel feature maps, which are composed of sliding different convolution kernels on the input image and performing certain operations; at each sliding position, an element-by-element multiplication and sum operation is performed between the convolution kernel and the input image to project the information within the receptive field into an element in the feature map.

[0022] Preferably, in step 2, the local model is a locally initialized model.

[0023] As a preference, in step 3, each passive participant k∈[H-1] in H k Add differential privacy noise perturbation To get Where σ is the standard deviation of the noise distribution, which is uploaded to the active participant;

[0024] Preferably, the specific implementation of step 4 includes the following sub-steps:

[0025] Step 4.1: The active participant uses the passive participant to expand the dataset label y and the passive participant's uploaded calculate in is the model parameter of the active participant; N is the number of samples, K is the number of clusters; Represent the global model training function and global model loss function respectively;

[0026] Step 4.2: Active participants based on corresponding gradient information Update the global model, i.e. calculate Where η1 is the global model learning rate of the active participant;

[0027] Step 4.3: The active participant will Return to each passive participant k∈[K-1], the passive participant updates the local model, that is, calculates Where η2 is the learning rate of the local model of the passive participant, θ k are local model parameters; then a new round of local model training is started; when the number of training rounds reaches the preset value, the vertical federated learning task is completed.

[0028] The technical solution adopted by the system of the present invention is: an accurate and secure vertical federated learning system based on feature transformation, including:

[0029] one or more processors;

[0030] A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the accurate and secure longitudinal federated learning method based on feature transformation.

[0031] The technical solution adopted by the product of the present invention is: a computer program product, including computer program instructions, which, when the computer program instructions are run on a computer, enable the computer to execute the accurate and secure longitudinal federated learning method based on feature conversion.

[0032] Compared with the prior art, the beneficial effects of the present invention include:

[0033] (1) This paper designs a self-supervised task to learn to increase the vertical federated data source by aligning unlabeled samples, so as to break through the limitation of aligning labeled samples.

[0034] (2) The present invention also proposes to provide security protection for the vertical federated learning system based on differential privacy;

[0035] (3) The present invention also proposes a feature conversion layer to reduce the adverse impact of differential privacy on model accuracy.

[0036] (4) The present invention improves the accuracy of the model and accelerates the training of the model.

[0037] (5) This invention provides a possible solution for vertical federated learning deployment based on differential privacy in practical scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The technical solution of this invention is further illustrated below using embodiments and specific implementation methods. In addition, some drawings are used in the process of illustrating the technical solution. Those skilled in the art can also derive other drawings and the intent of the present invention based on these drawings without making any creative efforts.

[0039] Figure 1 is a method principle diagram of an embodiment of the present invention;

[0040] Figure 2 2. It is a flowchart of increasing a longitudinal federated data source by aligning unlabeled samples based on self-supervised task learning according to an embodiment of the present invention;

[0041] Figure 3 It is a local model structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to facilitate ordinary technicians in this field to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0043] Vertical federated learning allows multiple parties with different characteristics from the same set of users to jointly train machine learning models by sharing gradient updates while keeping the original data locally. However, without implementing privacy protection mechanisms, directly sharing gradient updates can easily leak privacy. A promising approach is to leverage differential privacy to add noise perturbations to the shared results, providing strong security guarantees. However, differential privacy can lead to significant loss in model accuracy, making it difficult to deploy in practice.

[0044] Differential privacy is a provable mathematical framework that uses carefully designed noise to add to samples. It ensures that any individual modification to a dataset has no significant effect on the output. Therefore, an attacker cannot infer the presence of individual data points in the dataset by analyzing the output.

[0045] This paper provides a potential solution for deploying vertical federated learning based on differential privacy in real-world scenarios. By designing a self-supervised task learning algorithm that leverages aligned unlabeled samples to augment the vertical federated data source, the paper overcomes the limitation of aligning labeled samples. Furthermore, differential privacy is used to provide security for the vertical federated learning system. Finally, a feature conversion layer is proposed to mitigate the adverse effects of differential privacy on model accuracy.

[0046] Please see Figure 1 ,This embodiment provides an accurate and secure longitudinal federated learning method based on feature transformation, involving an active participant and K-1 passive participants;

[0047] Step 1: The active participant and K-1 passive participants use aligned unlabeled samples based on self-supervised task learning to increase the longitudinal federated data source;

[0048] Please see Figure 2 In one embodiment, the specific implementation of step 1 includes the following sub-steps:

[0049] Step 1.1: Each participant uses a deep neural network model to extract features from the aligned unlabeled samples, obtains a feature representation vector, and uploads it to the active participant;

[0050] Step 1.2: The active participant performs aggregation and L2 normalization to obtain the aggregated feature representation, and then clusters the feature representation to generate cluster labels y = {y i ,…,y K};

[0051] Step 1.3: Active participants make decisions based on the prediction results And the clustering result y i Calculate the cross entropy loss function Where N is the number of samples, K is the number of clusters, and y ij ∈y i represents the cluster label for sample i, Indicates the predicted category label for sample i;

[0052] Step 1.4: Each participant shall Perform local model updates, and use the local model parameters learned by each participant to obtain a global pre-trained model for subsequent federated learning tasks. After obtaining the cluster labels of aligned unlabeled samples, the aligned unlabeled samples and corresponding cluster labels are added to each participant's local private dataset to complete the dataset expansion.

[0053] Step 2: Each passive participant completes the feature conversion layer and local model training using its private dataset;

[0054] In one embodiment, the feature conversion layer is a linear or nonlinear function, for example, a linear function includes y=ax+b, and a nonlinear function includes y=ax 2 +b, ReLU(ax+b), Sigmoid(ax+b), where a and b are hyperparameters, x and y are the corresponding input and output, and ReLU and Sigmoid are nonlinear activation functions.

[0055] In one embodiment, see Figure 3The local model is a feedforward neural network with multiple layers of interconnected nodes. It includes an input layer, one or more hidden layers, and an output layer. In each layer, a group of nodes performs linear or nonlinear transformations on the input signal and passes the processed information to the next layer. Linear transformations use weight matrices to linearly combine the inputs, while nonlinear transformations apply nonlinear activation functions (such as ReLU, Sigmoid, Tanh, etc.) to the results. The activation layer uses the ReLU activation function. This structure enables deep neural networks to learn complex features and demonstrate outstanding performance in specific fields such as computer vision, natural language processing, and speech recognition.

[0056] The hidden layer includes one or more convolutional layers, which perform dot product calculations on the convolution kernel and the input matrix of the layer. The activation function is usually ReLU. When the convolution kernel slides along the input matrix of the layer, the convolution operation generates a feature map, which is then contributed to the input of the next layer. This is followed by other layers, such as pooling layers and fully connected layers.

[0057] The convolution layer can generate a set of parallel feature maps, which are composed by sliding different convolution kernels on the input image and performing certain operations. At each sliding position, an element-wise product-sum operation is performed between the convolution kernel and the input image to project the information in the receptive field into an element in the feature map. The size of the convolution kernel is much smaller than the input image, and it acts on the input image in overlapping or parallel manner. All elements in a feature map are calculated by using a convolution kernel, that is, a feature map shares the same weights and bias terms.

[0058] The ReLU layer uses the function f(x)=max(0,x) as the activation function of this layer; it can enhance the nonlinear characteristics of the entire neural network without changing the convolutional layer calculation itself;

[0059] The pooling layer is actually a nonlinear form of downsampling, of which maximum pooling is the most common. It divides the input image into several rectangular regions and outputs the maximum value for each sub-region. The pooling layer continuously reduces the spatial size of the data, so the number of parameters and the amount of computation also decrease. Through the pooling layer, we can reduce the sensitivity of the convolutional layer to edges. The pooling layer calculates the output on a pooling window at a time and then moves the pooling window according to the stride. The pooling layer used has a stride of 2 and a 2×2 two-dimensional maximum pooling layer for the pooling window. Then, the maximum value of the four numbers in each block is taken, which will reduce the amount of data by 75%.

[0060] The fully connected layer connects each node in one layer to each node in another layer. After several convolution and maximum pooling layers, the prediction part of the neural network model is completed by the fully connected layer. The nodes in the fully connected layer are associated with all activations in the previous layer; their activations can be calculated as an affine transformation, that is, multiplied by a matrix and then added with an offset.

[0061] In one embodiment, the deep neural network model is a locally initialized model;

[0062] In one embodiment, the specific implementation of step 2 includes the following sub-steps:

[0063] Step 2.1: Each participant k∈[K] randomly selects a small batch of samples x k ;

[0064] Step 2.2: Each participant k∈[K] in x k Train the feature conversion layer and local model to obtain the model output results Among them C k (·) indicates that the model parameter is ξ k The feature conversion layer, θ k are local model parameters, Represents the expanded local model after adding the feature conversion layer.

[0065] Step 3: Each passive participant k∈[K-1] in H k Add differential privacy noise perturbation to get And upload it to the active participant; among them, H k Indicates the forward propagation result obtained by the passive side training;

[0066] In one embodiment, each passive participant k∈[K-1] in H k Add differential privacy noise perturbation To get Right now Where σ is the standard deviation of the noise distribution, which is uploaded to the active participant;

[0067] Step 4: The active participant performs global aggregation and completes the global model update, returning the corresponding gradient to the passive participant. The passive participant completes the local model update and starts a new round of local model training.

[0068] In one embodiment, the specific implementation of step 4 includes the following sub-steps:

[0069] Step 4.1: The active participant uses the passive participant to expand the dataset label y and the passive participant's uploaded calculate in is the model parameter of the active participant; N is the number of samples, K is the number of clusters; Respectively represent the calculation of the global model training function and the global model loss function;

[0070] Step 4.2: Active participants based on corresponding gradient information Update the global model, i.e. calculate Where η1 is the global model learning rate of the active participant;

[0071] Step 4.3: The active participant will Return to each passive participant k∈[K-1], the passive participant updates the local model, that is, calculates Where η2 is the learning rate of the local model of the passive participant, θ k are local model parameters; then a new round of local model training is started; when the number of training rounds reaches the preset value, the vertical federated learning task is completed.

[0072] This embodiment also provides an accurate and secure vertical federated learning system based on feature transformation, including:

[0073] one or more processors;

[0074] A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the accurate and secure longitudinal federated learning method based on feature transformation.

[0075] This embodiment also provides a computer program product, including computer program instructions, which, when executed on a computer, enable the computer to execute the accurate and secure longitudinal federated learning method based on feature transformation.

[0076] The present invention is further illustrated below through specific experiments. Table 1 shows the experimental results of the test accuracy (%) of this embodiment and other solutions. Table 2 shows the experimental results of the training time (s) and communication overhead (MB) of this embodiment and other solutions.

[0077] Table 1

[0078] plan MNIST CIFAR-10 CheXpert DPVFL 72.66 63.46 61.42 DGC 81.17 61.73 57.42 InstaHide 85.75 65.39 59.37 Ours 90.33 72.65 66.51

[0079] Table 2

[0080]

[0081] This experiment compares this implementation with other solutions in terms of test accuracy, training time, and communication overhead. Three datasets (MNIST, CIFAR-10, and CheXpert) were selected for the experiment. Table 1 shows the test accuracy of different solutions. It is observed that the test accuracy of this implementation is 4.58%–17.67% higher than that of other solutions. Table 2 shows the training time and communication overhead of different solutions. It is found that this implementation exhibits comparable training time and communication overhead on different datasets compared to DPVFL, DGC, and InstaHide. The above experiments demonstrate that this implementation improves model accuracy, accelerates model training, and reduces communication overhead.

[0082] This paper designs an accurate and secure longitudinal federated learning solution based on feature transformation. By designing a self-supervised task learning approach, it utilizes aligned unlabeled samples to augment the longitudinal federated data source, overcoming the limitation of aligned labeled samples. It then provides security for the longitudinal federated learning system based on differential privacy. Finally, it proposes a feature transformation layer to mitigate the adverse effects of differential privacy on model accuracy. Compared to previous solutions, this paper improves model accuracy and accelerates model training. This provides a potential solution for deploying longitudinal federated learning based on differential privacy in real-world scenarios.

[0083] It should be understood that the embodiments described above are only some of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention may be arbitrarily combined with each other to form a feasible technical solution. Such combination is not restricted by the order of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is mutually inconsistent or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0084] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. An accurate and secure longitudinal federated learning method based on feature transformation, involving an active participant and K-1 passive participants, where: K is the preset value; It is characterized in that it includes the following steps: Step 1: The active participant and K-1 passive participants use aligned unlabeled samples to increase the longitudinal federated data source; The specific implementation of step 1 includes the following sub-steps: Step 1.1: Each participant uses a deep neural network model to extract features from the aligned unlabeled samples, obtains a feature representation vector, and uploads it to the active participant; Step 1.2: The active participant performs aggregation and L2 normalization to obtain the aggregated feature representation, and then clusters the feature representation to generate cluster labels y = {y i ,…,y K }; Step 1.3: Active participants make decisions based on the prediction results And the clustering result y i Calculate the cross entropy loss function Where N is the number of samples, K is the number of clusters, and y ij ∈y i represents the cluster label for sample i, Indicates the predicted category label for sample i; Step 1.4: Each participant shall Perform local model updates, and use the local model parameters learned by each participant to obtain a global pre-trained model for subsequent federated learning tasks. After obtaining the cluster labels of the aligned unlabeled samples, the aligned unlabeled samples and the corresponding cluster labels are added to each participant's local private dataset to complete the dataset expansion. Step 2: Each passive participant completes the feature conversion layer and local model training using its private dataset; The specific implementation of step 2 includes the following sub-steps: Step 2.1: Each participant k∈[K] randomly selects a small batch of samples x k ; Step 2.2: Each participant k∈[K] in x k Train the feature conversion layer and local model to obtain the model output results Among them C k (·) indicates that the model parameter is ξ k The feature conversion layer, θ k are local model parameters, Represents the local model expanded after adding the feature conversion layer; The feature conversion layer is a linear function or a nonlinear function; the nonlinear function includes y=ax 2 +b, ReLU(ax+b), Sigmoid(ax+b), where a and b are hyperparameters, x and y are the corresponding input and output, and ReLU and Sigmoid are nonlinear activation functions; Step 3: Each passive participant k∈[K-1] in H k Add noise disturbance to get And upload it to the active participant; among them, H k Indicates the forward propagation result obtained by the passive side training; Step 4: The active participant performs global aggregation and completes the global model update, returning the corresponding gradient to the passive participant. The passive participant completes the local model update and starts a new round of local model training.

2. The accurate and secure longitudinal federated learning method based on feature transformation according to claim 1 is characterized by: In step 2, the local model is a feedforward neural network with multiple layers of interconnected nodes, including an input layer, one or more hidden layers, and an output layer; in each layer, a group of nodes performs linear or nonlinear transformation on the input signal and passes the processed information to the next layer; The hidden layer includes one or more convolution layers, which perform dot product calculations on the convolution kernel and the hidden layer input matrix. When the convolution kernel slides along the input matrix of the hidden layer, the convolution operation generates a feature map, which is then contributed to the input of the next layer; the convolution layer is used to generate a set of parallel feature maps, which are composed of sliding different convolution kernels on the input image and performing certain operations; at each sliding position, an element-by-element multiplication and sum operation is performed between the convolution kernel and the input image to project the information within the receptive field into an element in the feature map.

3. The accurate and secure longitudinal federated learning method based on feature transformation according to claim 1 is characterized by: In step 2, the local model is a locally initialized model.

4. The accurate and secure longitudinal federated learning method based on feature transformation according to claim 1 is characterized by: In step 3, each passive participant k∈[K-1] in H k Add differential privacy noise perturbation To get Where σ is the standard deviation of the noise distribution, which is uploaded to the active participant.

5. The accurate and secure longitudinal federated learning method based on feature transformation according to any one of claims 1 to 4, characterized in that: The specific implementation of step 4 includes the following sub-steps: Step 4.1: The active participant uses the passive participant to expand the dataset label y and the passive participant's uploaded calculate in is the model parameter of the active participant; N is the number of samples, K is the number of clusters; Represent the global model training function and global model loss function respectively; Step 4.2: Active participants based on corresponding gradient information Update the global model, i.e. calculate Where η1 is the global model learning rate of the active participant; Step 4.3: The active participant will Return to each passive participant k∈[K-1], the passive participant updates the local model, that is, calculates Where η2 is the learning rate of the local model of the passive participant, θ k are local model parameters; then a new round of local model training is started; when the number of training rounds reaches the preset value, the vertical federated learning task is completed.

6. An accurate and secure vertical federated learning system based on feature transformation, characterized by: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the accurate and secure longitudinal federated learning method based on feature transformation as described in any one of claims 1 to 5.

7. A computer program product comprising computer program instructions, characterized in that: When the computer program instructions are executed on a computer, the computer is caused to execute the accurate and secure longitudinal federated learning method based on feature transformation according to any one of claims 1 to 5.

Images