Accurate and safe longitudinal federated learning method, system and product based on feature conversion

Abstract

The invention discloses an accurate and safe longitudinal federated learning method, system and product based on feature transformation, and the method comprises the steps: firstly, increasing a longitudinal federated data source through aligning unmarked samples based on self-supervised task learning, so as to break through the limitation of aligning marked samples; then each passive participant completes feature conversion layer and local model training on a local private data set; then each passive participant adds differential privacy noise disturbance in a forward propagation process and uploads the differential privacy noise disturbance to the active participant; the active participant carries out global aggregation and completes model updating, and a corresponding gradient is returned to the passive participant; and finally, passively participating in updating the local model and starting a new round of local model training. The accuracy of the model is improved, and the training of the model is accelerated. Therefore, a possible solution is provided for longitudinal federal learning deployment based on differential privacy in an actual scene.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology, and relates to a vertical federated learning method, system and product, and specifically to an accurate and secure vertical federated learning method, system and product based on feature conversion. Background Art

[0002] Vertical federated learning enables multiple parties to collaboratively train models, with the active party leveraging auxiliary features from the passive party to enhance model performance. Vertical federated learning typically keeps data locally and only exchanges model updates, which has been successfully applied in fields such as banking, healthcare, and e-commerce. Although vertical federated learning ensures privacy protection by keeping data local, there is still the problem of privacy leakage. For example, the active party can use feature reconstruction attacks to infer sensitive features of the passive party.

[0003] To address this issue, it is a common practice to integrate privacy protection mechanisms into vertical federated learning, which can be divided into two categories: cryptographic privacy protection mechanisms and non-cryptographic privacy protection mechanisms. Cryptographic privacy protection mechanisms use secure computing techniques (e.g., homomorphic encryption, secret sharing) to perform computations on multi-party data, but they impose a huge burden on computing and communication, especially for deep neural networks. Therefore, for vertical federated learning applications with high efficiency requirements or complex computing tasks, non-cryptographic privacy protection mechanisms (e.g., differential privacy) are preferred. However, one challenge in applying differential privacy to vertical federated learning is the significant drop in model accuracy. Although it provides good privacy protection, such a severe loss in model accuracy may jeopardize the accuracy and reliability of the results, especially in sensitive fields such as medical diagnosis or finance, which may lead to some security risks. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides an accurate and secure longitudinal federated learning method, system and product based on feature conversion, which improves the accuracy of the model and accelerates the training of the model.

[0005] The technical solution adopted by the method of the present invention is: an accurate and secure longitudinal federated learning method based on feature conversion, involving entities including an active participant and K-1 passive participants, where K is a preset value;

[0006] The following steps are involved:

[0007] Step 1: The active participant and K-1 passive participants use aligned unlabeled samples to increase the longitudinal federated data source;

[0008] Step 2: Each passive participant completes the feature conversion layer and local model training using its private dataset;

[0009] Step 3: Each passive participant k∈[K-1] in H k Add noise disturbance to get And upload it to the active participant; among them, H k Indicates the forward propagation result obtained by the passive party training;

[0010] Step 4: The active participant performs global aggregation and completes the global model update, and returns the corresponding gradient to the passive participant. The passive participant completes the local model update and starts a new round of local model training.

[0011] Preferably, the specific implementation of step 1 includes the following sub-steps:

[0012] Step 1.1: Each participant uses a deep neural network model to extract features from the aligned unlabeled samples, obtains feature representation vectors, and uploads them to the active participant;

[0013] Step 1.2: Active participants perform aggregation and L 2 Normalize to obtain aggregate feature representation, then cluster the feature representation to generate cluster labels y = {y i , …, y K};

[0014] Step 1.3: Active participants make decisions based on the prediction results And the clustering result y i Calculate the cross entropy loss function Where N is the number of samples, K is the number of clusters, and y ij ∈y i represents the cluster label for sample i, Indicates the predicted category label for sample i;

[0015] Step 1.4: Each participant shall Perform local model updates, and use the local model parameters learned by each participant to obtain a global pre-trained model for subsequent federated learning tasks; after obtaining the clustering labels of the aligned unlabeled samples, add the aligned unlabeled samples and the corresponding clustering labels to each participant's local private dataset to complete the dataset expansion.

[0016] Preferably, the specific implementation of step 2 includes the following sub-steps:

[0017] Step 2.1: Each participant k∈[K] randomly selects a small batch of samples x k ;

[0018] Step 2.2: Each participant k∈[K] in x k Train the feature conversion layer and local model to get the model output result Among them C k (·) indicates that the model parameter is ξ k The feature conversion layer, θ k are local model parameters, Represents the expanded local model after adding the feature conversion layer.

[0019] Preferably, in step 2, the feature conversion layer is a linear function or a nonlinear function; the nonlinear function includes y=ax 2 +b,ReLU(ax+b),Sigmoid(ax+b), where a and b are hyperparameters, x and y are the corresponding input and output, and ReLU and Sigmoid are nonlinear activation functions.

[0020] Preferably, in step 2, the local model is a feedforward neural network with multiple layers of nodes interconnected, including an input layer, one or more hidden layers and an output layer; in each layer, a group of nodes performs linear or nonlinear transformation on the input signal and passes the processed information to the next layer;

[0021] The hidden layer includes one or more convolution layers, including performing dot product calculations between the convolution kernel and the input matrix of the layer. When the convolution kernel slides along the input matrix of the layer, the convolution operation generates a feature map, which is then contributed to the input of the next layer; the convolution layer is used to generate a set of parallel feature maps, which are composed of sliding different convolution kernels on the input image and performing certain operations; at each sliding position, an element-by-element multiplication and summation operation is performed between the convolution kernel and the input image to project the information in the receptive field to an element in the feature map.

[0022] Preferably, in step 2, the local model is a locally initialized model.

[0023] As a preference, in step 3, each passive participant k∈[H-1] in H k Add differential privacy noise perturbation To get Where σ is the standard deviation of the noise distribution, which is uploaded to the active participant;

[0024] Preferably, the specific implementation of step 4 includes the following sub-steps:

[0025] Step 4.1: The active participant uses the passive participant to expand the dataset label y and the passive participant’s uploaded calculate in is the model parameter of the active participant; N is the number of samples, K is the number of clusters; Represent the global model training function and the global model loss function respectively;

[0026] Step 4.2: Active participants based on corresponding gradient information Update the global model, that is, calculate where η 1 is the global model learning rate of the active participant;

[0027] Step 4.3: Active participants will Return to each passive participant k∈[K-1], and the passive participant updates the local model, that is, calculates where η 2 is the learning rate of the local model of the passive participant, θ k are local model parameters; then a new round of local model training is started; when the number of training rounds reaches the preset value, the vertical federated learning task is completed.

[0028] The technical solution adopted by the system of the present invention is: an accurate and secure vertical federated learning system based on feature conversion, comprising:

[0029] one or more processors;

[0030] A storage device is used to store one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the accurate and secure longitudinal federated learning method based on feature transformation.

[0031] The technical solution adopted by the product of the present invention is: a computer program product, including computer program instructions, when the computer program instructions are run on a computer, the computer executes the accurate and secure longitudinal federated learning method based on feature conversion.

[0032] Compared with the prior art, the beneficial effects of the present invention include:

[0033] (1) The present invention designs a self-supervised task to learn to increase the vertical federated data source by aligning unlabeled samples, so as to break through the limitation of aligning labeled samples.

[0034] (2) The present invention also proposes to provide security protection for the vertical federated learning system based on differential privacy;

[0035] (3) The present invention also proposes a feature conversion layer to reduce the adverse impact of differential privacy on model accuracy.

[0036] (4) The present invention improves the accuracy of the model and accelerates the training of the model.

[0037] (5) The present invention provides a possible solution for vertical federated learning deployment based on differential privacy in practical scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The technical solution of this invention is further described below using embodiments and specific implementation methods. In addition, some drawings are also used in the process of describing the technical solution. For those skilled in the art, other drawings and the intention of the invention can also be obtained based on these drawings without paying creative labor.

[0039] Figure 1 is a method principle diagram of an embodiment of the present invention;

[0040] Figure 2 It is a flowchart of increasing a vertical federated data source by aligning unlabeled samples based on self-supervised task learning according to an embodiment of the present invention;

[0041] Figure 3 It is a local model structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0043] Vertical federated learning allows multiple parties with different characteristics of the same set of users to jointly train machine learning models by sharing gradient updates while keeping the original data locally. However, if a privacy protection mechanism is not implemented, directly sharing gradient updates can easily leak privacy. A promising approach is to use differential privacy to add noise perturbations to the shared results, thereby providing strong security guarantees. However, differential privacy can lead to excessive loss of model accuracy, making it difficult to deploy in practice.

[0044] Differential privacy is a provable mathematical framework that adds carefully designed noise to samples. It ensures that any individual modification to a dataset does not significantly affect the output. Therefore, an attacker cannot infer the presence or absence of individual data in the dataset by analyzing the output.

[0045] This paper provides a possible solution for the deployment of vertical federated learning based on differential privacy in practical scenarios. By designing a self-supervised task learning to increase the vertical federated data source by aligning unlabeled samples, the limitation of aligning labeled samples can be overcome; then, security is provided for the vertical federated learning system based on differential privacy; finally, a feature conversion layer is proposed to reduce the adverse effects of differential privacy on model accuracy.

[0046] Please see Figure 1 , This embodiment provides an accurate and secure longitudinal federated learning method based on feature conversion, involving entities including an active participant and K-1 passive participants;

[0047] Step 1: The active participant and K-1 passive participants use aligned unlabeled samples to increase the longitudinal federated data source based on self-supervised task learning;

[0048] Please see Figure 2 In one implementation, the specific implementation of step 1 includes the following sub-steps:

[0049] Step 1.1: Each participant uses a deep neural network model to extract features from the aligned unlabeled samples, obtains feature representation vectors, and uploads them to the active participant;

[0050] Step 1.2: Active participants perform aggregation and L 2 Normalize to obtain aggregate feature representation, then cluster the feature representation to generate cluster labels y = {y i , …, y K};

[0051] Step 1.3: Active participants make decisions based on the prediction results And the clustering result y i Calculate the cross entropy loss function Where N is the number of samples, K is the number of clusters, and y ij ∈y i represents the cluster label for sample i, Indicates the predicted category label for sample i;

[0052] Step 1.4: Each participant shall Perform local model updates, and use the local model parameters learned by each participant to obtain a global pre-trained model for subsequent federated learning tasks; after obtaining the clustering labels of the aligned unlabeled samples, add the aligned unlabeled samples and the corresponding clustering labels to each participant's local private dataset to complete the dataset expansion.

[0053] Step 2: Each passive participant completes the feature conversion layer and local model training using its private dataset;

[0054] In one embodiment, the feature conversion layer is a linear or nonlinear function, for example, a linear function includes y=ax+b, and a nonlinear function includes y=ax 2 +b,ReLU(ax+b),Sigmoid(ax+b), where a and b are hyperparameters, x and y are the corresponding input and output, and ReLU and Sigmoid are nonlinear activation functions.

[0055] In one embodiment, see Figure 3, the local model is a feedforward neural network with multiple layers of interconnected nodes. It includes an input layer, one or more hidden layers, and an output layer. In each layer, a group of nodes performs linear or nonlinear transformations on the input signal and passes the processed information to the next layer. Linear transformations use weight matrices to linearly combine the inputs, while nonlinear transformations apply nonlinear activation functions (such as ReLU, Sigmoid, Tanh, etc.) to the results. The activation layer uses the ReLU activation function. This structure enables deep neural networks to learn complex features and show excellent performance in specific fields such as computer vision, natural language processing, and speech recognition.

[0056] The hidden layer includes one or more convolutional layers, which includes performing dot product calculations between the convolution kernel and the input matrix of the layer, and its activation function is usually ReLU; when the convolution kernel slides along the input matrix of the layer, the convolution operation generates a feature map, which is then contributed to the input of the next layer; followed by other layers, such as pooling layers and fully connected layers;

[0057] The convolution layer can generate a set of parallel feature maps, which are composed by sliding different convolution kernels on the input image and performing certain operations; at each sliding position, an element-by-element product and sum operation is performed between the convolution kernel and the input image to project the information in the receptive field to an element in the feature map; the size of the convolution kernel is much smaller than the input image, and it acts on the input image in an overlapping or parallel manner. All elements in a feature map are calculated by a convolution kernel, that is, a feature map shares the same weight and bias term;

[0058] The ReLU layer uses the function f(x)=max(0,x) as the activation function of this layer; it can enhance the nonlinear characteristics of the entire neural network without changing the convolutional layer calculation itself;

[0059] The pooling layer is actually a nonlinear form of downsampling. Among them, the maximum pooling is the most common. It divides the input image into several rectangular areas and outputs the maximum value for each sub-area. The pooling layer will continuously reduce the spatial size of the data, so the number of parameters and the amount of calculation will also decrease. Through the pooling layer, we can reduce the sensitivity of the convolution layer to the edge. The pooling layer calculates the output on a pooling window each time, and then moves the pooling window according to the stride. The pooling layer used has a stride of 2 and a pooling window of 2×2. The two-dimensional maximum pooling layer, and then takes the maximum value of the 4 numbers in each block, which will reduce the amount of data by 75%;

[0060] The fully connected layer connects each node in one layer to each node in another layer. After several convolutional and maximum pooling layers, the prediction part of the neural network model is completed through the fully connected layer. The nodes in the fully connected layer are associated with all activations in the previous layer; their activations can be calculated as an affine transformation, that is, multiplied by a matrix and then added with an offset.

[0061] In one embodiment, the deep neural network model is a locally initialized model;

[0062] In one embodiment, the specific implementation of step 2 includes the following sub-steps:

[0063] Step 2.1: Each participant k∈[K] randomly selects a small batch of samples x k ;

[0064] Step 2.2: Each participant k∈[K] in x k Train the feature conversion layer and local model to get the model output result Among them C k (·) indicates that the model parameter is ξ k The feature conversion layer, θ k are local model parameters, Represents the expanded local model after adding the feature conversion layer.

[0065] Step 3: Each passive participant k∈[K-1] in H k Add differential privacy noise perturbation to get And upload it to the active participant; among them, H k Indicates the forward propagation result obtained by the passive party training;

[0066] In one embodiment, each passive participant k∈[K-1] in H k Add differential privacy noise perturbation To get Right now Where σ is the standard deviation of the noise distribution, which is uploaded to the active participant;

[0067] Step 4: The active participant performs global aggregation and completes the global model update, and returns the corresponding gradient to the passive participant. The passive participant completes the local model update and starts a new round of local model training.

[0068] In one implementation, the specific implementation of step 4 includes the following sub-steps:

[0069] Step 4.1: The active participant uses the passive participant to expand the dataset label y and the passive participant’s uploaded calculate in is the model parameter of the active participant; N is the number of samples, K is the number of clusters; They represent the calculation of the global model training function and the calculation of the global model loss function respectively;

[0070] Step 4.2: Active participants based on corresponding gradient information Update the global model, that is, calculate where η 1 is the global model learning rate of the active participant;

[0071] Step 4.3: Active participants will Return to each passive participant k∈[K-1], and the passive participant updates the local model, that is, calculates where η 2 is the learning rate of the local model of the passive participant, θ k are local model parameters; then a new round of local model training is started; when the number of training rounds reaches the preset value, the vertical federated learning task is completed.

[0072] This embodiment also provides an accurate and secure vertical federated learning system based on feature conversion, including:

[0073] one or more processors;

[0074] A storage device is used to store one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the accurate and secure longitudinal federated learning method based on feature transformation.

[0075] This embodiment also provides a computer program product, including computer program instructions, which, when executed on a computer, enable the computer to execute the accurate and secure longitudinal federated learning method based on feature transformation.

[0076] The present invention is further described below through specific experiments. Please see Table 1 below for the test accuracy (%) experimental results of this embodiment and other solutions. Table 2 shows the training time (s) and communication overhead (MB) experimental results of this embodiment and other solutions.

[0077] Table 1

[0078] plan MNIST CIFAR-10 CheXpert DPVFL 72.66 63.46 61.42 DGC 81.17 61.73 57.42 InstaHide 85.75 65.39 59.37 Ours 90.33 72.65 66.51

[0079] Table 2

[0080]

[0081] This experiment compares this implementation with other solutions in terms of test accuracy, training time and communication overhead. Three data sets (MNIST, CIFAR-10, CheXpert) were selected for the experiment. Table 1 shows the test accuracy of different solutions. It is observed that the test accuracy of this implementation is 4.58%–17.67% higher than that of other solutions. Table 2 shows the training time and communication overhead of different solutions. It is found that this implementation shows comparable training time and communication overhead on different data sets compared with DPVFL, DGC and InstaHide. The above experiments show that this implementation improves the accuracy of the model, accelerates the training of the model, and reduces the communication overhead.

[0082] The present invention designs an accurate and secure longitudinal federated learning scheme based on feature conversion. By designing a self-supervised task learning to increase the longitudinal federated data source by aligning unlabeled samples, the limitation of aligning labeled samples is broken; then, differential privacy is used to provide security for the longitudinal federated learning system; finally, a feature conversion layer is proposed to reduce the adverse effects of differential privacy on model accuracy. Compared with previous schemes, the present invention improves the accuracy of the model and accelerates the training of the model. This provides a possible solution for the deployment of longitudinal federated learning based on differential privacy in practical scenarios.

[0083] It should be understood that the embodiments described above are part of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0084] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. An accurate and secure longitudinal federated learning method based on feature transformation, involving an active participant and K-1 passive participants, where: K is the preset value; It is characterized by comprising the following steps: Step 1: The active participant and K-1 passive participants use aligned unlabeled samples to increase the longitudinal federated data source; Step 2: Each passive participant completes the feature conversion layer and local model training using its private dataset; Step 3: Each passive participant k∈[K-1] in H k Add noise disturbance to get And upload it to the active participant; among them, H k Indicates the forward propagation result obtained by the passive party training; Step 4: The active participant performs global aggregation and completes the global model update, and returns the corresponding gradient to the passive participant. The passive participant completes the local model update and starts a new round of local model training.

2. The accurate and secure longitudinal federated learning method based on feature transformation according to claim 1 is characterized in that: The specific implementation of step 1 includes the following sub-steps: Step 1.1: Each participant uses a deep neural network model to extract features from the aligned unlabeled samples, obtains feature representation vectors, and uploads them to the active participant; Step 1.2: The active participant performs aggregation and L2 normalization to obtain an aggregated feature representation, and then clusters the feature representation to generate cluster labels y = {y i , …, y K }; Step 1.3: Active participants make decisions based on the prediction results And the clustering result y i Calculate the cross entropy loss function Where N is the number of samples, K is the number of clusters, and y ij ∈y i represents the cluster label for sample i, Indicates the predicted category label for sample i; Step 1.4: Each participant shall Perform local model updates, and use the local model parameters learned by each participant to obtain a global pre-trained model for subsequent federated learning tasks; After obtaining the clustering labels of the aligned unlabeled samples, the aligned unlabeled samples and the corresponding clustering labels are added to each participant's local private dataset to complete the dataset expansion.

3. The accurate and secure longitudinal federated learning method based on feature transformation according to claim 1 is characterized in that: The specific implementation of step 2 includes the following sub-steps: Step 2.1: Each participant k∈[K] randomly selects a small batch of samples x k ; Step 2.2: Each participant k∈[K] in x k Train the feature conversion layer and local model to get the model output result Among them C k (·) indicates that the model parameter is ξ k The feature conversion layer, θ k are local model parameters, Represents the expanded local model after adding the feature conversion layer.

4. The accurate and secure longitudinal federated learning method based on feature transformation according to claim 1 is characterized in that: In step 2, the feature conversion layer is a linear function or a nonlinear function; the nonlinear function includes y=ax 2 +b,ReLU(ax+b),Sigmoid(ax+b), where a and b are hyperparameters, x and y are the corresponding input and output, and ReLU and Sigmoid are nonlinear activation functions.

5. The accurate and secure longitudinal federated learning method based on feature transformation according to claim 1 is characterized in that: In step 2, the local model is a feedforward neural network with multiple layers of interconnected nodes, including an input layer, one or more hidden layers, and an output layer; in each layer, a group of nodes performs linear or nonlinear transformation on the input signal and passes the processed information to the next layer; The hidden layer includes one or more convolution layers, including performing dot product calculations between the convolution kernel and the input matrix of the layer. When the convolution kernel slides along the input matrix of the layer, the convolution operation generates a feature map, which is then contributed to the input of the next layer; the convolution layer is used to generate a set of parallel feature maps, which are composed of sliding different convolution kernels on the input image and performing certain operations; at each sliding position, an element-by-element multiplication and summation operation is performed between the convolution kernel and the input image to project the information in the receptive field to an element in the feature map.

6. The accurate and secure longitudinal federated learning method based on feature transformation according to claim 1 is characterized in that: In step 2, the local model is a locally initialized model.

7. The accurate and secure longitudinal federated learning method based on feature transformation according to claim 1 is characterized in that: In step 3, each passive participant k∈[K-1] in H k Add differential privacy noise perturbation To get Where σ is the standard deviation of the noise distribution, which is uploaded to the active participant.

8. The accurate and secure longitudinal federated learning method based on feature transformation according to any one of claims 1 to 7, characterized in that: The specific implementation of step 4 includes the following sub-steps: Step 4.1: The active participant uses the passive participant to expand the dataset label y and the passive participant’s uploaded calculate in is the model parameter of the active participant; N is the number of samples, K is the number of clusters; Respectively represent the global model training function and the global model loss function; Step 4.2: Active participants based on corresponding gradient information Update the global model, that is, calculate Where η1 is the global model learning rate of the active participant; Step 4.3: Active participants will Return to each passive participant k∈[K-1], and the passive participant updates the local model, that is, calculates Where η2 is the learning rate of the local model of the passive participant, θ k are local model parameters; then a new round of local model training is started; when the number of training rounds reaches the preset value, the vertical federated learning task is completed.

9. An accurate and secure vertical federated learning system based on feature transformation, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the accurate and secure longitudinal federated learning method based on feature transformation as described in any one of claims 1 to 8.

10. A computer program product comprising computer program instructions, characterized in that: When the computer program instructions are executed on a computer, the computer is caused to execute the accurate and secure longitudinal federated learning method based on feature transformation as claimed in any one of claims 1 to 8.

Images