A longitudinal federated learning method based on data feature dimension segmentation representation

Abstract

The application discloses a longitudinal federated learning method based on data feature dimension segmentation representation, which firstly obtains data with the same ID sample by each data center, cuts local data into data subsets with the same feature quantity by each data center, and uploads the subset potential features to the server through the representation network, and calculates the global loss function and sends it to each data center. Secondly, each data center updates the local subset representation network according to the local and global loss function and uploads it to the server, and the server sends it to each data center after safe aggregation. Finally, each data center uploads the potential features obtained through the trained representation network to the server, and aggregates to form global data potential representation. The application improves the precision of the longitudinal federated learning method training model, and plays the potential of federated learning in multi-data center cooperation.

Description

Technical Field

[0001] This invention belongs to the field of distributed federated learning technology in artificial intelligence, and specifically relates to a vertical federated learning method based on data feature dimension segmentation representation. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, data has become a core resource driving technological progress and economic growth, and is now listed as one of the five major factors of production. In traditional AI paradigms, model training typically requires collecting and aggregating data to a central server. This method can lead to data privacy breaches due to the transmission of raw data. In reality, data is usually stored and managed in data centers across different regions or institutions, and the data between these institutions holds enormous potential value. How to achieve effective collaboration between different data centers while protecting data privacy is a key issue in the current intelligent development of various fields.

[0003] Data from different data centers often contains many samples with the same ID but different features due to differences in business operations and regions. Vertical federated learning is an important method for securely mining data features and achieving business intelligence in this scenario. This method addresses this problem by collaboratively training a global machine learning model across multiple data centers. Each data center can independently train a portion of the global model based on its local data features. In each training round, the server collects the local network outputs from all data centers, calculates gradients, and backpropagates the gradients back to each data center. Through iterative training, the model achieves joint optimization and learning. For example, data centers of different medical institutions in the same region may possess different medical diagnostic data for the same patients. This sensitive data is stored in each hospital's data center; for instance, a cancer hospital's data center holds a large amount of patient tumor feature data, while a health checkup hospital's data center holds more comprehensive patient health indicators. To maximize the utilization of medical resources, the two hospitals can collaboratively train relevant models locally without leaking patient data, enabling collaboration between these hospital data centers for research such as disease prevention.

[0004] However, existing longitudinal federated learning methods still have some limitations in practical applications, especially when each data center holds a small number of data labels for the same samples. This leads to a decline in the performance of collaboratively trained models because they cannot effectively utilize unlabeled samples. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention designs and implements a vertical federated learning method based on data feature dimension segmentation representation.

[0006] First, by segmenting the dataset based on data feature dimensions, the task of representing global data is transformed into a multi-view federated feature representation task, capturing the relationships between data from different data centers in the feature space. Specifically, based on the minimum feature dimension held by each data center, the data from all data centers is segmented into subsets of the same dimension.

[0007] Secondly, this invention trains the same local feature subset representation network in each data center using all unlabeled data. Based on the latent spatial relationships between local subsets and between subsets in each data center with the same ID, a loss function is constructed that can represent the reconstruction, contrast and distance between all subsets, so as to train a global feature subset representation network that can output the latent representation of all subsets.

[0008] Finally, based on this, the latent representations of all subsets are averaged and aggregated to form a latent representation that represents the global data. This global latent representation is used to train downstream classification models, etc.

[0009] This invention trains subset representations using unlabeled data to learn the latent representations of the global data. Simultaneously, during training, it fully leverages the connections in the latent space between samples with the same ID across different data centers to enhance the learning ability of the representation network, thereby significantly improving the performance of downstream supervised learning tasks. This method not only enhances the model's generalization ability but also optimizes data utilization efficiency, providing a novel solution for federated learning in complex data environments.

[0010] A longitudinal federated learning method based on data feature dimension segmentation representation includes the following steps:

[0011] S1: Each data center performs data alignment operations using methods such as privacy intersection to obtain data containing all samples with the same ID for training. Where N and D m These represent the number of samples and the number of sample features, respectively.

[0012] S2: Based on the data X held by each data center m Number of features D m Based on the minimum feature dimension, all local data are divided into data subsets with the same number of features. Gaussian noise is added to the subset to protect data privacy. The server distributes the global subset representation to each data center after network initialization.

[0013] S3: Each data center calculates its local loss function L local Simultaneously, a subset of latent features is projected and uploaded to the server via a representation network. The server collects all feature projections and calculates the global loss function L. global And distribute it to each data center.

[0014] S4: Each data center updates its local subset representation network based on the local loss function and the global loss function, and uploads the resulting representation network to the server.

[0015] S5: After the server performs a secure aggregation operation on the representation network of all data centers, it redistributes the data to each data center.

[0016] S6: Iterate from S3 to S5 until the local representation network converges. Each data center uploads the latent features obtained by the trained representation network from all subsets to the server, and aggregates them to form a global data latent representation for training of downstream classification tasks.

[0017] Furthermore, in S2:

[0018] S21: Given the different data dimensions in each data center, the local data in the data center is divided into subsets with the same dimension according to the amount of characteristic data. The number of subsets and the number of subsets are determined by the server, while allowing overlap between subsets.

[0019] S22: The global subset representation network consists of an encoder, a decoder, and a projector, forming the basic structure for training the representation network. The encoder projects the data onto the latent feature space, the decoder restores the latent features to the original data, and the projector projects the latent features onto a unified space. This space is used to upload the data to the server to calculate the loss while also protecting data privacy.

[0020] Furthermore, in S3:

[0021] S31: Local losses mainly include reconstruction loss, contrast loss, and distance loss. Reconstruction loss is calculated from the difference between the original data and the reconstructed data. Contrast loss is calculated from positive and negative sample pairs consisting of identical and different samples between different local subsets. Distance loss is mainly composed of the distance between identical samples in the feature space within different local subsets.

[0022] S32: The global loss mainly includes contrast loss and distance loss, which are calculated primarily from subset pairs formed between subsets of different data centers.

[0023] Furthermore, in S6: For the server, the subset representation network trained using unlabeled data will be distributed to all data centers, which can then convert all labeled data into latent feature representations and extract effective representations.

[0024] The server receives the latent representations of labeled samples from all data centers. By using the sample ID, the latent representations of the same samples are averaged and aggregated to form the latent representation of the global data. This not only protects data privacy but also extracts effective features from the data for classification tasks.

[0025] The beneficial effects of this invention are as follows: Addressing the problem that the varying number of data features held by different data centers prevents the use of identical local models for pre-training global representations of unlabeled data, this invention proposes a vertical federated learning method based on data feature dimension segmentation. By employing representation learning techniques to effectively utilize unlabeled data pre-training operations, and by using unlabeled samples for training, the potential representations of global data are pre-trained. This improves the accuracy of the vertical federated learning model, fully leveraging the potential of federated learning in multi-data center collaboration, and is highly significant in supporting the intelligent development of various fields. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention;

[0027] Figure 2 This is a flowchart of subset feature representation training in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and specific implementation steps.

[0029] A vertical federated learning method based on data feature dimension segmentation representation, such as Figure 1 and Figure 2 As shown, it includes the following steps:

[0030] Step 1: Data center data initialization

[0031] This example uses the MNSIT dataset for training. The MNSIT dataset consists of handwritten digit images in 10 categories. The encoder, decoder, and projector in the subset representation network are all composed of two-layer neural networks and the ReLU activation function. Two data centers, data center A and data center B, are used, which have different features for samples with the same ID.

[0032] Step 2: Local data partitioning in the data center

[0033] Based on the characteristics of the data held locally in data center A, the data in data center A is divided into subsets of the same dimension. and Where k represents the number of subsets, and Gaussian noise is then added to the subsets.

[0034] Step 3: Subset Representation Network Initialization

[0035] The server initializes the subset representation network based on the subset structure and distributes it to each data center.

[0036] Step 4: Calculate the subset representation network loss

[0037] Step 4.1: Calculate local loss

[0038] All data centers, based on local feature subsets Feature representation is obtained through an autoencoder. The original data is then obtained through a decoder. To improve the network's ability to extract data features, a reconstruction loss is constructed by calculating the difference between the original feature subset and the reconstructed samples:

[0039]

[0040] Where K is the total number of local subsets, N is the size of the batch data, and s k Lr is the reconstruction loss of the k-th subset, and Lr is the average reconstruction loss of all local subsets.

[0041] For feature subsets, the feature representations of subsets with the same ID sample should be spatially closer after projection by the projector and are considered positive samples. Subsets with different ID samples are considered negative samples. To better extract feature representations, the distance between positive samples should be reduced, and the distance between negative samples should be increased. Utilizing the principle of contrastive learning, for local subsets, the contrastive loss between subset pairs is calculated through pairwise pairing. The projected representation of any two subsets is as follows: and Where a, b ∈ K. The contrastive loss between any two subsets can be expressed as:

[0042]

[0043] Since there are multiple subset pairs locally, the total local contrast loss is calculated.

[0044]

[0045] in This represents the number of pairwise pairs in the K local subsets.

[0046] In addition to the contrast loss between subsets, the mean squared error (MSE) loss between the projected representations of subsets also needs to be calculated. The distance loss between subsets is expressed as:

[0047]

[0048] So, regarding the total local distance loss:

[0049]

[0050] Gain local loss L local =L r +L gcos +L gdis

[0051] Step 4.2: Calculate the global loss

[0052] Each data center projects the feature representations of all locally computed subsets to the server. Since samples from different data centers originate from samples with the same ID, the server needs to calculate the contrast and distance losses between subset pairs from different data centers. Unlike the contrast and distance losses within local subsets of each data center, the server calculates the global loss and distributes it to all participants. The subset projection representations between data centers still use a pairwise pairing method to calculate the loss, with paired subsets belonging to different participants. Assume the two participants are divided into... and Where A, B ∈ M, k a and k b Representing the participants P respectively A and P B The number of subsets it has. This represents a pair of projection feature subsets formed arbitrarily by two participants.

[0053]

[0054]

[0055] Similar to the local distance loss, calculate the global distance loss:

[0056]

[0057]

[0058] The global loss L is obtained global =L gcos +L gdis And send the global loss to each data center.

[0059] Step 5: Each data center uses the loss L = L obtained in Step 4. local +L global The local subset representation model is updated through backpropagation. The model is then uploaded to the server, where a secure aggregation method is used to average the models from all data centers before redistributing them to each data center.

[0060] Step 6: Repeat steps 4 and 5 until the subset representation network converges.

[0061] Step 7: Each data center uploads a subset of the labeled data representations to the server. The server then averages and aggregates all subset representations to construct a latent feature representation of the global data. This latent feature representation of the global data is used to train downstream classification tasks.

[0062] Table 1 shows the classification accuracy performance on the MNIST dataset. The tests were conducted using 80% unlabeled data and different proportions of labeled data. Local indicates the local training method that uses only labeled data. The FedSplitNN method is a traditional vertical federated learning method for splitting networks that uses only labeled data.

[0063] FedHSSL uses unlabeled data but does not consider the loss between subsets of data centers. Therefore, the FedHSSL method outperforms the other two methods that only use labeled data. Because our method introduces the relationship between features between data centers, its performance surpasses the other three methods.

[0064] Experimental results show that this invention effectively improves the performance of downstream classification tasks by using unlabeled data to train a representation network on a subset of data to help construct latent feature representations of global data.

[0065] Table 1

[0066]

Claims

1. A vertical federated learning method based on data feature dimension segmentation representation, characterized in that, Includes the following steps: S1: Each data center performs data alignment through privacy-preserving intersection to obtain all training data with samples having the same ID. in and These represent the number of samples and the number of sample features, respectively. S2: Based on the number of features in each data center, the local data is divided into data subsets with the same number of features. The server then distributes the global subset representations to each data center after network initialization. The global subset representation network consists of an encoder, a decoder, and a projector. The encoder projects the data onto the latent feature space, the decoder restores the latent features to the original data, and the projector projects the latent features onto a unified space. The data is uploaded to the server to calculate the loss while protecting data privacy. S3: Each data center calculates its local loss function and projects a subset of latent features to the server through the representation network; the server collects all feature projections, calculates the global loss function, and distributes it to each data center. S4: Each data center updates its local subset representation network based on the local loss function and the global loss function, and uploads the resulting representation network to the server; S5: After the server performs security aggregation on the representation network of each data center, it distributes the data to each data center. S6: Iterate from S3 to S5 until the local representation network converges. Each data center uploads the latent features obtained by the trained representation network from all subsets to the server, and aggregates them to form a global data latent representation for training of downstream classification tasks.

2. The vertical federated learning method based on data feature dimension segmentation representation according to claim 1, characterized in that, Specifically, dividing all local data into subsets with the same number of features means that, for each data center with different data dimensions, the local data in the data center is divided into subsets with the same dimension according to the amount of feature data. The number of subsets and the number of subsets are determined by the server, while allowing overlap between subsets.

3. The vertical federated learning method based on data feature dimension segmentation representation according to claim 1, characterized in that, Step S2 also includes adding Gaussian noise to the data subset to protect data privacy.

4. The vertical federated learning method based on data feature dimension segmentation representation according to claim 3, characterized in that, The local loss includes the characterization network reconstruction loss, contrast loss, and distance loss. The reconstruction loss is calculated from the difference between the original data and the reconstructed data. The contrast loss is calculated from the positive and negative sample pairs consisting of the same and different samples between different local subsets. The distance loss is composed of the distance between the same samples in the feature space between different local subsets. The global loss includes contrast loss and distance loss, which are calculated from subset pairs formed between subsets of different data centers.

5. The vertical federated learning method based on data feature dimension segmentation representation according to claim 1, characterized in that, During the iteration process, for the server, the subset representation network trained with unlabeled data is distributed to all data centers. The data centers convert all labeled data into latent feature representations and extract effective representations.

6. The vertical federated learning method based on data feature dimension segmentation representation according to any one of claims 1 to 5, characterized in that, The specific process of the aggregation to form a potential representation of global data is as follows: The server receives the potential representations of labeled samples from all data centers, and aggregates the potential representations of the same samples by average through the sample ID to form the potential representation of global data.

Images