A self-supervised longitudinal federated learning method and apparatus based on instance similarity and dynamic balancing pooling.

Abstract

This invention belongs to the field of federated learning technology and discloses a self-supervised longitudinal federated learning method and apparatus based on instance similarity and dynamic balancing pooling. By combining contrastive self-supervised methods with instance similarity, it helps each participant's private longitudinal federated learning model capture more important inter-domain knowledge, thereby improving the generalization and representation capabilities of each participant's model while avoiding regularization conflicts. Furthermore, a dynamic balancing pooling module is proposed to fine-tune the pre-trained model and dynamically balance inter-domain and intra-domain knowledge in downstream supervised tasks, thereby improving the performance of the joint longitudinal federated learning model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of federated learning, and particularly relates to a self-supervised vertical federated learning method and device based on instance similarity and a dynamic balanced pool. BACKGROUND

[0002] Federated learning (FL) is a promising paradigm that enables multiple independent participants to collaboratively perform machine learning tasks while protecting user privacy data. Vertical federated learning (VFL) is one of the important research fields in FL, and its application scenario is that the data sets owned by the participants are different in feature space, but there may be partial overlap in sample ID space. VFL has shown great application prospects in finance, advertising and medical fields due to its ability to enhance model performance by utilizing complementary data while protecting data privacy, and has attracted widespread attention from academia and industry.

[0003] Traditional VFL usually uses aligned and labeled data to train the VFL model. However, the process of data labeling is time-consuming and expensive, and the aligned samples between participants are relatively scarce. Therefore, in actual application scenarios, participants can only obtain a limited number of aligned and labeled samples, which is not conducive to the training of the VFL model.

[0004] In order to facilitate the use of a large amount of valuable unlabeled data held by participants, current VFL frameworks usually integrate contrastive self-supervised learning (CSSL) methods. For example, the CSSL method is combined with the cross-view method to utilize the labeled samples and unlabeled samples of each participant. However, the CSSL method usually conflicts with the conventional regularization method aimed at improving the performance of the model, because the CSSL method aims to capture internal information of representation to maximize the discriminability between different samples, which deviates significantly from the goal of the conventional regularization method. For example, Kullback-Leibler (KL) divergence is a recognized effective regularization technique. However, existing research has found that the introduction of KL divergence into the CSSL method increases the instability in the training process. The small changes in the representation learned by the CSSL method can cause significant fluctuations in the KL divergence, thereby weakening the representation ability of the model and negatively affecting the performance of the model. Therefore, how to solve the conflict between the CSSL method and the conventional regularization method to improve the performance of the VFL model has become a problem to be solved. SUMMARY

[0005] The purpose of the present application is to provide a self-supervised vertical federated learning method and device based on instance similarity and a dynamic balanced pool, which solves the conflict between the CSSL method and the conventional regularization method and improves the performance of the VFL model.

[0006] To achieve the above-mentioned purpose, the technical solutions adopted by the present application are:

[0007] In a first aspect, the present application provides a self-supervised longitudinal federated learning method based on instance similarity and dynamic balance pool, which is used for collaborative training of a joint longitudinal federated learning model by K participants and a server. The joint longitudinal federated learning model comprises a teacher network, a local network, a dynamic balance pool module and a classifier. The K participants include active participants with labels and passive participants without labels. The self-supervised longitudinal federated learning method based on instance similarity and dynamic balance pool comprises a pre-training phase and a fine-tuning phase, wherein:

[0008] The pre-training phase comprises:

[0009] Each participant collaboratively trains the respective teacher network using the aligned samples. In the collaborative training, the instance similarity is calculated based on the output of the teacher network, and the loss is calculated in combination with the instance similarity.

[0010] Each participant trains the respective local network using the local private samples and the respective teacher network.

[0011] Each participant uploads the local network to the server. The server aggregates the local networks to generate a global network, and then distributes the global network to each participant as a new local network.

[0012] The pre-training phase is repeated until the pre-training is completed.

[0013] The fine-tuning phase comprises:

[0014] Each participant inputs the aligned and labeled samples into the respective teacher network and local network, respectively, and aggregates the intermediate representations output by the teacher network and the intermediate representations output by the local network using the dynamic balance pool module to obtain the respective final representation.

[0015] The passive participants send the final representation to the active participants for aggregation to obtain an aggregated representation.

[0016] The active participants convert the aggregated representation into a predicted value using the classifier, and train the dynamic balance pool module and the classifier according to the predicted value and the label.

[0017] The fine-tuning phase is repeated until the fine-tuning is completed, and the trained joint longitudinal federated learning model is output.

[0018] The following also provides several optional modes, but not as an additional limitation to the above overall scheme, but only as a further supplement or preference. Without technical or logical contradiction, each optional mode can be combined with the above overall scheme, and multiple optional modes can also be combined.

[0019] As a preferred embodiment, the collaborative training of the respective teacher network by each participant using the aligned samples comprises:

[0020] The teacher network comprises a teacher encoder and a teacher projector, the aligned sample is input into the teacher encoder to generate a teacher intermediate representation, and then the teacher intermediate representation is input into the teacher projector to generate a teacher final representation;

[0021] A first positive sample pair is generated based on the teacher final representation, and a representation self-supervised loss is calculated according to the first positive sample pair;

[0022] Instance similarities of each participant are calculated based on the teacher intermediate representation, a second positive sample pair is generated according to the instance similarities, and an instance self-supervised loss is calculated according to the second positive sample pair;

[0023] The representation self-supervised loss and the instance self-supervised loss are aggregated as a co-training final loss, and the teacher network is updated according to the co-training final loss.

[0024] Preferably, the first positive sample pair is generated based on the teacher final representation, and the representation self-supervised loss is calculated according to the first positive sample pair, comprising:

[0025] The teacher final representation of the participant k is taken

[0026] The teacher final representation of each participant is uploaded to the server, and the server calculates an average teacher representation The average teacher representation is downloaded to each participant; The average teacher representation is downloaded to each participant;

[0027] After each participant receives the average teacher representation The teacher final representation converted based on the same sample The average teacher representation The teacher final representation converted based on the same sample

[0028] Preferably, the instance similarities of each participant are calculated based on the teacher intermediate representation, the second positive sample pair is generated according to the instance similarities, and the instance self-supervised loss is calculated according to the second positive sample pair, comprising:

[0029] The teacher intermediate representation of the participant k is taken The instance similarity matrix of the participant k is constructed As follows:

[0030]

[0031] In the formula, (·) T is a transpose operation, ω is a temperature hyperparameter, and ||·||2 is an L2 norm;

[0032] ​get the instance similarity S of the participant k k As follows:

[0033]

[0034] wherein, Remove_diag(·) represents an operation of eliminating diagonal elements in a matrix;

[0035] Each participant uploads the instance similarity S to the server, and the server calculates the target instance similarity k and sends the target instance similarity to each participant;

[0036] After each participant receives the target instance similarity , the instance similarity S converted based on the same sample k is matched with the target instance similarity as a second positive sample pair, and a self-supervised loss calculated based on the second positive sample pair is taken as an instance self-supervised loss.

[0037] As a preference, each participant uses local private samples and a respective teacher network to train a respective local network, including:

[0038] The local network is taken as a local online network, and a contrastive self-supervised learning method is adopted to obtain a local target network based on the local online network;

[0039] A data enhancement strategy is adopted to convert the local private samples into a first enhanced view and a second enhanced view;

[0040] The first enhanced view is input into the local online network, and the second enhanced view is input into the local target network to generate a third positive sample pair;

[0041] The second enhanced view is input into the local online network, and the first enhanced view is input into the local target network to generate a fourth positive sample pair;

[0042] A self-supervised loss calculated based on the third positive sample pair and the fourth positive sample pair is taken as a symmetric loss;

[0043] The first enhanced view is input into the local online network, and the first enhanced view is input into the teacher network to generate a fifth positive sample pair;

[0044] The second enhanced view is input into the local online network, and the second enhanced view is input into the teacher network to generate a sixth positive sample pair;

[0045] A self-supervised loss calculated based on the fifth positive sample pair and the sixth positive sample pair is taken as a contrastive knowledge transfer loss;

[0046] ​The local online network is updated by combining the symmetric loss and the contrastive knowledge distillation loss.

[0047] As preferred, each participant uploads the local network to the server, and the server aggregates the local networks to generate the global network, including:

[0048] The local network comprises a local encoder and a local projector, and each participant divides the local encoder into a local bottom encoder and a local top encoder;

[0049] Each participant uploads the local top encoder and the local projector to the server;

[0050] The server aggregates the local top encoders to obtain a global top encoder, aggregates the local projectors to obtain a global projector, and takes the global top encoder and the global projector as the global network.

[0051] As preferred, the local encoder comprises two fully connected layers, and the last layer of the local encoder is divided as the local top encoder and the remaining layers are divided as the local bottom encoder;

[0052] Alternatively, the local encoder comprises an embedding layer and two fully connected layers, and the last layer of the local encoder is divided as the local top encoder and the remaining layers are divided as the local bottom encoder;

[0053] Alternatively, the local encoder is a ResNet-18 network, and the last three modules of the local encoder are divided as the local top encoder and the remaining layers are divided as the local bottom encoder.

[0054] As preferred, the dynamic balance pool module is used to aggregate the intermediate representations output by the teacher network and the intermediate representations output by the local network to obtain the final representations, including:

[0055] Taking the intermediate representation output by the teacher network of the participant k and the intermediate representation output by the local network of the participant k Calculate the forgetting factor O k As follows:

[0056]

[0057] In the formula, denotes the concatenation operation along the feature axis, MLP(·) is a multi-layer perceptron, and σ(·) is a sigmoid function;

[0058] Calculate the update factor U k As follows:

[0059]

[0060] Extract the implicit information Ik As follows:

[0061]

[0062] In the formula, φ(·) is an activation function, FC(·) is a full connection layer, and is a Hadamard product;

[0063] The final representation is calculated as follows:

[0064]

[0065] In the formula, R k is the final representation obtained by the participant k.

[0066] In a second aspect, the present application discloses a self-supervised longitudinal federated learning device based on instance similarity and a dynamic balance pool, which comprises K participants and a server, and the K participants and the server cooperatively train a joint longitudinal federated learning model, wherein the joint longitudinal federated learning model comprises a teacher network, a local network, a dynamic balance pool module and a classifier, the participants with labels are active parties, and the rest are passive parties, and the K participants and the server perform the following operations:

[0067] In the pre-training stage:

[0068] Each participant cooperatively trains the respective teacher network using the aligned samples, and in the cooperative training, the instance similarity is calculated based on the output of the teacher network, and the loss is calculated in combination with the instance similarity;

[0069] Each participant trains the respective local network using the local private samples and the respective teacher network;

[0070] Each participant uploads the local network to the server, the server aggregates the local networks to generate a global network, and the global network is distributed to each participant as a new local network;

[0071] Each participant and the server repeatedly perform the pre-training stage until the pre-training is completed;

[0072] In the fine-tuning stage:

[0073] Each participant respectively inputs the aligned and labeled samples into the respective teacher network and local network, and aggregates the intermediate representations output by the teacher network and the intermediate representations output by the local network using the dynamic balance pool module to obtain the respective final representation;

[0074] The passive party sends the final representation to the active party for aggregation to obtain an aggregated representation;

[0075] The active party converts the aggregated representation into a predicted value using the classifier, and trains the dynamic balance pool module and the classifier according to the predicted value and the label;

[0076] Each participant and the server repeatedly perform the fine-tuning stage until the fine-tuning ends, and output the trained joint vertical federated learning model.

[0077] The self-supervised vertical federated learning method and device based on instance similarity and dynamic balance pool provided by the application, in the pre-training stage, compared with the traditional CSSL method, the application combines the CSSL method with the instance similarity, which can help the private VFL model of each participant to capture more important inter-domain knowledge, thereby improving the generalization ability and representation ability of the model of each participant, and avoiding the regularization conflict. In addition, the application proposes a dynamic balance pool (DBP) module, which is used for fine-tuning the pre-training model in the downstream supervised task and dynamically balancing the inter-domain and intra-domain knowledge, thereby improving the performance of the joint VFL model. A large number of experiments on image type data sets and table type data sets show that the application achieves leading performance in the scene with scarce labels, and does not produce the regularization conflict problem. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 The schematic diagram is for the sample of the application;

[0079] Figure 2 The flowchart is for the self-supervised vertical federated learning method based on instance similarity and dynamic balance pool of the application;

[0080] Figure 3 The flowchart is for the pre-training stage of the application;

[0081] Figure 4 The flowchart is for the downstream supervised task of the application;

[0082] Figure 5 The structural schematic diagram is for the DBP module of the application. DETAILED DESCRIPTION

[0083] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0084] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0085] As in the conventional VFL setup, there are K participants (indexed by k) and a server collaboratively training a joint VFL model. Participant k holds a local private dataset of samples where is the vertical partitioning of the complete dataset Assume only participant 1 has labels, so it is chosen as the active participant, while other participants only have features, so they are all passive participants. Participants link their datasets by aligning samples with the same ID, and thus obtain the complete dataset However, in real-world scenarios, the complete dataset only contains extremely limited aligned and labeled samples where is the part held by participant k, while is the corresponding true label held by the active participant. Aligned but unlabeled samples are denoted as where is the part held by participant k. In addition, denotes the unaligned private samples of participant k. Figure 1 An example of data partitioning is shown for two participants, where participant 1 holds this round of private samples where participant 2 holds this round of private samples

[0086] Unlike most conventional VFL models that only use limited aligned and labeled data X al , the present embodiment uses all aligned and unaligned data to pre-train local private models of each participant. In addition, the present embodiment subsequently requires a fine-tuning stage to utilize a small amount of aligned and labeled samples X al to perform downstream supervised tasks. The present embodiment aims to improve the performance of the joint VFL model in downstream supervised tasks. Therefore, the performance of the present embodiment can be evaluated by downstream supervised tasks.

[0087] As Figure 2As shown, the embodiment proposes a self-supervised vertical federated learning method based on instance similarity and dynamic balance pool (ISSVFL for short), which is used for K participants and a server to collaboratively train a joint vertical federated learning model, the joint vertical federated learning model includes a teacher network, a local network, a dynamic balance pool module and a classifier, and the K participants with labels are active parties and the rest are passive parties. The core idea of ISSVFL is to realize the representation level knowledge transfer by fusing the CSSL method and instance similarity in the VFL environment, so as to improve the domain generalization ability and representation ability of the private VFL model of each participant, while avoiding the regularization conflict problem. In addition, ISSVFL proposes a DBP module for dynamically balancing inter-domain knowledge and intra-domain knowledge. Specifically, ISSVFL includes two stages: a pre-training stage and a fine-tuning stage.

[0088] (1) Pre-training stage: as shown in Figure 3 , this stage uses all available samples to pre-train the private VFL model of each participant in the label scarce scenario, so that the private model can generate representations of inter-domain knowledge and intra-domain knowledge. The private VFL model of each participant is the teacher network and local network part in the joint vertical federated learning model.

[0089] The embodiment introduces the CSSL method in the pre-training stage. The CSSL method is based on instance differentiation tasks, and each instance and its transform is regarded as an independent class. The CSSL method directly compares the representation instead of the real label, aiming to generate a representation that can effectively distinguish different instances. ISSVFL integrates three representative CSSL methods into the VFL environment, namely MoCo (Momentum Contrast), BYOL (Bootstrap Your Own Latent) and SimSiam (Simple Siamese). For MoCo and BYOL, the target network is the momentum version of the online network, while the target network of SimSiam is equivalent to the online network. The target network applies the gradient stop operation. In addition, BYOL and SimSiam additionally increase a predictor on the top layer of the online network. In addition, MoCo regards other samples in the same batch as negative samples after matching the positive sample pair, and the positive sample pair and the negative sample participate in the self-supervised loss calculation at the same time. In order to clearly represent, Figure 3 the predictor of BYOL and SimSiam is omitted in

[0090] (1.1) Each participant uses the aligned sample to collaboratively train the teacher network, and in the collaborative training, the instance similarity is calculated based on the output of the teacher network, and the loss is calculated in combination with the instance similarity. The teacher network includes a teacher encoder and a teacher projector, and if BYOL and SimSiam are used, it also includes a teacher predictor.

[0091] For each party, the features held by the party are equivalent to a longitudinal split of the whole feature set, and can be regarded as complementary views of each other. The complementary views contain rich inter-domain knowledge and naturally form positive samples. Therefore, each party uses the aligned samples to collaboratively train its own teacher network through the CSSL method, so that the teacher network can learn more inter-domain knowledge.

[0092] Specifically, given a batch of aligned samples The longitudinal split part held by the participant k is denoted as For the participant k, its teacher encoder converts the input into an intermediate representation (i.e., the teacher intermediate representation), and then the teacher projector converts the intermediate representation into a final representation (i.e., the teacher final representation). When BYOL or SimSiam is used, the teacher predictor converts the final into a predicted representation

[0093] In order to control the communication overhead and the computational overhead, each participant uploads the final representation to the server, and the server calculates the average teacher representation and then distributes the average teacher representation to each participant. In this process, all representations involved in communication apply gradient stopping operations. Because each positive sample pair is composed of complementary views of the same sample, after receiving , each participant can easily match the corresponding first positive sample pair (or ), is an element in is an element in is an element in is an element in is an element in is an element in Then, each participant k updates its own teacher network (i.e., or ) by minimizing the federated collaborative self-supervised learning (FCSSL) loss (referred to as the representation self-supervised loss), as shown in equation (1).

[0094]

[0095] where is the self-supervised loss using MoCo, BYOL or SimSiam.

[0096] Intermediate representation implies general structural information, which helps to improve the generalization ability of VFL model. This embodiment proposes a representation-level knowledge transfer method based on similarity of aligned sample instances (referred to as FCISL), which aims to capture important inter-domain knowledge contained in intermediate representation by constructing instance similarity matrix.

[0097] Specifically, given the same batch of aligned samples Teacher encoder of participant k Map to d-dimensional intermediate representation For each participant k, the instance similarity matrix constructed by its teacher network is As shown in formula (2).

[0098]

[0099] Where (·) T is the transpose operation, and ω is the temperature hyperparameter. Then, FCISL eliminates the diagonal elements, i.e. the similarity value of each sample with itself. The final instance similarity S k calculated by the teacher network of participant k is as shown in formula (3).

[0100]

[0101] Where Remove_diag(·) represents the operation of eliminating diagonal elements in the matrix.

[0102] Considering the computational and communication burden, each participant uploads the instance similarity S k to the server, and the server calculates the target instance similarity and sends the target instance similarity to each participant. In this process, all instance similarities apply gradient stop operation. Similar to FCSSL, after receiving , participant k matches the corresponding second positive sample pair k in S and Then, each participant updates its own teacher network by aligning the target instance similarity using CSSL. The FCISL loss of participant k (referred to as instance self-supervised loss) is as shown in formula (4).

[0103]

[0104] ​As shown in formula (4), the FCISL method aims to realize the representation level knowledge transfer of VFL, so that the model can capture more rich inter-domain knowledge from the representations of different participants, thereby improving the generalization ability and representation ability of the teacher network, while avoiding the generation of regularization conflicts.

[0105] As shown in formula (5), the aggregation of and in formula (1) is .

[0106]

[0107] Wherein μ is a hyperparameter that balances and .

[0108] (1.2) Each participant uses local private samples and its own teacher network to train its own local network.

[0109] In the local training of ISSVFL, each participant respectively uses its local private samples to pre-train its local network, so as to learn the intra-domain knowledge through the local network. At the same time, ISSVFL adopts a contrastive knowledge transfer (CKT) module to prevent the local network from overfitting to its local private data distribution.

[0110] For participant k, given a batch of local private samples , the data augmentation strategy is adopted to convert into corresponding first augmented view and second augmented view . Based on the same data augmentation strategy, different outputs are obtained according to the same input. In this embodiment, the local network is taken as a local online network, and a contrastive self-supervised learning method is adopted to obtain a local target network based on the local online network.

[0111] The first augmented view is input into the local online network, and the second augmented view is input into the local target network to generate a third positive sample pair. Specifically, the local online encoder f l k The first augmented view V1 k is converted into an intermediate representation , while the local target encoder converts the second augmented view into an intermediate representation . Then, the local online projector converts the intermediate representation into an online representation Local target projector intermediate representation Transform into target representation When using BYOL or SimSiam, local online predictors Further Transform into predictive representation Since each positive sample originates from the same sample, participant k can be directly matched. (or )and The corresponding positive sample pairs are used to obtain the third positive sample pair. (or ), for One of the elements, for One of the elements, for One of the elements.

[0112] The second enhanced view is input into the local online network, and the first enhanced view is input into the local target network to generate a fourth positive sample pair. Specifically, this embodiment uses a symmetric computation path, which swaps the first enhanced view V1. k Second Enhanced View To obtain as input to the local network (or and Participant k can also be directly matched. (or )and The corresponding positive sample pairs are used to obtain the fourth positive sample pair. (or ), for One of the elements, for One of the elements, for One of the elements.

[0113] The self-supervised loss is calculated as a symmetric loss based on the third and fourth positive sample pairs, as shown in Equation (6). Participant k minimizes the symmetric loss. To update its local online network (i.e., f) l k , or ).

[0114]

[0115] By minimizing Positive sample pairs attract each other, while negative sample pairs repel each other. For each participant, its local target network is a momentum version of the local online network, and the momentum update mode depends on the specific CSSL method.

[0116] The first enhanced view is input into the local online network, and the first enhanced view is input into the teacher network to generate a fifth positive sample pair; the second enhanced view is input into the local online network, and the second enhanced view is input into the teacher network to generate a sixth positive sample pair. The specific ISSVFL further adopts a CKT module, combines CSSL with knowledge transfer, and transfers the cross-domain knowledge in the teacher network to the local network while keeping the teacher network unchanged. Specifically, for each participant k, the CKT module generates a sixth positive sample pair V1 k and The frozen teacher network calculates the representations and Since each positive sample pair originates from the same enhanced view, participant k can directly match (or ) and the corresponding positive sample in (or ) to obtain the sixth positive sample pair (or ). Similarly, participant k can directly match (or ) and the corresponding positive sample in (or ) to obtain the fifth positive sample pair

[0117] (or ). Therefore, the CKT loss of participant k is as shown in formula (7).

[0118]

[0119] where λ is used to control the strength of CKT.

[0120] The local online network is updated by combining the contrast self-supervised learning loss and the contrast knowledge transfer loss, and CKT cooperates with the local SSL loss of formula (6) to balance the knowledge from other parties and the local, thereby improving the inter-domain generalization performance and the intra-domain discrimination performance of the local network. Therefore, as shown in formula (8), participant k updates the local online network, i.e., the local network, by minimizing the total local loss

[0121] (1.3) Each participant uploads the local network to the server, the server aggregates the local networks to generate a global network, and the global network is distributed to each participant as a new local network.

[0122] The model aggregation algorithm is crucial for VFL model training, which can integrate all the knowledge of the local network under the premise of protecting data privacy. ISSVFL adopts the commonly used aggregation mechanism FedAvg, so that the aggregated global encoder can generate a set of general representations that are easy for each participant to share.

[0123] Since the features held by each participant in the VFL environment are heterogeneous, participant k uploads the local network to the server for aggregation. Specifically, participant k first uploads the local encoder f l k to the server, which is divided into a participant-specific local bottom encoder and a local top encoder Then upload to the server. The server is responsible for aggregating the uploaded local top encoder, local projector and local predictor to obtain the global top encoder Global projector and global predictor Where n k is the number of samples owned by participant k, and N is the total number of samples owned by all participants. Finally, the server sends back to each participant as the new local network.

[0124] Where the division of the local encoder is the existing method, and the present embodiment does not limit it. For example, if the local encoder contains two fully connected layers, the last layer of the local encoder is divided into a local top encoder, and the remaining layers are divided into a local bottom encoder. Or, if the local encoder contains an embedding layer and two fully connected layers, the last layer of the local encoder is divided into a local top encoder, and the remaining layers are divided into a local bottom encoder. Or, if the local encoder is a ResNet-18 network, the last three modules of the local encoder are divided into a local top encoder, and the remaining layers are divided into a local bottom encoder.

[0125] (1.4) Re-execute steps (1.1)-(1.3) to perform the next round of training until the pre-training is completed. The present embodiment sets the pre-training completion condition as the iteration reaching a preset number of times.

[0126] (2) Fine-tuning phase: After the pre-training phase, participant k obtains the teacher encoder f T k and the local online encoder (i.e. local encoder) f l kFurther based on the teacher encoder and the local encoder of each participant, a joint federated learning model in the vertical direction is obtained, and the joint federated learning model in the vertical direction comprises a teacher network, a local network, a dynamic balance pool module and a classifier, wherein each participant has the teacher network, the local network and the dynamic balance pool module, and the active participant further has the classifier. The teacher network and the local network in each participant are arranged side by side, respectively receive input aligned and labeled samples, and input outputs to the dynamic balance pool module. The downstream supervision task is cooperatively performed by all participants using a small amount of aligned and labeled samples (i.e., X al ) to fine-tune the classifier held by the active participant to adapt to a specific downstream supervision task. The downstream supervision task process is shown in Figure 4 .

[0127] (2.1) Each participant respectively inputs the aligned and labeled samples into the teacher network and the local network, and aggregates the intermediate representations output by the teacher network and the intermediate representations output by the local network by using the dynamic balance pool module to obtain the final representation of each participant.

[0128] The teacher encoder of the participant k converts the input aligned and labeled samples of the participant k into an intermediate representation The local online encoder f of the participant k l k converts into an intermediate representation In order to provide the downstream supervision task with a representation containing more rich general information, the ISSVFL proposes a novel dynamic balance pool (DBP) module for dynamically balancing inter-domain and intra-domain knowledge in the VFL environment.

[0129] It should be noted that the same network output is represented by the same symbol in this embodiment, and the input is not distinguished, for example, the teacher encoder outputs to the aligned sample The output of the teacher encoder to the aligned and labeled sample is also represented as So as to facilitate understanding that they are all outputs of the teacher encoder In other embodiments, the same network can use different symbols to represent the output for different inputs, and this embodiment does not limit it.

[0130] As shown in Figure 5 , the DBP module converges and to provide the final representation R k to the downstream supervision task.

[0131] Specifically, as shown in equation (9), in the DBP module, the forgetting factor O k is calculated by the forgetting pool.

[0132]

[0133] where, represents the concatenation operation along the feature axis, σ(·) is the sigmoid function, and MLP(·) is the multi-layer perceptron.

[0134] Likewise, as shown in equation (10), the update factor U k is calculated by the update pool.

[0135]

[0136] The forgetting factor O k and the update factor U k are calculated in the same way, but the parameters of the multi-layer perceptron are different.

[0137] Then, as shown in equation (11), the implicit information I k is calculated by the implicit information extractor.

[0138]

[0139] where φ(·) is the activation function, FC(·) is the fully connected layer, and ⊙ is the Hadamard product.

[0140] As shown in equation (12), the final representation R k obtained by the participant k is calculated by the balancing pool of the DBP module.

[0141]

[0142] In the DBP module, the forgetting factor O k decides whether to ignore the knowledge from the teacher encoder. The update factor U k decides how much knowledge in the implicit information is retained to the final representation R k .

[0143] (2.2) The passive party sends the final representation to the active party for aggregation to obtain an aggregated representation; the active party converts the aggregated representation into a predicted value using a classifier, and trains the dynamic balancing pool module and the classifier according to the predicted value and the label.

[0144] Referring to the conventional VFL method, the passive party then sends the final representation R j (j∈[2,K]) to the active party for aggregation. As shown in equation (13), the unique classifier g c held by the active party converts the aggregated representation into a final predicted value

[0145]

[0146] wherein, denotes the aggregated representation after the aggregation.

[0147] Finally, as shown in equation (14), the global objective function is minimized by Fine-tuning can be performed collaboratively to train the dynamic balancing pool module and the classifier.

[0148]

[0149] wherein, l CE is the cross-entropy loss, is the corresponding real label held by the active party, and the first participating party is taken as an example in the embodiment. The final performance of the ISSVFL is evaluated after fine-tuning. If there is only one active party among the participating parties, the final representation of all passive parties can be sent to the active party for aggregation. If there are multiple active parties among the participating parties, the final representation of each passive party is sent to each active party respectively, each active party aggregates the final representation and calculates the loss respectively, and finally the loss calculated by all active parties is aggregated in proportion as the final loss, so as to realize collaborative fine-tuning.

[0150] (2.3) Repeat steps (2.1)-(2.2) to perform fine-tuning until fine-tuning is completed.

[0151] In order to demonstrate the advantages of the ISSVFL method of the present application, the following experiments are performed.

[0152] (1) Experimental setup.

[0153] The performance of the ISSVFL is evaluated on image data sets and table data sets.

[0154] (1.1) Dataset and model: To evaluate the performance of ISSVFL, experiments are conducted on four datasets, NUS-WIDE, Avazu, breast histopathology images (BHI) and ModelNet. The first two are tabular datasets, and the latter two are image datasets. For the NUS-WIDE dataset, this experiment selects data from 10 classes out of 81 available classes for the multi-class classification task. To simulate the VFL environment, one party holds the image features, and the other party holds the text features. For the Avazu dataset, this experiment randomly allocates 14 categorical features and 8 continuous features to the two parties to predict the click rate. In addition, the categorical features will be converted into 32-dimensional embeddings. Considering the computational burden, this experiment randomly selects 800,000 samples as the training set and 200,000 samples as the test set. For the BHI dataset, this experiment allocates two different images of the same patient to the two parties for the binary classification task. For the ModelNet dataset, this experiment selects 15 classes for the multi-class classification task, each class contains multiple 3D objects, and 12 images are generated for each object to represent different perspectives. To simulate the VFL environment, the 12 perspectives are evenly allocated to four parties, each receiving 3 adjacent perspectives. To increase the complexity of the dataset, this experiment randomly selects one image from each participating party to construct the VFL sample for each object.

[0155] The model architecture of each party on each dataset is shown in Table 1. All classifiers g c held by the active party are composed of a single fully connected layer (FC). For all datasets, all projectors are composed of three FCs, each with a dimension of 512. All predictors are composed of two FCs, with FC dimensions set to 128 and 512, respectively.

[0156] Table 1 Model architecture of different datasets

[0157] Dataset local encoder and teacher encoder (f l and f T ) Local top encoder f lt ]] NUS-WIDE 2 FCs f l the last layer of Avazu 1 Embedding layer + 2 FCs f l the last layer of BHI ResNet-18 f l the last three modules ModelNet ResNet-18 f l the last three modules

[0158] (1.2) Baseline methods: To comprehensively demonstrate the performance of the proposed ISSVFL, this invention compares ISSVFL with the following advanced methods in the VFL environment with label scarcity.

[0159] A, FedCVT is a semi-supervised learning method that expands the training dataset through representation estimation and pseudo-label prediction, thereby improving the performance of the VFL model in scenarios with limited aligned and labeled samples. FedCVT is designed specifically for the VFL scenario of two parties.

[0160] B, FedHSSL is a self-supervised learning framework that can utilize all available samples from each participant to achieve SOTA performance in label-scarce scenarios. FedHSSL integrates three representative CSSL methods, including BYOL, MoCo and SimSiam, to pretrain the VFL model of the participants. According to the difference of the integrated CSSL method, FedHSSL is extended to three different baseline models, namely FedHSSL-BYOL, FedHSSL-MoCo and FedHSSL-Simsiam.

[0161] For fair comparison, the proposed ISSVFL uses the same number of aligned and labeled samples as all baseline methods in both pre-training and fine-tuning stages. In the downstream supervised task, the number of aligned and labeled samples used for fine-tuning varies from 200 to 1000, and the average of five runs is taken as the experimental result.

[0162] (1.3) Evaluation index: For the NUS-WIDE dataset and the ModelNet dataset, the Top-1 accuracy of the classifier held by the active party is used to evaluate the final performance. For the Avazu dataset, the area under the ROC curve (AUC) is used as the evaluation index, and the smaller the AUC value represents the worse performance. For the BHI dataset, the F1-Score is used as the evaluation index.

[0163] (1.4) Data augmentation: For the NUS-WIDE dataset, 30% of the features are distorted by replacing the original values with random values. For the Avazu dataset, the continuous features are processed in the same way as the NUS-WIDE dataset, and the category features are replaced by additional untrained embedding vectors. For the BHI dataset and the ModelNet dataset, random resize and then crop to the input size, direct standardization, when the input size is 32, do not Gaussian blur; if the size is greater than 32, Gaussian blur with a probability of 50%.

[0164] (1.5) Implementation details: In ISSVFL, each participant trains locally using all of its local samples. Meanwhile, 40% of the dataset samples are used as aligned samples for co-training. Co-training and local training are alternated, and each training method can run for multiple cycles to minimize the communication cost. The pre-training rounds of all datasets are set to 40 in this experiment. The local encoders of ISSVFL and all baselines use the same model architecture under all datasets. In the fine-tuning stage, this experiment explores the performance of the jointly trained VFL model after fine-tuning at learning rates of 0.025 and 0.01, respectively, and selects the best results for presentation. Each MLP of the DBP module consists of two FCs. For the Avazu dataset, the DBP module is discarded in this experiment. For all datasets, the hyperparameter λ is set to 0.5. For the NUS-WIDE and Avazu datasets, the hyperparameters μ and ω are set to 2.0 and 0.03, respectively, and for the BHI and ModelNet datasets, the hyperparameters μ and ω are fixed to 0.5 and 0.01, respectively.

[0165] (2) Experimental results.

[0166] This experiment compares the performance of ISSVFL integrated with BYOL, MoCo, and SimSiam (i.e., baseline models ISSVFL-BYOL, ISSVFL-MoCo, and ISSVFL-SimSiam) with the baseline methods on the four datasets. The results are shown in Table 2.

[0167] Table 2 Comparison of performance of different methods on four datasets

[0168]

[0169]

[0170] The two-party scenario in the table represents a scenario with two participants, and the four-party scenario represents a scenario with four participants.

[0171] Studies have shown that all ISSVFL-based methods generally have significant performance improvements compared to FedCVT on all datasets. For example, when using 200 aligned and labeled samples, the best-performing ISSVFL method improved the performance on the NUSWIDE dataset by 13.3%, on the Avazu dataset by 5.0%, and on the BHI dataset by 10.0% compared to FedCVT. In the case of extremely limited aligned and labeled samples, the missing features inferred by FedCVT are more susceptible to noise interference, increasing the difficulty of accurately estimating missing features and reducing the overall effectiveness of semi-supervised methods.

[0172] The method based on FedHSSL shows robust performance in the scenario of label scarcity. However, FedHSSL simply applies the CSSL method, which is insufficient to exploit the potential of the SSL framework in VFL. In view of the above shortcomings, the FCISL method proposed by the application further combines CSSL with instance-level similarity, solves the regularization conflict problem, and captures important cross-domain knowledge. In addition, the DBP module proposed by the application can dynamically balance cross-domain and intra-domain knowledge, thereby generating more meaningful representations to further improve the performance of the joint VFL model. Experimental results show that, under the condition of using 200 aligned and labeled samples, the performance of ISSVFL-SimSiam is improved by 2.5% compared with FedHSSL-SimSiam on the NUS-WIDE dataset, and the performance is improved by 1.6% on the Avazu dataset. Similarly, the performance of ISSVFL-BYOL and ISSVFL-MoCo on the NUS-WIDE and Avazu datasets is also better than that of FedHSSL-BYOL and FedHSSL-MoCo.

[0173] It is found that when more aligned and labeled samples are used in the fine-tuning stage, the performance of ISSVFL on the BHI dataset is significantly improved. Therefore, the FCISL method of ISSVFL improves the performance of the joint VFL model on the image dataset, while avoiding the regularization conflict with the CSSL method. In addition, the performance of ISSVFL on the ModelNet dataset is also excellent, which verifies its adaptability to multi-party scenarios, that is, as the number of participants increases, ISSVFL can still effectively deal with the label scarcity problem in VFL.

[0174] In most cases, when more aligned and labeled samples are used, the performance of ISSVFL improves faster than that of the baseline method, indicating that it can capture cross-domain knowledge in the representation of different participant models and can more efficiently transfer knowledge to the local network.

[0175] (3) Ablation experiment.

[0176] To evaluate the contribution of each module of ISSVFL to the overall performance, different variants created by removing modules are compared in this experiment. Specifically, “-” means no module is removed, “w / o C” means the co-training step in the pre-training phase (i.e., step 1.1) is removed, “w / o F” means the FCISL method in the pre-training phase is removed, “w / o D” means the DBP module in the fine-tuning phase is removed, and “w / o DF” means both the DBP module and the FCISL method are removed. The ablation experiments are conducted for ISSVFL-BYOL, ISSVFL-MoCo, and ISSVFL-SimSiam, respectively, using 200 aligned and labeled samples for fine-tuning. Table 3 lists the average results of each variant in five experiments. To minimize the impact of randomness, this experiment designates 30% of the dataset samples as aligned samples for collaborative training.

[0177] Table 3 Performance comparison of various variants on different datasets

[0178]

[0179] The results show that the method based on the complete ISSVFL always performs the highest accuracy, and removing any module will cause a performance drop. It is particularly noteworthy that removing the co-training step will cause a significant performance drop on all datasets, indicating that the co-training step has a very important impact on model performance. For each CSSL method, removing the DBP and FCISL methods causes a performance drop of about 2% on the NUS-WIDE dataset, thus verifying the effectiveness of the two modules.

[0180] The FCISL method significantly improves the final performance except for the slight drop in the performance of ISSVFL-MoCo on the Avazu dataset. Thus, in most ablation experiments, the FCISL method captures the implicit cross-domain knowledge from the representations of different party models, thus enhancing the representation capability of the local model. The learnable DBP module also plays a positive role in improving the final performance. For each CSSL method, the variant that removes the DBP module has a slight performance drop on all datasets, indicating that the DBP module can dynamically balance cross-domain knowledge and intra-domain knowledge and provide representations rich in general information for downstream supervised tasks.

[0181] In another embodiment, a self-supervised longitudinal federated learning device based on instance similarity and dynamic balance pool is provided, which comprises K participants and a server, and the K participants and the server collaboratively train a joint longitudinal federated learning model, the joint longitudinal federated learning model comprising a teacher network, a local network, a dynamic balance pool module and a classifier, the K participants with labels as active parties and the rest as passive parties, the K participants and the server performing the following operations:

[0182] In the pre-training phase:

[0183] Each participant collaboratively trains its own teacher network using aligned samples, and in the collaborative training, instance similarity is calculated based on the output of the teacher network, and the loss is calculated in combination with the instance similarity;

[0184] Each participant trains its own local network using local private samples and its own teacher network;

[0185] Each participant uploads the local network to the server, the server aggregates each local network to generate a global network, and the global network is distributed to each participant as a new local network;

[0186] Each participant and the server repeats the pre-training phase until the pre-training is completed;

[0187] In the fine-tuning phase:

[0188] Each participant respectively inputs the aligned and labeled samples into its own teacher network and local network, and aggregates the intermediate representations output by the teacher network and the intermediate representations output by the local network using the dynamic balance pool module to obtain its own final representation;

[0189] The passive party sends the final representation to the active party for aggregation to obtain an aggregated representation;

[0190] The active party converts the aggregated representation into a predicted value using the classifier, and trains the dynamic balance pool module and the classifier according to the predicted value and the label;

[0191] Each participant and the server repeats the fine-tuning phase until the fine-tuning is completed.

[0192] The specific limitations of the self-supervised longitudinal federated learning device based on instance similarity and dynamic balance pool can be referred to the limitations of the self-supervised longitudinal federated learning method based on instance similarity and dynamic balance pool in the above, which will not be repeated here.

[0193] Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as there is no conflict, any combination of the technical features should be considered within the scope of the present disclosure.

[0194] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A self-supervised longitudinal federated learning method based on instance similarity and dynamic balanced pool, used for training a joint longitudinal federated learning model by one server and one participant, characterized in that, The joint longitudinal federated learning model comprises a teacher network, a local network, a dynamic balance pool module and a classifier, The self-supervised longitudinal federated learning method based on instance similarity and dynamic balance pool comprises a pre-training stage and a fine-tuning stage, wherein: The pre-training stage comprises: Each participant uses the aligned image samples to collaboratively train the respective teacher network, and in the collaborative training, instance similarity is calculated based on the output of the teacher network, and a loss is calculated in combination with the instance similarity; Each participant uses the local private image samples and the respective teacher network to train the respective local network; Each participant uploads the local network to the server, the server aggregates the local networks to generate a global network, and the global network is distributed to each participant as a new local network; The pre-training stage is repeatedly executed until the pre-training is completed; The pre-training stage comprises: The teacher network comprises a teacher encoder and a teacher projector, the aligned image samples are input into the teacher encoder to be converted into a teacher intermediate representation, and then the teacher intermediate representation is input into the teacher projector to be converted into a teacher final representation; A first positive sample pair is generated based on the teacher final representation, and a representation self-supervised loss is calculated according to the first positive sample pair; Instance similarity of each participant is calculated based on the teacher intermediate representation, a second positive sample pair is generated according to the instance similarity, and an instance self-supervised loss is calculated according to the second positive sample pair; The representation self-supervised loss and the instance self-supervised loss are aggregated as a collaborative training final loss, and the teacher network is updated according to the collaborative training final loss; The fine-tuning stage comprises: Each participant respectively inputs the aligned and labeled image samples into the respective teacher network and local network, and aggregates the intermediate representations output by the teacher network and the intermediate representations output by the local network by using a dynamic balance pool module to obtain the respective final representation; The passive party sends the final representation to the active party for aggregation to obtain an aggregated representation; The active party converts the aggregated representation into a predicted value by using a classifier, and trains the dynamic balance pool module and the classifier according to the predicted value and the label; The fine-tuning stage is repeatedly executed until the fine-tuning is completed, and a trained joint longitudinal federated learning model is output; The fine-tuning stage comprises: taking the participant intermediate representation of the teacher network output , and the participant intermediate representation of the local network output computing the forgetting factor as follows: ； wherein denotes a concatenation operation along the feature axis, is a multi-layer perceptron, is a sigmoid function; Computing update factors As follows: ； Extracting implicit information As follows: ； wherein is an activation function, is a fully connected layer, is a Hadamard product; The final representation is calculated as follows: ； In the formula, parties The resulting final characterization.

2. The self-supervised longitudinal federated learning method based on instance similarity and dynamic balanced pool according to claim 1, wherein, The fine-tuning stage comprises: taking a participant final representation of the teacher ; Each participant uploads the final representation of the teacher to the server, which computes the average teacher representation and distributes the average teacher representation to each participant; The average teacher representation is received by each participant The teacher final representation converted based on the same sample and the average teacher representation The matching is taken as a first positive sample pair, and the self-supervised loss is calculated based on the first positive sample pair as the representation self-supervised loss.

3. The self-supervised longitudinal federated learning method based on instance similarity and dynamic balanced pool according to claim 1, wherein, The fine-tuning stage comprises: Take Participants Teacher intermediate representation Building Participants Instance similarity matrix as follows: ； wherein is a transpose operation, is a temperature hyperparameter, is an L2 norm; Obtaining Participants Example Similarity As follows: ； wherein denotes the operation of eliminating the diagonal elements of the matrix; Each participant will upload instance similarities to the server, which will compute target instance similarities and issue target instance similarities to each participant; Each participant receives the target instance similarity After, the instance similarity obtained based on the same sample conversion And the target instance similarity Match as a second positive sample pair, calculate the self-supervised loss based on the second positive sample pair as the instance self-supervised loss.

4. The self-supervised longitudinal federated learning method based on instance similarity and dynamic balanced pool according to claim 1, wherein, The fine-tuning stage comprises: The local network is used as a local online network, and a contrastive self-supervised learning method is used to obtain a local target network based on the local online network; A data enhancement strategy is used to convert the local private image samples into a first enhanced view and a second enhanced view; The first enhanced view is input into the local online network, and the second enhanced view is input into the local target network to generate a third positive sample pair; The second enhanced view is input into the local online network, and the first enhanced view is input into the local target network to generate a fourth positive sample pair; A self-supervised loss is calculated based on the third positive sample pair and the fourth positive sample pair as a symmetric loss; inputting the first enhanced view into the local online network, inputting the first enhanced view into the teacher network to generate a fifth positive sample pair; inputting the second enhanced view into the local online network, inputting the second enhanced view into the teacher network to generate a sixth positive sample pair; calculating a self-supervised loss based on the fifth positive sample pair and the sixth positive sample pair as a contrastive knowledge transfer loss; updating the local online network in combination with the symmetric loss and the contrastive knowledge transfer loss.

5. The self-supervised longitudinal federated learning method based on instance similarity and dynamic balanced pool according to claim 1, wherein, The local network includes a local encoder and a local projector, and each participant divides the local encoder into a local bottom encoder and a local top encoder. Each participant uploads the local top encoder and the local projector to the server. The server aggregates the local top encoders to obtain a global top encoder, aggregates the local projectors to obtain a global projector, and takes the global top encoder and the global projector as the global network. The local encoder includes two fully connected layers, and the last layer of the local encoder is divided as the local top encoder and the remaining layers are divided as the local bottom encoder.

6. The self-supervised longitudinal federated learning method based on instance similarity and dynamic balanced pool according to claim 5, characterized in that, Alternatively, the local encoder includes an embedding layer and two fully connected layers, and the last layer of the local encoder is divided as the local top encoder and the remaining layers are divided as the local bottom encoder. Alternatively, the local encoder is a ResNet-18 network, and the last three modules of the local encoder are divided as the local top encoder and the remaining layers are divided as the local bottom encoder. In the pre-training phase:

7. An apparatus for self-supervised longitudinal federated learning based on instance similarity and dynamic balanced pool, characterized in that, The self-supervised longitudinal federated learning device based on instance similarity and dynamic balance pool comprises a server, the The server and the participants jointly train a joint longitudinal federated learning model, the joint longitudinal federated learning model comprising a teacher network, a local network, a dynamic balance pool module and a classifier, The participants with labels are active parties, and the rest are passive parties, The server and the participants perform the following operations: Each participant uses the aligned image samples to collaboratively train their own teacher network, and in the collaborative training, the instance similarity is calculated based on the output of the teacher network, and the loss is calculated in combination with the instance similarity. Each participant uses the local private image samples and their own teacher network to train their own local network. Each participant uploads the local network to the server, and the server aggregates the local networks to generate a global network, and then distributes the global network to each participant as a new local network. Each participant and the server repeat the pre-training phase until the pre-training is completed. The teacher network includes a teacher encoder and a teacher projector, and the aligned image samples are input into the teacher encoder to convert into a teacher intermediate representation, and then the teacher intermediate representation is input into the teacher projector to convert into a teacher final representation. Based on the teacher final representation, a first positive sample pair is generated, and a representation self-supervised loss is calculated according to the first positive sample pair. Based on the teacher intermediate representation, the instance similarity of each participant is calculated, a second positive sample pair is generated according to the instance similarity, and an instance self-supervised loss is calculated according to the second positive sample pair. The representation self-supervised loss and the instance self-supervised loss are aggregated as a collaborative training final loss, and the teacher network is updated according to the collaborative training final loss. In the fine-tuning phase: Each participant inputs the aligned and labeled image samples into their own teacher network and local network respectively, and uses a dynamic balance pool module to aggregate the intermediate representations output by the teacher network and the local network to obtain their own final representation. ​ ​ The passive party sends the final representation to the active party for aggregation to obtain an aggregated representation; The active party converts the aggregated representation into a prediction value by using the classifier, and trains the dynamic balance pool module and the classifier according to the prediction value and the label; The participating parties and the server repeatedly perform the fine-tuning stage until the fine-tuning ends, and output a trained joint longitudinal federated learning model; The intermediate representation output by the teacher network and the intermediate representation output by the local network are aggregated by using the dynamic balance pool module to obtain respective final representations, including: taking the participant intermediate representation of the teacher network output , and the participant intermediate representation of the local network output , computing the forgetting factor as follows: ； wherein denotes a concatenation operation along the feature axis, is a multi-layer perceptron, is a sigmoid function; Computing update factors As follows: ； Extracting implicit information As follows: ； wherein is an activation function, is a fully connected layer, is a Hadamard product; The final representation is calculated as follows: ； In the formula, parties The resulting final characterization.

Images