Federal learning method and system based on domain-invariant text representation and global prior in domain

Abstract

The application discloses a kind of federal learning method and system based on domain invariant text representation and domain global prior, by introducing text embedding as global representation, using the class label of natural language description to generate stable, unbiased, and have the global feature representation of domain invariance, by the mutual guidance of local feature and text embedding vector, can effectively reduce intra-class distance and expand inter-class distance, to alleviate data heterogeneity problem.Meanwhile, the introduced domain prior module generates global sample prior by aggregating observable data embedding, which can help local model better understand global data distribution.Compared with the global model or prototype generated by traditional image features, the method based on text embedding and domain global prior does not depend on local data quality, and naturally has denoising and generalization ability, effectively alleviating the problem that global target changes with training.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of distributed machine learning, and relates to a personalized federated learning method and system based on domain-invariant text representation and intra-domain global prior, which is a personalized federated learning method combined with intra-domain global prior, and is especially suitable for optimization of domain-invariant text representation in a cross-client data distribution heterogeneous scene. BACKGROUND

[0002] With the increasing demand for data privacy protection, federated learning (FL) as a distributed machine learning technology that can realize collaborative modeling without centralized data has attracted more and more attention. This technology allows multiple devices or organizations to jointly train a global model without sharing raw data, and is widely used in medical, financial and mobile device scenarios with high requirements for data privacy.

[0003] Traditional federated learning methods such as FedAvg aggregate local model parameters by weighted averaging to build a global model, thereby realizing collaborative learning across devices or organizations. However, due to the significant statistical heterogeneity (non-IID) of data distribution among clients, direct aggregation of local models often leads to a decline in the performance of the global model, and poor performance on some clients (Kairouz P, McMahan H B, Avent B, et al. Advances and open problems in federated learning [J]. Foundations and trends® in machine learning, 2021, 14(1-2): 1-210).

[0004] To solve this problem, personalized federated learning (pFL) has become a research hotspot in recent years. This type of method aims to learn the optimal personalized model for each client based on sharing global information. Existing personalized federated learning methods mainly guide local model learning through the following two types of means: one is to use the global model as a regularization term to constrain the local model (such as pFedMe, Ditto), and the other is to use global prototypes to align local feature distribution (such as FedProto, FedPHP).

[0005] However, these existing methods based on global models or prototypes often have the following shortcomings (such as Figure 1The problems are: (a) and (b): 1. Global information is easily affected by dominant clients: the global model or prototype is usually an aggregation of different distribution client models, which is easy to be biased towards the dominant client with large data volume or good quality, thereby becoming an unfair convergence target; 2. High risk of misleading local models: uploading of low-quality or noisy local models will affect the quality of global information, and then mislead the training of other clients; 3. Convergence target changes constantly: since each round of aggregation is based on the latest local model, the distribution of the global model or prototype changes constantly during collaborative training, so that the local model is always optimized towards a dynamic target, which is difficult to stabilize.

[0006] These problems limit the application effect and generalization ability of the personalized federated learning method in the non-independent and identically distributed scene. Therefore, it is urgent to provide a stable, unbiased and domain-invariant global guidance mechanism to improve the performance of local models and enhance the stability and fairness of the system. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a personalized federated learning method and system based on domain-invariant text representation and intra-domain global prior. The method introduces text embedding as a global representation, and generates stable, unbiased and domain-invariant global feature representation using the category label described by natural language. Compared with the traditional global model or prototype generated by relying on image features, the text embedding does not depend on the quality of local data, and naturally has the ability to denoise and generalize, effectively alleviating the problem of changing global target with training.

[0008] The technical scheme adopted by the present application is as follows:

[0009] A personalized federated learning method based on domain-invariant text representation and intra-domain global prior, comprising introducing domain-invariant text representation and intra-domain prior module in the process of training local data to obtain local model at each client, specifically:

[0010] In the training, image and text set dual-modal input is used, the input image is extracted to obtain image features through the image encoder participating in the training, and the text set is extracted to obtain training text embedding and frozen text embedding through the text encoder participating in the training and the frozen text encoder, respectively. In this process, the matching between the image features and the text embedding features is optimized by using the contrast loss of image-text alignment; wherein the frozen text encoder receives the global parameters issued by the server and locks the update;

[0011] The training text is embedded in the domain prior module to calculate the domain prior. The domain prior and image features are added and input into the trained classifier. At the same time, the image features are input into the frozen classifier separately. The results of the trained classifier and the frozen classifier are added and output as the prediction result. This process is supervised by the task loss.

[0012] The frozen text encoder and frozen classifier are respectively the global text encoder and global classifier downloaded from the server after the previous iteration step, and are locked for update in the current iteration step.

[0013] In the above technical solution, further, the contrast loss of the image-text alignment It consists of two symmetric log-negative probability terms, where and Represent the normalized similarity probabilities based on frozen text embedding and training text embedding respectively. middle is the ratio of the two parts, where the numerator is , represents the image features Frozen text embedding with similar tags The negative exponential similarity between ; the denominator is composed of repeated numerators ( ) and the negative sample summation term, the negative sample summation term ( ) represents the image features and all heterogeneous frozen text embeddings The second term is the sum of the similarities of middle A structure symmetrical to the first one, but with different parameters, using image features that do not participate in gradient updates , text embedding uses trainable text embedding and . Its molecular parts are: , indicating image features that do not participate in the update Training text embeddings with the same labels as image features The negative exponential similarity of ; its denominator is composed of repeated numerators ( ) and the negative sample summation term, the negative sample summation term ( ) indicates image features that do not participate in the update With all heterogeneous training text embeddings The sum of the similarities. That is:

[0014] .

[0015] in, is the image feature, is the frozen text embedding of the same label as the image feature, frozen text embeddings for image features with different class labels, image features not participating in the update, training text embeddings for image features with the same class label, training text embeddings for image features with different class labels, a similarity function denoted as: .

[0016] Further, the intra-domain prior module calculates the intra-domain prior using the training text embeddings and the data distribution of each class in the training set, specifically:

[0017] convert each class label in the global label set into a standardized text description, and generate a corresponding set of text embedding vectors through the first client: where represents the text embedding representation of the first client corresponding to the first class.

[0018] By combining the proportion of each class sample in the local data in the number of local data classes, the text embedding representation in the text embedding vector set is weighted and averaged to obtain the intra-domain prior of the client.

[0019] Further, the intra-domain prior is:

[0020] ;

[0021] where is an indicator function used to determine whether the label of the sample belongs to the class , is the number of classes present in the local data.

[0022] Further, the task loss is , which measures the matching degree between the one-hot encoding vector of the real label and the model prediction output by calculating the cross entropy between them. The model prediction output is obtained by the sum of two parts: the first part is the classifier result in the training process, and the second part is the result of a frozen classifier . The outputs of these classifiers are normalized by the softmax function to generate the final probability distribution. That is:

[0023]

[0024] wherein, is a one-hot encoding vector of the true label, denotes a softmax function, is a result of the trained classifier, is a result of the frozen classifier.

[0025] Further, each client trains locally aims to minimize the sum of the contrastive loss and the task loss of the image-text alignment.

[0026] Further, after the current iteration step is completed, all clients upload the locally trained models to the server, including the text encoder participating in training, the image encoder participating in training, and the trained classifier, the server calculates the corresponding average model respectively to obtain the global model, including the global text encoder, the global image encoder, and the global classifier, and returns to each client, the client initializes the text encoder participating in training and the frozen text encoder to the global text encoder, initializes the image encoder to the global image encoder, initializes the classifier participating in training and the frozen classifier to the global image classifier, and the frozen text encoder and the frozen classifier are locked and not updated in the local training process, and the loop iteration is performed.

[0027] The application also provides a personalized federated learning system based on domain-invariant text representation and intra-domain global prior, which is used to implement the method as any one of the above.

[0028] The application also provides an electronic device, comprising:

[0029] one or more processors;

[0030] a memory for storing one or more programs;

[0031] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as any one of the above.

[0032] The application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the method as any one of the above.

[0033] The application has the following beneficial effects:

[0034] The method of the application converts the data label into a text description, and embeds the text into a vector as a global representation. These embedding vectors are unbiased and do not favor any particular client. Through mutual guidance of local features and text embedding vectors, the intra-class distance can be effectively reduced and the inter-class distance can be expanded, thereby alleviating the data heterogeneity problem. The text embedding distribution remains stable in the collaborative learning process, providing a consistent goal for model optimization. This stability, combined with the anti-noise and domain-invariant characteristics of the text embedding, makes it an ideal global representation. At the same time, the introduced intra-domain prior module generates a global sample prior by aggregating observable data embeddings, which can help the local model better understand the global data distribution. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is the optimization process and distribution of different global representations: (a) the global prototype representation is easily disturbed by noisy data, and the distribution changes significantly during collaborative training; (b) the global model representation also produces a continuously changing distribution during collaborative learning; (c) as a global representation, the text embedding is not only not affected by noisy data, but also maintains stable distribution characteristics during training.

[0036] Figure 2 is the training structure framework (left) and communication mechanism (right) in the method of the application.

[0037] Figure 3 is the flowchart of the intra-domain prior module in the method of the application.

[0038] Figure 4 is the test accuracy change curve of different methods in the communication round in the embodiment of the application. DETAILED DESCRIPTION

[0039] The specific embodiments of the embodiments of the application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the embodiments of the application, and are not used to limit the embodiments of the application.

[0040] The personalized federated learning method based on domain-invariant text representation and intra-domain global prior provided by the application is a domain-invariant text representation optimization method combined with intra-domain global prior, especially suitable for cross-client data distribution heterogeneous scenarios, referred to as FedDTR. This method is based on text embedding as a global representation, and has unique advantages compared to other methods of different global representations, such as Figure 1As shown in FIG. 1, (a) global prototype representation is vulnerable to noisy data and its distribution changes significantly during the co-training process; (b) global model representation also produces a continuously changing distribution during the co-learning process; (c) text embedding as global representation is not only immune to noisy data, but also maintains stable distribution characteristics during the training process. The technical implementation of the FedDTR framework (its training structure and communication process are shown in FIG. 2) is described in detail as follows. Figure 2

[0041] In view of the defects of the existing method, the present application innovatively introduces two core components: domain-invariant text representation and intra-domain prior module. Text embedding guides local feature extraction through a dual mechanism: on the one hand, it promotes the closer aggregation of similar sample features, and on the other hand, it expands the distance between different class features. At the same time, these text embeddings will be dynamically adjusted to optimize the mapping from the text space to the image space.

[0042] The intra-domain prior module provides each client with domain-specific global prior knowledge by integrating local samples and text embeddings. This design has dual advantages: 1) enhances the model's ability to recognize global data distribution; 2) effectively suppresses the overfitting phenomenon of personalized models on local data. Based on the functional difference between the feature extractor and the classification head , the present application adopts a parameter shunting strategy: uploading the trained classification head parameters to the server, and downloading the global classification head parameters to the frozen head , thereby achieving efficient fusion of global information.

[0043] Domain-invariant text representation:

[0044] This part focuses on how to build a fair and stable global convergence target through unbiased and domain-invariant text embedding. FedDTR adopts a dual-modal input: where represents the input image of the th client, is a set of texts (such as corresponding to the sentence "This is a cat", using natural language to describe the class label). The processing flow is as follows:

[0045] 1. Feature extraction stage: image features: generate through the encoder . Text features: tokenize the text set into a vector set (C is the number of classes, and d is the vector dimension), and then output the embedding vector through the text encoder .

[0046] ​2. Joint optimization objective: design contrastive loss function Achieve double optimization: align image features with same-class text embeddings (reduce intra-class distance) and map text embeddings to image space (enhance class representativeness). The similarity calculation is defined as .

[0047] 3. Stability enhancement mechanism: to solve the problem of text embedding fluctuation in the training process, introduce a frozen text encoder : receive global parameters issued by the server and lock updates. Generate global text embeddings consistent across clients .

[0048]

[0049] where is the image feature that does not participate in gradient update.

[0050] Intra-domain prior module:

[0051] In the process of federated learning, the data distribution of each client is usually independent and inconsistent with each other, that is, the data of each client comes from different data domains. For the th client, the label set of its training data is denoted as , where the global label set is , and the local data set of the client is , and the corresponding label set is .

[0052] In traditional personalized federated learning methods, the task of each local model can be expressed as , that is, the model only classifies and predicts the label categories contained in the local. In order to realize collaborative training across clients, the local model task needs to be expanded to , that is, prediction is made in the entire global label space. However, this expansion will change the classification structure of the model, so that the local model cannot focus on its original subtask, affecting the training effect.

[0053] To solve the above problems, the present application proposes a task guiding mechanism based on intra-domain prior, which helps the local model understand the overall data distribution by introducing intra-domain prior, alleviates the influence of task structure change, and improves the personalized modeling ability.

[0054] The specific technical solutions are as follows:

[0055] 1. Introduction of text embedding representation: first, each label in the global label set is converted into a standardized text description to generate a corresponding text embedding vector set: . Where represents the one client corresponds to the first class of text embedding representation.

[0056] 2. Construction of intra-domain prior representation: the present application obtains the intra-domain prior representation of the client by weighted average of the text embedding in combination with the proportion of samples of each category in the local data , and the calculation formula is as follows:

[0057]

[0058] wherein is an indicator function for judging whether the sample belongs to the category , is the number of categories existing locally. The global prior represents the global prior of the category distribution in the client domain, and by inputting the prior and the image feature into the model, the local model can obtain global information from a specific perspective and improve the adaptability to the overall task.

[0059] Target optimization:

[0060] 1. Double classification head structure design: in order to retain the local personalization ability while utilizing the global knowledge to guide the training, the present application divides the model classifier into a local head and a frozen head : the local head receives local training updates; the frozen head remains as the global classifier parameters downloaded from the server and is not updated during the training process. Both participate in prediction simultaneously during inference, and the individualization and versatility are realized by fusion output.

[0061] 2. Design of local optimization objective function: the overall loss function includes two parts, the contrast loss of image-text alignment , used for optimizing the matching between image features and text embedding; and the task loss , used for training the model classifier, and defined as follows:

[0062] wherein represents a softmax function, is a one-hot encoding vector of the real label.

[0063] The final goal of local training is to minimize the following comprehensive loss: .

[0064] 3. Communication and collaboration strategy: in each round of communication, the server and the client exchange the text encoder, the image encoder and the classifier parameters, and the unified frozen model is used for global alignment across clients to ensure the consistency of global information.

[0065] According to a specific embodiment of the present application, the training process and the inference process using the method of the present application are as follows:

[0066] Training process: When a batch of training data (images and corresponding labels) is input, first, the image encoder is used to obtain the image features, and then the corresponding labels are converted into natural language text descriptions. The formed text set is first encoded into a text vector set using word2vec, and the text vector set is simultaneously input into the text encoder participating in training and the text encoder with frozen parameters to obtain the training text embedding and the frozen text embedding. The image features and the frozen text embedding are added one by one according to , so that the image features are pulled closer to the text embedding of the corresponding label and are pushed away from the text embedding of other labels. The training text embedding and the image features make the corresponding training text embedding closer to the image features and the other text embedding farther away from the image features.

[0067] The intra-domain prior module calculates the intra-domain prior using the training text embedding and the data distribution of each category (the percentage of each class) calculated from the training set. The intra-domain prior is added one by one to the image features and input into the training classifier, and the image features are separately input into the frozen classifier. The results of the two classifiers are added to obtain the final result, and the final result is supervised by the target function training.

[0068] After each training is completed, the client uploads the text encoder participating in training, the image encoder, and the image classifier participating in training to the server. The server calculates the average model corresponding to each model as the global model, and the server returns the global text encoder, the global image encoder, and the global image classifier to the client. The client initializes the text encoder participating in training and the frozen text encoder as the global text encoder, initializes the image encoder as the global image encoder, and initializes the image classifier participating in training and the frozen image classifier as the global image classifier. The loop iteration is performed.

[0069] Inference process: input an image through the trained image encoder to obtain the image features, and input the text vector set through the trained text encoder to obtain the text embedding. The intra-domain prior is calculated by the intra-domain prior module, and the intra-domain prior and the image features are added one by one and input into the trained classifier. The image features are separately input into the frozen classifier, and the results of the two classifiers are added to obtain the final prediction result.

[0070] Experimental verification

[0071] To verify the performance of the method FedDTR in effectiveness, scalability, stability and convergence speed, the present embodiment carries out system comparison experiments with 13 current mainstream personalized federated learning methods based on multiple public image classification and natural language processing tasks, and designs ablation experiments to analyze the contribution of each sub-module.

[0072] I. Experimental setup

[0073] (1) Comparison methods. The selected comparison methods cover the following categories: traditional federated learning methods: FedAvg, FedProx; global and local structure decoupling methods: FedPer, FedRoD, FedRep; personalized methods based on global model reference: Ditto, pFedMe; personalized methods based on global prototype reference: FedProto, FedPHP; other personalized methods: Per-FedAvg, FedFomo, FedAMP, FedALA. The method FedDTR is compared with the above methods in all task scenarios according to the same training round and communication strategy.

[0074] (2) Dataset and task division. Computer vision tasks use 5 public image classification datasets: MNIST, Fashion-MNIST (FMNIST), CIFAR-10, CIFAR-100, Tiny-ImageNet; natural language processing tasks use 2 text classification datasets: AG News, Amazon Review.

[0075] (3) Model structure and learning rate setting. MNIST, FMNIST, CIFAR-10, CIFAR-100, Tiny-ImageNet use a 4-layer convolutional neural network (CNN) as the base model; to test scalability, ResNet-18 is also used as a deep network on Tiny-ImageNet; for AG News and Amazon Review, fastText and 3-layer multi-layer perceptron (MLP) are used respectively; the setting of local learning rate is: the CNN and MLP structure is set to 0.005, and the ResNet-18 and fastText are set to 0.1.

[0076] (4) Heterogeneity setting. To simulate the real federated learning environment, two typical data Non-IID settings are designed in the experiment: Pathological Setting (pathological setting): the client only contains a small number of categories, the number of samples is uneven and there is no overlap; MNIST / CIFAR-10 / CIFAR-100 is allocated 2 / 2 / 10 categories respectively; Practical Setting (practical setting): the Dirichlet distribution is used to sample to control the unevenness of the data category distribution of the client; for the category , the probability of its sample allocation to the client is , and the default parameters . The involved data sets strictly control the sample number and label unevenness to test the robustness of the model to the heterogeneity scene.

[0077] (5) Training strategy and running environment. The total number of clients is set to 20, and the default is to participate in all rounds (participation rate (J =1.0)); 75% of the local data of each client is used for training, and 25% is used for testing; the local training batch size is 10, and the number of local iterations per round is set to 1; the total communication round is 2000 rounds; all methods are implemented based on PyTorch 1.7, and run on an Ubuntu 16.04 server with the following configurations: dual-channel Intel Xeon Silver 4210 CPU; 256GB memory; 8 NVIDIA 2080 Ti graphics cards.

[0078] II. Experimental results

[0079] (1) To comprehensively verify the adaptability and superior performance of the FedDTR method of the present application in different heterogeneity environments, experiments are carried out in pathological setting and practical setting. In the pathological setting, MNIST, CIFAR-10 and CIFAR-100 are selected, and each client only has a small number of non-overlapping category labels; in the practical setting, MNIST, CIFAR-10, CIFAR-100, Tiny-ImageNet and AG News are selected, and Dirichlet distribution is used to control the label distribution unevenness to simulate a more realistic federated environment.

[0080] In the above two settings, FedDTR is compared with 13 mainstream personalized federated learning methods. The experimental results show that the method FedDTR obtains the optimal performance in all tasks and scenarios, and in the image and text classification tasks, the accuracy is significantly higher than that of the traditional method and the existing personalized strategy, and the method FedDTR shows good convergence, stability and generalization ability. In the basic visual tasks such as MNIST and CIFAR-10, the method FedDTR is superior to the traditional methods such as FedAvg and FedProx; in the complex class tasks such as CIFAR-100 and Tiny-ImageNet, the method FedDTR significantly outperforms the personalized methods such as FedPer and FedProto; in the AG News text classification task, the method FedDTR performs best in accuracy and stability.

[0081] The above results fully show that the method FedDTR has excellent cross-scene adaptability and can effectively solve the key technical problems of performance degradation and optimization instability existing in the existing personalized federated learning methods. The specific results are shown in the following table:

[0082]

[0083] To further verify the accuracy and convergence of the method FedDTR in different tasks, experiments are conducted in the AmazonReview text classification task and the Fashion-MNIST (FMNIST) image classification task, and the same parameter settings are adopted. In the experiment, the test accuracy changes of each method in the communication round are recorded, and the convergence curves are drawn as shown in Figure 4

[0084] The results show that in the AmazonReview and FMNIST tasks, the method FedDTR always maintains the highest test accuracy in the whole training process, and has fast convergence speed and small fluctuation, showing good stability; in contrast, the method FedProto shows the slowest convergence speed and the lowest final accuracy in the two tasks, indicating that its adaptability to non-independent and identically distributed data is limited. The above experimental results further prove that the method FedDTR proposed in the application has consistent and superior performance in different types of tasks, and can effectively improve the robustness and convergence efficiency of the personalized federated learning system.

[0085] ​Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, the methods can be tangibly embodied in a machine-readable storage medium having stored thereon instructions that can be used to program a processing system to perform the methods. The machine-readable storage medium can be magnetic (e.g., magnetic disks), optical (e.g., optical discs), electrical (e.g., memory devices), or any combination thereof. Such software can be transmitted using any apparatus adapted to transfer a set of instructions from one place to another. A machine-readable storage medium is a computer-related media that stores computer-readable instructions, data structures, program modules or other data. Examples of computer- related media include RAM, ROM, EEPROM, CD-ROM or any other storage device.

[0086] The present application is described in reference to the flowchart and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0087] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0089] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and operation described. Accordingly, all such variations are intended to fall within the scope of the present application.

Claims

1. A personalized federated learning method based on domain-invariant text representation and domain-wide global priors, characterized by: This involves introducing domain-invariant text representation and domain prior modules in the process of training local data on each client to obtain a local model. Specifically: The training process uses bimodal input of images and text collections. The input image is extracted through the image encoder participating in the training to obtain image features, and the text collection is extracted through the text encoder participating in the training and the frozen text encoder to obtain training text embedding and frozen text embedding respectively. In this process, the contrast loss of image-text alignment is used to optimize the matching between image features and text embedding features. The frozen text encoder receives global parameters sent by the server and locks the update. The training text is embedded in the domain prior module to calculate the domain prior. The domain prior and image features are added and input into the trained classifier. At the same time, the image features are input into the frozen classifier separately. The results of the trained classifier and the frozen classifier are added and output as the prediction result. This process is supervised by the task loss. The frozen text encoder and frozen classifier are respectively the global text encoder and global classifier downloaded from the server after the previous iteration step, and are locked for update in the current iteration step.

2. The personalized federated learning method based on domain-invariant text representation and domain-wide prior according to claim 1 is characterized in that: The contrast loss of the image-text alignment consists of two symmetrical log-negative probability terms, which represent the negative logarithms of the normalized similarity probabilities based on the frozen text embedding and the training text embedding, respectively. The normalized similarity probability based on the frozen text embedding is the ratio of the two parts, where the numerator of the two parts is: image feature Frozen text embedding with similar tags Negative exponential similarity between , the denominator is and the sum of the negative sample summation, is the similarity function, expressed as: , the negative sample sum represents the image feature Freeze text embedding with all heterogeneous The normalized similarity probability based on the training text embedding adopts the same structure, but with different parameters, and uses image features that do not participate in gradient updates. , text embedding uses trainable text embedding and , whose numerator is: image features that do not participate in the update Training text embeddings with similar labels Negative exponential similarity , the denominator is and the sum of the corresponding negative sample summation items, where the corresponding negative sample summation items are image features that do not participate in the update With all heterogeneous training text embeddings The sum of similarities.

3. The personalized federated learning method based on domain-invariant text representation and domain-wide prior according to claim 1 is characterized in that: The domain prior module uses the training text embedding and the data distribution of each classification in the training set statistics to calculate the domain prior, specifically: Set the global tag Each label in the Each client generates a corresponding set of text embedding vectors: ,in Indicates the Client corresponds to Text embedding representation of the class; By combining the proportion of samples of each category in the local data of the client in the number of local data categories, the text embedding representation in the text embedding vector set is weighted averaged to obtain the domain prior of the client.

4. The personalized federated learning method based on domain-invariant text representation and domain-wide prior according to claim 3 is characterized in that: The domain prior is: ； in Is the indicator function, used to judge the sample Tags Belongs to category , is the number of categories present in the local data.

5. The personalized federated learning method based on domain-invariant text representation and domain-wide prior according to claim 1 is characterized in that: The task loss is calculated by calculating the one-hot encoding vector of the true label The cross entropy between the predicted output and the model is used to measure the degree of match between the two. The predicted output of the model is obtained by the sum of two parts: the first part is the classifier result during training. , the second part is the result of freezing the classifier , and the outputs of these classifiers are normalized by the softmax function to generate the final probability distribution.

6. The personalized federated learning method based on domain-invariant text representation and domain-wide prior according to claim 1, characterized in that: The goal of local training for each client is to minimize the sum of the contrast loss and task loss of image-text alignment.

7. The personalized federated learning method based on domain-invariant text representation and domain-wide prior according to claim 1 is characterized in that: After the current iteration step is completed, all clients upload the trained local models to the server, including the text encoder, image encoder and trained classifier involved in the training. The server calculates the corresponding average models respectively to obtain the global model, including the global text encoder, global image encoder and global classifier, and transmits it back to each client. The client initializes the text encoder and frozen text encoder involved in the training as the global text encoder, the image encoder as the global image encoder, and the classifier involved in the training and the frozen classifier as the global image classifier. During the local training process, the parameters of the frozen text encoder and frozen classifier are locked and not updated, and the loop iteration is performed.

8. A personalized federated learning system based on domain-invariant text representation and domain-wide global priors, characterized by: Used to implement the method according to any one of claims 1 to 7.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions, wherein the instructions are used to implement the method according to any one of claims 1 to 7 when executed.

Images