A cache model-based comparative visual language model federated fine-tuning method

Abstract

The application discloses a contrast visual language model federated fine-tuning method based on a cache model, which comprises the following steps: 1, generating a class balanced data set by using a generative pre-training model on a server; 2, extracting visual features of a synthetic image and encoding a label on the server to construct an initial cache model; 3, issuing the cache model on the server, extracting image and text features on a client, and calculating the similarity of the two to obtain an initial prediction result; 4, querying the cache features based on the cosine similarity, retrieving the label, and generating a cache prediction result; 5, obtaining a final output by weighting and fusing the CLIP prediction and the cache prediction through hyperparameters; 6, optimizing the cache model by using a cross-entropy loss and a stochastic gradient descent on a local side; and 7, aggregating cache model parameters of all clients on the server to complete a round of federated training and evaluation.

Description

Technical Field

[0001] This invention relates to the field of fine-tuning contrastive visual language models using federated learning methods, and more particularly to a federated fine-tuning method for contrastive visual language models based on a cache model. Background Technology

[0002] With the rapid development of artificial intelligence, visual language models have become a core technology in the field of multimodal learning, capable of simultaneously modeling image and text signals to achieve cross-modal understanding. Among them, contrastive visual language models, represented by CLIP, rely on large-scale image and text data and contrastive learning strategies to achieve significant results in tasks such as image classification, and possess good versatility and transferability. Their performance can be further improved through fine-tuning on domain-specific data. However, the data required for fine-tuning is usually scattered across different institutions or terminal devices, and cannot be centrally used for unified training due to privacy, security, and storage limitations. To address this, federated learning, as a distributed, privacy-friendly collaborative training framework, enables each client to train independently on local data and share only model updates, thereby achieving joint optimization of large models while protecting data from leaving the local machine.

[0003] Existing methods for fine-tuning visual language models mainly include full fine-tuning and parameter-efficient fine-tuning (PEFT). Full fine-tuning requires updating all model parameters, which is resource-intensive and prone to overfitting when data is limited. In contrast, parameter-efficient fine-tuning only adjusts a small number of newly added or low-rank parameters, significantly reducing computational and storage costs. Zhou et al. (Reference: Zhou K, Yang J, Loy C C, et al. Learning to prompt for vision-language models[J]. International Journal of Computer Vision, 2022, 130(9): 2337-2348.) proposed an automated prompt learning method, CoOp, for pre-trained visual language models. CoOp models the context words in the prompt as learnable vectors and optimizes only these context vectors to adapt to specific tasks while keeping the original model parameters frozen. CoOp can take the form of shared context or category-specific context to adapt to general or fine-grained classification tasks.

[0004] Federated fine-tuning achieves downstream task learning while protecting data privacy by enabling local adaptation on each client based on a large-scale pre-trained model. PromptFL (reference: Guo T, Guo S, Wang J, et al. PromptFL: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model[J]. IEEE Transactions on Mobile Computing, 2023, 23(5): 5179-5194.) proposes a lightweight fine-tuning method that introduces prompt learning into federated learning. By allowing each client to optimize the prompt vectors of the pre-trained visual language model on local data without updating large-scale model parameters, it significantly reduces computational and communication overhead. While maintaining privacy constraints, this method enables each client to efficiently adapt to downstream tasks using a small amount of data and share prompt updates through a federated aggregation mechanism, achieving collaborative lightweight fine-tuning of a large model.

[0005] However, applying federated learning directly to large-scale visual language models such as CLIP still faces significant obstacles. On one hand, the sheer number of parameters in these models necessitates substantial computational resources from clients during local training and incurs high communication overhead during model update uploads, making efficient federated training difficult in resource-constrained real-world applications. On the other hand, each client typically possesses only a limited amount of data with a non-independent and identically distributed (non-IID) data set, which reduces the quality of local updates, slows the convergence of the global model, and limits overall performance improvement. Summary of the Invention

[0006] Purpose of the invention: To address the aforementioned challenges, this invention proposes a federated fine-tuning method for contrastive visual language models based on a caching model, thereby efficiently optimizing the adaptation process of visual language models such as CLIP in federated learning scenarios. The method includes the following steps:

[0007] Step 1: On the server side, use the generative pre-trained model DALL·E to input the dataset category labels in a formatted manner into the generative model, generate an image for each label, and generate a synthetic dataset with a balanced class distribution.

[0008] Step 2: On the server side, use CLIP's image encoder to extract visual features W1 from the synthetic dataset, and convert the labels of the synthetic images into one-hot vectors W2. The visual feature weights and one-hot vectors together constitute the caching model.

[0009] Step 3: The server distributes the initialized cache model to all federated clients. On each client, the images from the local dataset are input into the CLIP image encoder to generate image feature vectors f. vision Descriptive text of N categories is input into CLIP's text encoder to generate text feature vectors f. text Composition matrix W text Calculate the similarity logits1 between the image feature vector and the text-like feature vector;

[0010] Step 4: Query visual features f based on cosine similarity. vision The distance between the visual feature W1 and the cached feature is calculated, and then the corresponding label value stored in W2 is retrieved to obtain the result logits2 based on the cache model.

[0011] Step 5: Mix logits1 based on the pre-trained CLIP with logits2 retrieved from the cached model using the hyperparameter α to obtain the final predicted logits.

[0012] Step 6: Optimize the cache model using cross-entropy loss and stochastic gradient descent. During each round of optimization, the one-hot vector W2 remains frozen, and only the visual feature weights are updated on the client side.

[0013] Step 7: After training all local clients is completed, the updated cached model parameters are sent back to the server for weighted aggregation to complete one round of federated training. The server then conducts round-by-round evaluation and redistributes the cached model to each client at the start of the next round.

[0014] In step 1 of this invention, OpenAI's DALL·E is used for data generation. The labels of N classes are formatted as "a photo of a [CLASS]" and input into DALL·E. K images are generated for each label, as follows:

[0015] I N，K =DALL·E(T) N ),

[0016] Where T N Text input representing N category labels, I N，K This represents NK images generated for these labels. This method can automatically generate a synthetic dataset with a balanced class distribution. Specifically, generating only a small number of images for each class can optimize server-side performance load.

[0017] Step 2 of the present invention includes:

[0018] Step 2-1: The pre-trained CLIP image encoder uses ResNet-50 as the backbone network. The CLIP image encoder extracts visual features from a synthetic dataset containing NK images. The dimension of the visual features is C, represented as follows:

[0019] W1 = ImageEncoder(I N，K ),

[0020] Where W1 is a matrix with C rows and NK columns, representing the cached visual feature weights;

[0021] Step 2-2, the corresponding label L of the NK image N，K It is converted into a one-hot encoded vector, represented as:

[0022] W2 = OneHot(L N，K ),

[0023] Here, W2 is a matrix with N rows and NK columns, representing the cached one-hot vector weights. The visual feature weights W1 and the one-hot vector weights W2 together constitute the caching model, which is used to store the visual features pre-calculated from the synthetic image and their corresponding labels.

[0024] Step 3 of the present invention includes:

[0025] Step 3-1: In order to protect data privacy while using local data, a federated learning paradigm is applied to optimize the update process of the cache model. After initializing the cache model on the synthetic dataset, the server distributes the visual feature weights and one-hot vector weights in the cache model to each client, and each client loads the cache model locally.

[0026] Step 3-2, on the client, transfer the image D from the local dataset. k The image encoder input to CLIP generates a C-dimensional image feature vector f. vision , is represented as:

[0027] f vision =ImageEncoder(D k );

[0028] Step 3-3: On the client side, input the descriptive text "a photo of a [CLASS]" for N categories into CLIP's text encoder to generate N C-dimensional text feature vectors f. text This forms an N×C matrix W. text , is represented as:

[0029] W text =TextEncoder(Input)N );

[0030] Steps 3-4: Next, calculate the cosine similarity between the image feature vector and the text-like feature vector, which is to multiply their matrices, as follows:

[0031]

[0032] In step 4 of this invention, the logits of each image are calculated using a caching model, expressed as:

[0033]

[0034] This step can be viewed as first measuring image features f. vision The distance between the cached image features W1 and W2 is calculated, and then the corresponding label value stored in W2 is retrieved. Essentially, the caching model performs a nearest-neighbor feature retrieval operation, converting pre-cached visual information into classification logits. The hyperparameter β adjusts the sharpness of the distance-to-logits transformation. A larger β amplifies the differences between feature distances, generating weights with more peak characteristics, thus forcing the model to rely primarily on the nearest-neighbor cached features; conversely, a smaller β produces smoother, more evenly distributed logits, allowing the model to aggregate information from a wider range of cached features.

[0035] In step 5 of this invention, based on the value retrieved from the caching model, logits1 from the pre-trained CLIP is fused with logits2 obtained from the caching model to obtain the final predicted logits, expressed as:

[0036] logits=logits1+α·logits2,

[0037] Here, α is a hyperparameter that can be adjusted according to the task requirements. When the task needs to absorb more specific knowledge, a larger α is more appropriate; while when the model needs more generalization ability, it is recommended to use a smaller α.

[0038] In step 6 of this invention, the caching model is optimized on the local client using cross-entropy loss and stochastic gradient descent based on the local dataset. During the t-th round of local optimization, the one-hot vector weights W2 of the caching model remain frozen, and only the visual feature weights W1 are updated on client k, as shown below:

[0039]

[0040] Step 7 of the present invention includes:

[0041] Step 7-1: For each client k that has completed t rounds of local training, the visual feature parameters of the completed cached model will be updated. The data is sent back to the server for aggregation, similar to FedAvg (reference: McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C] / / Artificial intelligence and statistics.PMLR, 2017: 1273-1282.), and weighted aggregation is performed based on the number of samples from each client, as shown below:

[0042]

[0043] in S represents the visual features of the cache model in iteration t+1. t Let n represent the clients participating in the update in round t. k This represents the number of samples on the k-th client, and n represents the total number of samples across all clients. This represents the local cache model visual feature weights on the k-th client at iteration t+1;

[0044] Step 7-2: After aggregation, on the server side, based on the aggregated cached model visual features... The training performance was evaluated using the test dataset; the aggregated cached model visual features were then re-aggregated. The data is distributed to each client participating in the update in this round. The client then reloads the visual feature parameters into its local cache model, completing the entire federated training process.

[0045] Beneficial effects:

[0046] 1. This invention proposes a federated fine-tuning method for contrastive visual language models based on a cached model. This cached model consists only of visual features and one-hot encoded labels, resulting in a lightweight structure and a significantly smaller number of parameters compared to a complete pre-trained model. By replacing traditional whole-model fine-tuning with cached module fine-tuning, this invention can significantly reduce communication and training overhead on resource-constrained clients.

[0047] 2. This invention introduces a novel strategy for initializing and fine-tuning cache modules. By utilizing a generative pre-trained model to synthesize a class-balanced dataset, cache module initialization can be completed without relying on any additional public or private data. During federated fine-tuning, the original model remains frozen, and only a subset of the cache module's parameters are trained. This preserves the model's prior knowledge while achieving efficient adaptation and rapid convergence under limited local data conditions. Attached Figure Description

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0049] Figure 1 This is a schematic diagram of the process of the present invention.

[0050] Figure 2 This is a schematic diagram of the process of the present invention.

[0051] Figure 3 This is a flowchart illustrating the class-balanced synthetic dataset generation algorithm proposed in this invention.

[0052] Figure 4 This is a flowchart illustrating the federated training algorithm for the cache model proposed in this invention.

[0053] Figure 5 This diagram illustrates the comparative experimental results of the proposed method on different client data distributions on datasets such as ImageNet. Figure 6 This is a schematic diagram showing the experimental results comparing the communication load and training memory load of the method proposed in this invention. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] like Figure 1 As shown, a federated fine-tuning method for a contrastive visual language model based on a caching model includes seven steps:

[0056] In step 1 of this invention, as follows Figure 2 As shown, OpenAI's DALL·E is used for data generation. The labels for N classes are formatted as "a photo of a [CLASS]" and input into DALL·E. K images are generated for each label, as follows:

[0057] I N，K =DALL·E(T) N ),

[0058] Where T NText input representing N category labels, I N，K This represents NK images generated for these labels. This method can automatically generate a synthetic dataset with a balanced class distribution. Specifically, generating only a small number of images for each class can optimize server-side performance load.

[0059] Step 2 of the present invention includes:

[0060] Step 2-1, as follows Figure 3 As shown, the pre-trained CLIP image encoder uses ResNet-50 as the backbone network. The CLIP image encoder extracts visual features from a synthetic dataset containing NK images. The dimension of the visual features is C, represented as follows:

[0061] W1 = ImageEncoder(I N，K ),

[0062] Where W1 is a matrix with C rows and NK columns, representing the cached visual feature weights;

[0063] Step 2-2, the corresponding label L of the NK image N,K It is converted into a one-hot encoded vector, represented as:

[0064] W2 = OneHot(L N，K ),

[0065] Here, W2 is a matrix with N rows and NK columns, representing the cached one-hot vector weights. The visual feature weights W1 and the one-hot vector weights W2 together constitute the caching model, which is used to store the visual features pre-calculated from the synthetic image and their corresponding labels.

[0066] Step 3 of the present invention includes:

[0067] Step 3-1, as follows Figure 3 As shown, in order to protect data privacy while using local data, a federated learning paradigm is applied to optimize the update process of the cache model. After initializing the cache model on the synthetic dataset, the server distributes the visual feature weights and one-hot vector weights in the cache model to each client, and each client loads the cache model locally.

[0068] Step 3-2, on the client, transfer the image D from the local dataset. k The image encoder input to CLIP generates a C-dimensional image feature vector f. vision , is represented as:

[0069] f vision =ImageEncoder(D k );

[0070] Step 3-3: On the client side, input the descriptive text "a photo of a [CLASS]" for N categories into CLIP's text encoder to generate N C-dimensional text feature vectors f. text This forms an N×C matrix W. text , is represented as:

[0071] W text =TextEncoder(Input) N );

[0072] Steps 3-4: Next, calculate the cosine similarity between the image feature vector and the text-like feature vector, which is to multiply their matrices, as follows:

[0073]

[0074] In step 4 of this invention, the logits of each image are calculated using a caching model, expressed as:

[0075]

[0076] This step can be viewed as first measuring image features f. vision The distance between the cached image features W1 and W2 is calculated, and then the corresponding label value stored in W2 is retrieved. Essentially, the caching model performs a nearest-neighbor feature retrieval operation, converting pre-cached visual information into classification logits. The hyperparameter β adjusts the sharpness of the distance-to-logits transformation. A larger β amplifies the differences between feature distances, generating weights with more peak characteristics, thus forcing the model to rely primarily on the nearest-neighbor cached features; conversely, a smaller β produces smoother, more evenly distributed logits, allowing the model to aggregate information from a wider range of cached features.

[0077] In step 5 of this invention, based on the value retrieved from the caching model, logits1 from the pre-trained CLIP is fused with logits2 obtained from the caching model to obtain the final predicted logits, expressed as:

[0078] logits=logits1+α·logits2,

[0079] Here, α is a hyperparameter that can be adjusted according to the task requirements. When the task needs to absorb more specific knowledge, a larger α is more appropriate; while when the model needs more generalization ability, it is recommended to use a smaller α.

[0080] In step 6 of this invention, as follows Figure 3As shown, the caching model is optimized on the local client using cross-entropy loss and stochastic gradient descent based on the local dataset. During the t-th round of local optimization, the one-hot vector weights W2 of the caching model remain frozen, and only the visual feature weights W1 are updated on the client k, as shown below:

[0081]

[0082] Step 7 of the present invention includes:

[0083] Step 7-1: For each client k that has completed t rounds of local training, the visual feature parameters of the completed cached model will be updated. The data is sent back to the server for aggregation, similar to FedAvg (reference: McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C] / / Artificial intelligence and statistics.PMLR,2017:1273-1282.), and weighted aggregation is performed based on the number of samples from each client, as shown below:

[0084]

[0085] in S represents the visual features of the cache model in iteration t+1. t Let n represent the clients participating in the update in round t. k This represents the number of samples on the k-th client, and n represents the total number of samples across all clients. This represents the visual feature weights of the local cache model on the k-th client at iteration t+1.

[0086] Step 7-2: After aggregation, on the server side, based on the aggregated cached model visual features... The training performance was evaluated using the test dataset; the aggregated cached model visual features were then re-aggregated. The data is distributed to each client participating in the update in this round. The client then reloads the visual feature parameters into its local cache model, completing the entire federated training process.

[0087] Example

[0088] To verify the effectiveness of this invention, we conducted experimental evaluations on four widely used image classification datasets: ImageNet, Stanford-Cars, SUN397, and UCF101. In a few-shot setting, each class in all of these datasets contained only 16 training samples.

[0089] To evaluate the robustness of the proposed method under data heterogeneity, we employed three different data partitioning strategies: **Independent and Identically Distributed (iid):** Data is evenly distributed across clients, with each client having the same set of categories and a relatively balanced distribution of sample sizes. For example, when 16 images per category are assigned to 10 clients, the first 4 clients receive 1 image per category, and the last 6 clients receive 2 images per category. **Dirichlet Non-Independent and Identically Distributed (dir):** Data is randomly partitioned using a Dirichlet distribution with parameter Dir (0.1) to simulate a moderately non-independent and identically distributed scenario. **Extremely Non-Independent and Identically Distributed (pat):** Each client has independent and non-overlapping sets of categories. For example, when 100 categories are assigned to 10 clients, each client contains only 10 non-overlapping categories.

[0090] To verify the effectiveness of the proposed algorithm for fine-tuning, we used the original CLIP model and several representative parameter efficient fine-tuning (PEFT) methods as baselines. These baselines included CLIP (reference: Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural language supervision [C] / / International conference on machine learning. PmLR, 2021: 8748-8763.), FedCLIP (reference: Lu W, Hu X, Wang J, et al. Fedclip: Fast generalization and personalization for clip in federated learning [J]. arXiv preprint arXiv: 2302.13485, 2023.), and FedLoRA (reference: Zanella M, Ben Ayed I. Low-rank few-shot adaptation of vision-language models [C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern). Recognition.2024: 1593-1603.), PromptFL (Reference: Guo T, Guo S, Wang J, et al. Promptfl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model [J]. IEEE Transactions on Mobile Computing, 2023, 23(5): 5179-5194.), FedOTP (Reference: Li H, Huang W, Wang J, et al.Global and local promptscooperation via optimal transport for federated learning [C] / / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 12151-12161.), and a federated fine-tuning method for contrastive visual language models based on a cache model, denoted as CacheFL, is proposed in this invention. For all CLIP-based methods, ResNet-50 is used as the image encoder, and Transformer is used as the text encoder.

[0091] All experiments were conducted using the PyTorch framework on a Tesla V100 SXM2 32GB GPU. Model training employed a stochastic gradient descent (SGD) optimizer with a learning rate of 0.001 and a cross-entropy loss function. All federated fine-tuning methods involved 20 rounds of global communication, with each round consisting of only one local training epoch.

[0092] Based on experimental evaluation (e.g.) Figure 4 As shown, this invention demonstrates significant advantages over various representative baseline methods across multiple datasets and different data distribution settings. Whether in independent and identically distributed (IOD) or non-independent IOD scenarios, the proposed method consistently achieves optimal or near-optimal performance on most evaluation datasets. In contrast, FedCLIP and FedLoRA even exhibit performance degradation under low data volume conditions (only 16 training samples per class). This is because the parameter-efficient fine-tuning module is difficult to train sufficiently, and modifications to the CLIP backbone parameters may disrupt the model's inherent generalization ability, further limiting the adaptation effect. The performance degradation of PromptFL is particularly pronounced when client data is highly heterogeneous. Although introducing the FedProx regularization term can alleviate this problem to some extent, its stability and overall performance are still inferior to the method of this invention. Furthermore, while FedOTP achieves competitive results on some datasets, the excessive personalization tendency brought about by the local prompting mechanism may weaken the model's global generalization ability, ultimately leading to a decline in overall test performance.

[0093] The experimental results above demonstrate that our algorithm possesses two major advantages. First, to mitigate the adverse effects of heterogeneous client data, a class-balanced synthetic dataset is introduced during the initialization phase, effectively compensating for the lack of class information and providing more consistent and balanced prior knowledge to each client. Second, unlike modular methods such as Adapter and LoRA that modify the internal parameters of the backbone network, our caching module completely maintains the pre-trained CLIP model parameters unchanged during training. This design enhances the adaptability to downstream tasks while maximizing the preservation of the generalization characteristics of the original CLIP model, enabling us to exhibit robust and superior performance across diverse datasets and data distribution settings.

[0094] like Figure 5 As shown, the significant differences in communication overhead among the various evaluation methods are illustrated. Full model fine-tuning requires the transmission of over 102 million parameters in each round of communication, resulting in unacceptable communication costs in real-world federated learning scenarios. While parameter-efficient fine-tuning methods like FedCLIP alleviate this problem to some extent (approximately 2.1 million parameters), their communication overhead remains considerable. In contrast, CacheFL only requires the transmission of approximately 102,000 parameters in each round of communication, achieving approximately 1000 times the communication overhead compared to full model fine-tuning and approximately 20 times less than FedCLIP, while remaining within a low-cost range comparable to cue-based fine-tuning methods.

[0095] At the same time, such as Figure 5 As shown, the training overhead of each method was compared from the perspective of GPU memory usage. Full model fine-tuning consumed the most memory, reaching 12.86GB, further highlighting its high computational resource requirements. Parameter-efficient fine-tuning methods reduced memory usage to varying degrees: FedCLIP reduced memory requirements to 8.79GB, FedLoRA to 7.24GB, while the cue-based fine-tuning method achieved the lowest memory usage at only 6.01GB. CacheFL consumed 7.10GB of memory, also within the range of lightweight methods and comparable to other efficient methods; more importantly, it still achieved significantly better performance than existing methods even in low-data-volume scenarios.

[0096] This invention provides a federated fine-tuning method for a contrastive visual language model based on a caching model. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for federated fine-tuning of a contrastive visual language model based on a caching model, characterized in that, Includes the following steps: Step 1: On the server side, use the generative pre-trained model DALL·E to input the dataset category labels in a formatted manner into the generative model, generate an image for each label, and generate a synthetic dataset with a balanced class distribution. Step 2: On the server side, use CLIP's image encoder to extract visual features W1 from the synthetic dataset, and convert the labels of the synthetic images into one-hot vectors W2. The visual feature weights and one-hot vectors together constitute the caching model. Step 3: The server distributes the initialized cache model to all federated clients. On each client, the images from the local dataset are input into the CLIP image encoder to generate image feature vectors f. vision Descriptive text of N categories is input into CLIP's text encoder to generate text feature vectors f. text Composition matrix W text Calculate the similarity logits1 between the image feature vector and the text-like feature vector; Step 4: Query visual features f based on cosine similarity. vision The distance between the visual feature W1 and the cached feature is calculated, and then the corresponding label value stored in W2 is retrieved to obtain the result logits2 based on the cache model. Step 5: Mix logits1 based on the pre-trained CLIP with logits2 retrieved from the cached model using the hyperparameter α to obtain the final predicted logits. Step 6: Optimize the cache model using cross-entropy loss and stochastic gradient descent. During each round of optimization, the one-hot vector W2 remains frozen, and only the visual feature weights are updated on the client side. Step 7: After training all local clients is completed, the updated cached model parameters are sent back to the server for weighted aggregation to complete one round of federated training. The server then conducts round-by-round evaluation and redistributes the cached model to each client at the start of the next round.

2. The method for federated fine-tuning of a contrastive visual language model based on a caching model as described in claim 1, characterized in that, In step 1, OpenAI's DALL·E is used for data generation. The labels of N classes are formatted as "aphotoofa[CLASS]" and input into DALL·E to generate K images for each label, as follows: THE N，K =DALL·E(T N ), Where T N Text input representing N category labels, I N，K This represents NK images generated for these labels. This method can automatically generate a synthetic dataset with a balanced class distribution. Specifically, it only needs to generate a small number of images for each class, which can optimize server performance load.

3. The method for federated fine-tuning of a contrastive visual language model based on a caching model as described in claim 2, characterized in that, Step 2 includes: Step 2-1: The pre-trained CLIP image encoder uses ResNet-50 as the backbone network. The CLIP image encoder extracts visual features from a synthetic dataset containing NK images. The dimension of the visual features is C, represented as follows: W1=ImageEncoder(I N，K ), Where W1 is a matrix with C rows and NK columns, representing the cached visual feature weights; Step 2-2, the corresponding label L of the NK image N,K It is converted into a one-hot encoded vector, represented as: W2=OneHot(L N,K ), W2 is a matrix with N rows and NK columns, representing the cached one-hot vector weights. The visual feature weights W1 and the one-hot vector weights W2 together constitute the cache model, which is used to store the visual features and their corresponding labels pre-calculated from the synthetic image.

4. The method for federated fine-tuning of a contrastive visual language model based on a caching model as described in claim 3, characterized in that, Step 3 includes: Step 3-1: In order to protect data privacy while using local data, a federated learning paradigm is applied to optimize the update process of the cache model. After initializing the cache model on the synthetic dataset, the server distributes the visual feature weights and one-hot vector weights in the cache model to each client, and each client loads the cache model locally. Step 3-2, on the client, transfer the image D from the local dataset. k The image encoder input to CLIP generates a C-dimensional image feature vector f. vision , is represented as: f vision =ImageEncoder(D k ); Step 3-3: On the client side, input the descriptive text "a photo of a [CLASS]" of N categories into CLIP's text encoder to generate N C-dimensional text feature vectors f. text This forms an N×C matrix W. text , is represented as: W text =TextEncoder(Input N ); Steps 3-4: Next, calculate the cosine similarity between the image feature vector and the text-like feature vector, which is to multiply their matrices, as follows:

5. The method for federated fine-tuning of a contrastive visual language model based on a caching model as described in claim 4, characterized in that, In step 4, the logits of each image are calculated using the caching model, represented as: This step can be viewed as first measuring image features f. vision The distance between the cached image features W1 and W2 is calculated, and then the corresponding label value stored in W2 is retrieved. Essentially, the caching model performs a nearest-neighbor feature retrieval operation, converting pre-cached visual information into classification logits. The hyperparameter β is used to adjust the sharpness of the transformation from distance to logits. A larger β amplifies the differences between feature distances, generating weights with more peak characteristics, thus forcing the model to rely mainly on the nearest-neighbor cached features; conversely, a smaller β produces smoother, more evenly distributed logits, allowing the model to aggregate information from a wider range of cached features.

6. The method for federated fine-tuning of a contrastive visual language model based on a caching model as described in claim 5, characterized in that, In step 5, based on the values ​​retrieved from the caching model, logits1 from the pre-trained CLIP is fused with logits2 obtained from the caching model to obtain the final predicted logits, expressed as: logits=logits1+α·logits2, Here, α is a hyperparameter that can be adjusted according to the task requirements. When the task needs to absorb more specific knowledge, a larger α is more appropriate; while when the model needs more generalization ability, it is recommended to use a smaller α.

7. The method for federated fine-tuning of a contrastive visual language model based on a caching model as described in claim 6, characterized in that, In step 6, the cached model is optimized on the local client using cross-entropy loss and stochastic gradient descent based on the local dataset. During the t-th round of local optimization, the one-hot vector weights W2 of the cached model remain frozen, and only the visual feature weights W1 are updated on the client k, as shown below:

8. The method for federated fine-tuning of a contrastive visual language model based on a caching model as described in claim 7, characterized in that, Step 7 includes: Step 7-1: For each client k that has completed t rounds of local training, the visual feature parameters of the completed cached model will be updated. The data is sent back to the server for aggregation, similar to FedAvg (reference: McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C] / / Artificial intelligence and statistics.PMLR, 2017: 1273-1282.), and weighted aggregation is performed based on the number of samples from each client, as shown below: in S represents the visual features of the cache model in iteration t+1. t Let n represent the clients participating in the update in round t. k This represents the number of samples on the k-th client, and n represents the total number of samples across all clients. This represents the local cache model visual feature weights on the k-th client at iteration t+1; Step 7-2: After aggregation, on the server side, based on the aggregated cached model visual features... The training performance was evaluated using the test dataset; the aggregated cached model visual features were then re-aggregated. The data is distributed to each client participating in the update in this round. The client then reloads the visual feature parameters into its local cache model, completing the entire federated training process.

Images