A personalized picture retrieval method in a federated learning scenario

Abstract

The application discloses a personalized picture retrieval method in a federated learning scenario. It includes: 1) loading a public picture-text pair dataset to generate a model training dataset; 2) based on the text information in the dataset in step 1), the tokenizer word segmentation tool is used to obtain the mapping relationship between words and Token positions; 3) based on the picture information in the dataset in step 1), the picture tensor is obtained after preprocessing; 4) a unified model is constructed based on the pre-training model CLIP; 5) the unified model is trained using the text and image expressions obtained in steps 2) and 3); 6) image retrieval is performed using the unified model trained in step 5), and pictures consistent with the query text are obtained. The application designs a private retrieval database for the federated learning client, integrates client-specific information into the pictures and text, and solves the problems of retrieval accuracy decline and slow convergence caused by the non-independent and identically distributed characteristics of data in the federated learning scenario.

Description

Technical Field

[0001] This invention belongs to the field of federated learning, specifically relating to a personalized image retrieval method in a federated learning scenario. Background Technology

[0002] Federated learning (FL) has gained significant attention due to its privacy-preserving capabilities, enabling the training of machine learning models across distributed devices without sharing private data. This invention focuses on applying CLIP models to federated learning, leveraging their pre-trained knowledge for text-to-image retrieval tasks from multiple private databases. CLIP has garnered considerable attention for its ability to align text and images. However, directly applying CLIP to federated learning presents two main challenges: CLIP has a large number of parameters, and the communication and computational overhead becomes unbearable when a central server iteratively updates all optimization weights submitted by clients. Furthermore, transmitting a large number of parameters between the client and server could potentially leak users' private information.

[0003] Some methods introduce adaptive modules or a few hints during training, freezing the rest of the model. However, in federated learning scenarios, the private data of different clients are not independently and identically distributed. These methods ignore the problem of data heterogeneity, leading to slower convergence and reduced model accuracy.

[0004] In this paper, we address the aforementioned issues by enhancing CLIP-encoded representations with additional text-image information. Specifically, we establish a private database on each client containing client-specific personalized information. Then, based on a lightweight attention module, we fuse each text query with the top-k most similar image-text pairs retrieved from the private database. This fusion process injects client-specific information into the text representation, resolving the data heterogeneity problem. Furthermore, communication between the client and server is limited to the lightweight fusion module, significantly reducing computational overhead and communication costs. Summary of the Invention

[0005] To address the problems in the background technology, this invention provides a personalized image retrieval method for federated learning scenarios. The purpose of this invention is to design a private retrieval database for the federated learning client, containing the client's personalized information. Then, based on a lightweight attention-based module, the text is fused with the information retrieved from the private database to obtain a text representation containing specific client information, thereby solving the problem of data heterogeneity. Furthermore, communication between the client and server is limited to the lightweight fusion module, significantly reducing computational and communication costs.

[0006] The technical solution adopted in this invention includes the following steps:

[0007] Step 1) Generate a model training dataset based on the publicly available image-text dataset;

[0008] Step 1) specifically refers to:

[0009] 1.1) Organize a dataset of N image-text pairs:

[0010]

[0011] Where (x, y) represents an image-text pair; n i Let represent the number of image-text pairs contained in the i-th dataset, i∈{1,...,N}; j represents the j-th image-text pair;

[0012] 1.2) Assign N datasets to N federated learning clients.

[0013] Step 2) For the text information in the dataset from Step 1), obtain the positional mapping relationship between words and tokens using the tokenizer.

[0014] Step 2) specifically refers to:

[0015] For each text in the dataset in step 1), the tokenizer is used to obtain the position mapping relationship between each word in the sentence and the token, that is, each text is represented by the position of the token.

[0016] Step 3) Preprocess the image information in the dataset from Step 1) to obtain image tensors;

[0017] Step 3) specifically refers to:

[0018] For each image in the dataset in step 1), a standard transformation is performed using the Preprocess module of the CLIP model, i.e., I = Preprocess(x);

[0019] Where I represents the image after preprocessing by the preprocessing module; x represents the input image.

[0020] Step 4) Construct a unified model based on the pre-trained model CLIP; the unified model includes a CLIP model with an embedded Adapter module, a Retrieval Memory module, and a Memory Bank module;

[0021] Step 4) specifically involves:

[0022] 4.1) The CLIP model with an embedded Adapter module is as follows: an Adapter module is inserted into the last Transformer layer of the CLIP encoder;

[0023] Given the residual output h of the MLP in the Transformer, the Adapter performs the following operations:

[0024] h′←h+φ(h·W down )·W up

[0025] Among them, W down and W up φ is the weight matrix of the Adapter, and φ is a non-linear activation function;

[0026] Let f be the CLIP model with the embedded Adapter module. Freeze the other parameters of the CLIP model and learn only the Adapter module to obtain the representation of image I and text T, where υ = f. image (I), t=f text (T);

[0027] 4.2) On each federated learning client, construct the Retrieval Memory module, specifically as follows:

[0028] 4.2.1) Using the text and images processed in steps 2 and 3 as data sources, before the start of each training round, the images and text are encoded using f with all parameters frozen, resulting in the image retrieval library V. M =[f image (I1), ..., f image (I M )] and text retrieval library T M =[f text (T1), ..., f text (T M )], V M and T M The stored content that makes up the Retrieval Memory module;

[0029] Among them, I M This represents the Mth image, T. M f represents the Mth text expression; image Represents the image encoder of the CLIP model; f text A text encoder representing the CLIP model;

[0030] 4.2.2) For each text representation t output by the CLIP model in step 4.1) i , t i∈t, from T M K results were retrieved from t. i The text expression with the highest cosine similarity From V M Extracting from and expressing text Corresponding image expression

[0031] 4.2.3) For each image representation υ output by the CLIP model in step 4.1), i ,v i ∈v, from V M K results were retrieved from υ. i The image with the highest cosine similarity From T M Extracting from and expressing images Corresponding text expression

[0032] 4.3) Construct an attention-based module called Memory Bank, specifically as follows:

[0033] 4.3.1) Using the information retrieved in step 4.2.2) and For text expression t i Personalization enhancement:

[0034]

[0035]

[0036] MultiheadAttn() represents the multi-head attention mechanism.

[0037] Using the information retrieved in step 4.2.3) and To express υ in images i Personalization enhancement:

[0038]

[0039]

[0040] 4.3.2) will The data is transmitted to a multilayer perceptron (MLP) to realize the processing of text representations. i Extracting complex features:

[0041]

[0042]

[0043] Will The data is transmitted to a multilayer perceptron (MLP) to realize the representation of images. i Extracting complex features:

[0044]

[0045]

[0046] 4.3.3) Take the result obtained in step 4.3.2) With text expression t i The enhanced text expression is obtained by adding the two elements together:

[0047]

[0048] The result obtained in step 4.3.2) With images to express υ i The enhanced image representation υ′ is obtained by addition. i :

[0049]

[0050] Step 5) Use the text and image representations obtained in Steps 2) and 3) to train the unified model;

[0051] Step 5) specifically involves:

[0052] 5.1) Based on the image representation υ obtained in step 4.1) i and text expression t i Calculate the image-text contrast loss L υ2t Text-image contrast loss L t2υ :

[0053]

[0054]

[0055] Where B is the batch size, σ is the cosine similarity between the image and text representations, and s is the scaling parameter;

[0056] Adding them together gives L contra =L υ2t +L t2υ ;

[0057] 5.2) The enhanced text representation t′ obtained based on 4.3.3) and 4.3.4) i And the image expresses υ′ i Calculate the contrast loss:

[0058]

[0059]

[0060] Adding them together gives L′ contra =L′ υ2t +L′ t2υ ;

[0061] 5.3) Based on L obtained in steps 5.1) and 5.2) contra and L′ contra Calculate the final loss function L = L contra +L′ contra The parameter w = Adapter∪MemoruBank is updated using backpropagation of the error.

[0062] 5.4) The client performs local updates based on 5.3). After the update round number E is satisfied, communication between the server and the client begins: the server communicates with the N clients regarding the learnable parameters {w}. n By performing average aggregation on |1≤n≤N}, we obtain:

[0063]

[0064] Distribute the updated learnable parameters w′ to N clients;

[0065] 5.5) Repeat step 5.4) until the number of communication exchanges R is satisfied.

[0066] Step 6) Use the unified model trained in Step 5) to perform image retrieval and obtain images that match the query text.

[0067] Step 6) specifically refers to:

[0068] Given a query text q and a candidate image set S, input the model trained in step 5) to obtain the text representation t′. q And candidate image representation {υ′1,υ′2,...,υ′ |S| The cosine similarity between the text and all candidate images is calculated as a matching score, and the image with the highest score is the final search result.

[0069] The beneficial effects of this invention are:

[0070] This invention introduces a lightweight personalized federated learning method to address the critical need for personalization in federated learning environments. By integrating client-specific retrieval memories into the federated learning framework, we have demonstrated a significantly enhanced ability of the model to incorporate personalized content. Extensive experiments on four widely used datasets show the robustness and adaptability of the proposed method. Furthermore, the invention requires only a small number of parameters to learn, highlighting its practicality in real-world deployments, particularly in resource-constrained environments.

[0071] This invention addresses the issues of decreased retrieval accuracy and slow convergence caused by the non-independent and identically distributed nature of data in federated learning scenarios by designing a private retrieval database for the federated learning client and fusing client-specific information into images and text. Furthermore, this invention trains only lightweight modules, freezing the parameters of large models. This reduces client computational overhead, and communication between the client and server is limited to a small number of learnable parameters, significantly lowering communication costs. Attached Figure Description

[0072] Figure 1 This is a flowchart of the present invention;

[0073] Figure 2 This invention visualizes the query results of the text query "There is a traffic light at an intersection, and a skyscraper is in the background." using the Open-CLIP model. Detailed Implementation

[0074] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0075] Step 1: Load the publicly available image-text pair dataset to generate the model training dataset.

[0076] Step 1) specifically refers to:

[0077] 1.1) Organize a dataset of N image-text pairs:

[0078]

[0079] Where (x, y) represents an image-text pair, and n i This indicates the number of image-text pairs in the dataset. Specifically, N=4, and four datasets—MSCOCO, Flickr30K, CC3M, and IAPR TC-12—are selected as training data.

[0080] 1.2) Assign N datasets to N federated learning clients.

[0081] Step 2: Based on the text information in the dataset from Step 1), obtain the positional mapping relationship between words and tokens using the tokenizer.

[0082] Step 2) specifically refers to:

[0083] For each text in the dataset in step 1), the tokenizer is used to obtain the position mapping relationship between each word in the sentence and the token, that is, each text is represented by the position of the token.

[0084] like Figure 1As shown, the input text “A giraffe drinking water from a river on abeach.” is processed by the tokenizer to obtain the tensor [49406,320,22826,5778,1573,633,320,2473,525,320,2117,269,49407,0,0,0,0,...].

[0085] Step 3) Preprocess the image information in the dataset from Step 1) to obtain image tensors.

[0086] Step 3) specifically refers to:

[0087] Using the CLIP model's preprocessing module Preprocess, a series of standard transformations, including scaling and regularization, are performed on each image in the dataset from step 1) to make it conform to the CLIP model's input, i.e., I = Preprocess(x).

[0088] Step 4) Construct a unified model based on the pre-trained CLIP model. The unified model includes the CLIP model with an embedded Adapter module, a Retrieval Memory module, and a Memory Bank module.

[0089] Step 4) specifically involves:

[0090] 4.1) CLIP Model with Embedded Adapter Module: An Adapter module is inserted into the last Transformer layer of the CLIP encoder. Given the residual output h of the MLP in the Transformer, the Adapter performs the following operations:

[0091] h'←h+φ(h·W doun )·W up

[0092] Among them, W down and W up φ is the weight matrix of the Adapter, and φ is a non-linear activation function.

[0093] Let f be the CLIP model with an embedded Adapter module, then υ = f imgae (I) is the representation of image I, t = f text (T) is the representation of text T.

[0094] like Figure 1As shown, the two trapezoids are CLIP's text and image encoders, which encode the text and images processed in steps 2) and 3). A trainable module Adapter is embedded inside the encoder, and the remaining parameters are frozen.

[0095] 4.2) On each federated learning client, construct the Retrieval Memory module, specifically as follows:

[0096] 4.2.1) Using the user's training data as the data source, before each round of training, the images and text are encoded using the frozen f to obtain V. M =[f image (I1), ..., f image (I M )] and T M =[f text (T1), ..., f text (T M )).

[0097] 4.2.2) For a text query t i From T M Retrieved K texts with the highest cosine similarity. From V M Extracting text from Corresponding image

[0098] like Figure 1 As shown, for a text expression t, the K texts with the highest similarity to t are retrieved from the Retrieval Memory [“a giraffe bending to drink water from a river”, “a giraffe in a sandyarea driking from a lake”, “A giraffe crouching down to drink from a waterhole”, ...] and their corresponding images.

[0099] 4.3) Construct an attention-based module called Memory Bank, specifically as follows:

[0100] 4.3.1) Using the information retrieved in 4.2.2) and For text expression t i Personalization enhancement:

[0101]

[0102]

[0103] like Figure 1 As shown, the text expression t is taken as Q, and the retrieved image and text expression are taken as K and V respectively. They are then passed into the Memory Bank to inject client-specific personalized information into the text expression.

[0104] 4.3.2) Put The data is passed to a multilayer perceptron (MLP) to realize the transfer from t. i Extracting complex features:

[0105]

[0106]

[0107] 4.3.3) The enhanced text expression is obtained by adding the two:

[0108]

[0109] like Figure 1 As shown, t′ is the enhanced text representation.

[0110] 4.3.4) For image queries i Repeat steps 4.2.2-4.3.3 to obtain the enhanced image representation υ′. i .

[0111] like Figure 1 As shown, image representation v is enhanced by retrieval to obtain image representation υ′ injected with personalized information.

[0112] Step 5: Train the unified model using the text and image representations obtained in Steps 2) and 3);

[0113] Step 5) specifically involves:

[0114] 5.1) Based on the image and text representations obtained in step 4.1), calculate the image-text contrast loss and the text-image contrast loss:

[0115]

[0116]

[0117] Where B is the batch size, σ calculates the cosine similarity between the image and text representations, and s is the scaling parameter. Adding them together yields L. contra =L υ2t +L t2υ .

[0118] 5.2) The enhanced text representation t′ obtained based on steps 4.3.3) and 4.3.4) iAnd the image expresses υ′ i Calculate the contrast loss:

[0119]

[0120]

[0121] Adding them together gives L′ contra =L′ υ2t +L′ t2υ .

[0122] 5.3) Based on L obtained in steps 5.1) and 5.2) contra and L′ contra The final loss function L = L is obtained. contra +L′ contra The parameter w = Adapter ∪ MemoryBank is updated using backpropagation of error.

[0123] like Figure 1 As shown, the parameters of the Adapter module and the Memory Bank module are updated according to the loss function, while the remaining parameters are frozen.

[0124] 5.4) After the client locally satisfies the update round number E, communication between the server and the client begins: the server exchanges information about the learnable parameters {w} of N clients. n The average aggregation of |1≤n≤N} yields... Distribute to N clients.

[0125] like Figure 1 As shown, the client uploads the parameters to the server. The server averages the parameters from all clients to obtain the updated parameters, which are then distributed to each client for the next round of local updates.

[0126] 5.5) Repeat step 5.4) until the number of communication exchanges R is satisfied.

[0127] Step 6) Use the unified model trained in Step 5) to perform image retrieval and obtain images that match the query text.

[0128] Step 6) specifically refers to:

[0129] Given a query text q and a candidate image set S, input the model trained in step 5) to obtain the text representation t. q And candidate image representation {υ1, υ2, ..., υ |S| The cosine similarity between the text and the image is calculated as a matching score, and the image with the highest score is the final search result. Specific implementation examples:

[0131] The query is performed using the text "There is a traffic light at an intersection, with a skyscraper in the background." Figure 2 The results are shown in the table below, comparing the results of the method of this invention with those of Open-CLIP. It can be observed that our method performs accurately when matching the factors "traffic lights" and "skyscrapers in the background," especially in the first query result. In contrast, Open-CLIP lacks accuracy when matching "skyscrapers," incorrectly identifying pillars as traffic lights in the fourth query result.

[0132] Comparison table of the performance of the proposed model and conventional models on a general image and text dataset.

[0133] MSCOCO Flickr 30K CC3M IAPR TC-12 Open-CLIP 0.3423 0.5968 0.5068 0.5192 FedAVG 0.3530 0.6162 0.4734 0.5952 FedProx 0.3648 0.6304 0.4944 0.6076 FedCLIP 0.3637 0.6274 0.5186 0.5604 ours 0.3744 0.6446 0.5472 0.6140

[0134] Comparison results:

[0135] (1) The evaluation index is recall: Recall@1∈[0,1], the larger the value, the better the effect.

[0136] (2) Open-CLIP is a method that directly loads pre-trained models for inference. FedAVG, FedProx, and FedCLIP are commonly used federated learning models. FedAVG learns all parameters of the model, FedProx introduces a regularization term on top of FedAVG to reduce imbalance, and FedCLIP adds an Adapter module at the end of the CLIP encoder to adapt to downstream tasks. Ours is the model of this invention.

[0137] (3) Compared with FedProx, the present invention improved the metrics on the four datasets by 0.96%, 1.42%, 5.28% and 0.64%, respectively;

[0138] (4) Compared with FedCLIP, the present invention improved the metrics on the four datasets by 1.07%, 1.72%, 2.86% and 5.36%, respectively.

Claims

1. A personalized image retrieval method in a federated learning scenario, characterized in that, Includes the following steps: 1) Generate a model training dataset based on publicly available image-text datasets; 2) For the text information in the dataset in step 1), the positional mapping relationship between words and tokens is obtained through the tokenizer. 3) Preprocess the image information in the dataset from step 1) to obtain image tensors; 4) Construct a unified model based on the pre-trained model CLIP; the unified model includes a CLIP model with an embedded Adapter module, a Retrieval Memory module, and a Memory Bank module; Step 4) specifically involves: 4.1) The CLIP model with an embedded Adapter module is as follows: an Adapter module is inserted into the last Transformer layer of the CLIP encoder; Given the residual output of the MLP in the Transformer The Adapter performs the following operations: in, and This is the weight matrix of the Adapter. It is a non-linear activation function; Let the CLIP model with an embedded Adapter module be denoted as The remaining parameters of the CLIP model are frozen, and only the Adapter module is learned to obtain the representation of image I and text T. , ; 4.2) On each federated learning client, construct the Retrieval Memory module, specifically as follows: 4.2.1) Using the text and images processed in steps 2 and 3 as data sources, before the start of each training round, freeze all parameters. Encode images and text to obtain an image retrieval library. and text search library , and The stored content that makes up the Retrieval Memory module; in, This represents the expression of the Mth image. This represents the Mth text expression; Represents an image encoder for the CLIP model; A text encoder representing the CLIP model; 4.2.2) For each text representation output by the CLIP model in step 4.1) , ,from K results were retrieved from the database. The text expression with the highest cosine similarity ,from Extracting from and expressing text Corresponding image expression ; 4.2.3) For each image output by the CLIP model in step 4.1), the representation... , ,from K results were retrieved from the database. The image with the highest cosine similarity ,from Extracting from and expressing images Corresponding text expression ; 4.3) Construct an attention-based module called Memory Bank, specifically as follows: 4.3.1) Using the information retrieved in step 4.2.2) and Textual expression Personalization enhancement: in, Indicating multi-head attention mechanism Using the information retrieved in step 4.2.3) and Expressing through images Personalization enhancement: 4.3.2) will , Transmitted to a multilayer perceptron (MLP) to realize the transformation from text representation Extracting complex features: Will , The data is transmitted to a multilayer perceptron (MLP) to represent images. Extracting complex features: 4.3.3) Take the result obtained in step 4.3.2) , With text expression The enhanced text expression is obtained by adding the two elements together: The result obtained in step 4.3.2) , With pictures The enhanced image expression is obtained by addition. : ； 5) Train the unified model using the text and image representations obtained in steps 2) and 3); 6) Use the unified model trained in step 5) to perform image retrieval and obtain images that match the query text.

2. The personalized image retrieval method in a federated learning scenario according to claim 1, characterized in that, Step 1) specifically refers to: 1.1) Organizing Image-text pair dataset: ； in, Represents an image-text pair; Indicates the first The number of image-text pairs contained in each dataset. ; Indicates the first Image-text pairs; 1.2) Put Each dataset is assigned to A federated learning client.

3. The personalized image retrieval method in a federated learning scenario according to claim 1, characterized in that, Step 2) specifically refers to: For each text in the dataset in step 1), the tokenizer is used to obtain the position mapping relationship between each word in the sentence and the token, that is, each text is represented by the position of the token.

4. The personalized image retrieval method in a federated learning scenario according to claim 1, characterized in that, Step 3) specifically refers to: For each image in the dataset from step 1), a standard transformation is performed using the Preprocess module of the CLIP model, i.e., the following steps are executed: ; in, This indicates the image after it has been processed by the preprocessing module; This represents the input image.

5. The personalized image retrieval method in a federated learning scenario according to claim 1, characterized in that, Step 5) specifically involves: 5.1) Image representation based on step 4.1) and text expression Calculate image-text contrast loss Text-image contrast loss : in, For batch size, Cosine similarity between images and text representations. For scaling parameters; Adding them together gives ; 5.2) Enhanced text representation based on 4.3.3) and 4.3.4) and images Calculate the contrast loss: Adding them together gives ; 5.3) Based on steps 5.1) and 5.2) and Calculate the final loss function. Using backpropagation of error to evaluate parameters Update; 5.4) The client performs local updates based on 5.3). After the update round number E is satisfied, communication between the server and the client begins: the server communicates with the N clients regarding the learnable parameters. After performing average polymerization, we obtain: Updated learnable parameters Distribute to N clients; 5.5) Repeat step 5.4 until the number of communication exchanges R is satisfied.

6. The personalized image retrieval method in a federated learning scenario according to claim 1, characterized in that, Step 6) specifically involves: Given a query text q and a candidate image set S, input the model trained in step 5) to obtain the text representation. and candidate image expression The cosine similarity between the text and all candidate images is calculated as a matching score, and the image with the highest score is the final search result.

Images