A semantically bridged federated image classification method

Abstract

This invention belongs to the field of image classification technology, specifically relating to a federated learning image classification method based on semantic bridging. This paper introduces a server-side semantic adapter and proposes a semantic bridging federated learning framework based on a pre-trained visual language model. The client utilizes a pre-trained visual language model (such as CLIP) as a stable semantic anchor, enabling local training to align visual features with the semantic knowledge of the global model. The server aggregates global semantic knowledge through the semantic adapter and optimizes the global classification head. At the local client level, this invention employs dual semantic distillation, forcing local features to align with the global semantic anchor at both the backbone and projection levels, while simultaneously utilizing memory replay technology to repair the decision boundaries of missing categories. Experiments show that this method significantly improves performance compared to state-of-the-art federated learning methods in non-independent and identically distributed environments.

Description

Technical Field

[0001] This invention belongs to the field of federated learning and image classification technology, and relates to a semantic bridging federated image classification method. Background Technology

[0002] Federated learning achieves privacy-preserving joint modeling among multiple clients through a "data-without-leaving-domain, collaborative model parameter updates" approach. Existing federated image classification methods typically aggregate model parameters uploaded by clients directly on the server side, with a typical method being FedAvg, which uses a weighted average based on sample size.

[0003] However, in real-world distributed scenarios, the data from different clients often exhibits significant statistical heterogeneity, meaning that the data distribution, class ratios, and even class sets differ across clients. Especially when the "missing class" phenomenon exists, some clients cannot access samples of certain classes during local training, which can easily lead to: firstly, significant deviations in the client model update direction, causing instability in the globally aggregated representation space; and secondly, the local classifier's discrimination boundary for missing classes gradually collapses, resulting in catastrophic forgetting.

[0004] Existing methods often mitigate label skew through data augmentation or sample synthesis, but most still primarily focus on the quantity of samples, making it difficult to establish strictly consistent semantic references across different clients. Furthermore, the repair of decision boundaries for missing categories is often insufficient. Therefore, it is necessary to propose a federated image classification method that can simultaneously utilize global semantic priors and balance local discriminative capabilities with global consistency. Summary of the Invention

[0005] To address the issues of weight drift, missing category forgetting, and cross-client semantic inconsistency that arise in existing non-independent identically distributed federated learning technologies, this invention proposes a semantic bridging federated learning method. Semantic bridging refers to establishing a mapping path from the semantic space to the visual feature space of the student model on the server side during federated learning training, utilizing the global semantic prototypes provided by the pre-trained visual language model. The teacher semantic target generated by this mapping is then used to guide local training on the client side. Specifically, the server first constructs a global semantic prototype set. ,in This represents the semantic embedding vector extracted and normalized by the pre-trained CLIP model. Indicates the corresponding category label, This represents the total number of semantic prototypes. Subsequently, the server maps these semantic prototypes to the visual feature space where the student model backbone network output resides via a semantic adapter, forming teacher semantic features that can be used for client alignment and playback. Through this method, the present invention achieves efficient transfer of global semantic knowledge from the server side to the client's local visual model without accessing the client's original private data, thereby constructing a unified semantic bridging federated learning method.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A semantically bridging federated image classification method includes the following steps:

[0008] S1. The goal of establishing a federated optimization system is:

[0009] ,

[0010] in This indicates a loss in local surveillance tasks. Indicates the client The number of samples, This represents the number of clients; based on the optimization objective, the server and clients optimize the model. Perform random initialization.

[0011] S2. Construct a semantic feature set on the server. ,in This is the normalized CLIP embedding vector. Set its corresponding category label;

[0012] The server uses an adapter to embed and map the pre-trained visual language model CLIP into the visual feature space where the student model's backbone network output resides:

[0013] ,

[0014] in This represents the visual target features of the teacher obtained through semantic feature transformation. For the core feature dimensions of the student model, Indicates a semantic adapter. These are adapter parameters.

[0015] The server utilizes a global projector. and global classifier head The training aims to improve teachers' visual target features, with the following objectives:

[0016] ,

[0017] in, Represents the CLIP embedding vector. This indicates its corresponding label. This represents the cross-entropy loss.

[0018] Before each round of communication begins, the server freezes the core parameters of the student model and utilizes the semantic feature set. Training adapter Global Projector and global classifier head Then, the trained teacher semantic module is sent to the client.

[0019] S3, The client uses the frozen adapter. With global projector Generate teacher goals:

[0020] ,

[0021] The client aligns the semantic features between student batches and teacher batches obtained from the server, with the loss defined as:

[0022] ,

[0023] in, Let represent the feature vectors to be aligned on the student side and the teacher side, respectively. and Let these represent the feature vector sets of the student side and the teacher side, respectively. For temperature parameters, “·” indicates inner product operation.

[0024] The client applies the above alignment target at both the backbone and projection levels to obtain...

[0025] ,

[0026] The semantic alignment loss of the composition:

[0027] ,

[0028] in, These are non-negative weighting coefficients.

[0029] S4. The client randomly samples CLIP embedding vectors from the semantic prototypes corresponding to the missing categories. And via the frozen adapter Generate pseudo-visual features:

[0030] ,

[0031] The client inputs pseudo-visual features into the projection module and classifier head of the student model, and freezes the backbone encoder. Under the premise of updating the projector With classifier head The semantic replay loss is:

[0032] ,

[0033] in, The corresponding category label.

[0034] S5, Building Clients The final local training objective function is:

[0035] ,

[0036] in, It is a non-negative hyperparameter.

[0037] The client performs model training based on local data and training objectives, and uploads the trained model parameters to the server.

[0038] S6. The server performs weighted aggregation based on the client's model:

[0039] ,

[0040] in This represents the global model after aggregation. This represents the set of clients participating in the aggregation.

[0041] After aggregation is complete, the server uses the global model. Classify the test images;

[0042] S7. To meet the requirements of image classification tasks, firstly, select the model to be trained and collect the corresponding training data from public datasets; then, distribute the data to each client and complete the model parameter configuration; based on this, perform federated training on the model using the methods described in S1 to S6 respectively, and finally obtain the trained federated image classification model.

[0043] S8. Use the trained federated image classification model to classify the target image.

[0044] The beneficial effects of this invention are that the method of this invention utilizes the stable semantic information provided by the pre-trained visual language model CLIP to achieve cross-client semantic alignment, missing category compensation, and robust global model training in non-independent and identically distributed scenarios without accessing the client's original private data. Attached Figure Description

[0045] Figure 1 This is the overall flowchart of the present invention;

[0046] Figure 2 This is a schematic diagram of the client training process of the present invention. Detailed Implementation

[0047] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] Example:

[0049] This example demonstrates the classification performance of the method by combining it with a real-world image classification target. Figure 1 The diagram shows the overall process for this example, including semantic bridging federated learning on the server side and local model training on the client side. The specific method is as follows:

[0050] S1. Establish the optimization objective and initialize the model parameters:

[0051] First, establish the federal optimization objective:

[0052] ,

[0053] in This indicates a loss in local surveillance tasks. Indicates the client The number of samples, Indicates the number of clients.

[0054] Based on this optimization goal, the server and client optimize the model. Perform random initialization.

[0055] S2, Server-side training of semantic adapter:

[0056] First, the server constructs a semantic feature set. ,in This is the normalized CLIP embedding vector. Set its corresponding category label.

[0057] Then, the server uses an adapter to embed and map the pre-trained visual language model CLIP into the visual feature space where the student model backbone network output resides:

[0058] ,

[0059] in This represents the visual target features of the teacher obtained through semantic feature transformation. For the core feature dimensions of the student model, Indicates a semantic adapter. These are adapter parameters.

[0060] Finally, the server utilizes a global projector. and global classifier head The training for this teacher characteristic aims to achieve the following:

[0061] ,

[0062] in, Represents the CLIP embedding vector. This indicates its corresponding label. This represents the cross-entropy loss.

[0063] Before each round of communication begins, the server freezes the core parameters of the student model and utilizes the semantic feature set. Training adapter Global Projector and global classifier head Then, the trained teacher semantic module is sent to the client.

[0064] The client-side processing procedure is as follows: Figure 2 As shown, it includes the following steps:

[0065] S3. Client-side semantic alignment:

[0066] The client is based on the frozen adapter With global projector Generate teacher goals:

[0067] ,

[0068] Then, the client aligns the semantic features between the student batches and the teacher batches obtained from the server, with the loss defined as:

[0069] ,

[0070] in, Let represent the feature vectors to be aligned on the student side and the teacher side, respectively. and Let these represent the feature vector sets of the student side and the teacher side, respectively. For temperature parameters, “·” indicates inner product operation.

[0071] The client applies the above alignment target at both the backbone and projection levels to obtain...

[0072] ,

[0073] This leads to the semantic alignment loss of the combination:

[0074] ,

[0075] in, These are non-negative weighting coefficients.

[0076] S4. The client performs semantic replay:

[0077] The client randomly samples CLIP embedding vectors from the semantic prototypes corresponding to the missing categories. And via the frozen adapter Generate pseudo-visual features:

[0078] ,

[0079] Subsequently, the client inputs the pseudo-visual features into the projection module and classifier head of the student model, and freezes the backbone encoder. Under the premise of updating the projector With classifier head Therefore, the semantic replay loss is:

[0080] ,

[0081] in, The corresponding category label.

[0082] S5. The client constructs the overall objective function and trains the local model:

[0083] Client The final local training objective function is written as:

[0084] ,

[0085] in, These are non-negative hyperparameters. Then, the client performs model training based on local data and the aforementioned training objectives, and uploads the trained model parameters to the server.

[0086] S6, Server-side global aggregation model:

[0087] The server performs weighted aggregation based on the client's model.

[0088] ,

[0089] in This represents the global model after aggregation. This represents the set of clients participating in the aggregation. After aggregation is complete, the server applies the global model. A classification experiment was conducted on the test images.

[0090] S7. Construct and train a semantic bridging federated method based on image classification tasks:

[0091] Dataset selection and hyperparameter determination:

[0092] CIFAR-10, CIFAR-100, and Tiny-ImageNet were selected as image classification validation datasets. The data were partitioned using a Dirichlet distribution with non-independent and identically distributed elements, and the heterogeneity parameter β was set to 0.1, 0.5, and 1.0. The number of clients was set to 10, the number of communication rounds to 100, the number of local training rounds to 10, and the batch size to 64. The optimizer used was the stochastic gradient descent (SGD) optimizer, with an initial learning rate of 0.01, momentum of 0.9, and weight decay of 1e-5. The projection dimension was set to 256, and the temperature parameter was... The value was set to 0.07, the number of training rounds for the server-side semantic adapter was set to 5, and the server-side learning rate was set to 0.001.

[0093] S8. Validation experiments based on image classification datasets such as CIFAR-10, CIFAR-100, and Tiny-ImageNet;

[0094] The experimental results are shown in Table 1 below:

[0095] Table 1. Validation results of the image classification dataset

[0096]

[0097] On the CIFAR-10 dataset, when β=0.1 and β=0.5, the method of this invention achieves 79.84% and 87.09% respectively, both of which are superior to the comparative methods such as FedAvg, FedProx, MOON, FedMut and FedMR.

[0098] On the CIFAR-100 dataset, the improvement of the proposed method is even more significant. When β=0.1, the accuracy reaches 67.74%, an improvement of 4.07 percentage points compared to MOON's 63.67%; when β=0.5, the accuracy reaches 72.13%, an improvement of 5.03 percentage points compared to MOON's 67.10%, and an improvement of 7.29 percentage points compared to FedAvg's 64.84%. On the Tiny-ImageNet dataset, when β=0.5, the proposed method achieves 27.89%, an improvement of 3.50 percentage points compared to FedAvg's 24.39%. Experimental results show that the proposed method demonstrates a more significant performance improvement in scenarios with a large number of classes and high heterogeneity.

Claims

1. A semantic bridging federated image classification method, characterized in that, Includes the following steps: S1. The goal of establishing a federated optimization system is: ， in This indicates a loss in local surveillance tasks. Indicates the client The number of samples, This represents the number of clients; based on the optimization objective, the server and clients optimize the model. Perform random initialization; S2. Construct a semantic feature set on the server. ,in The normalized CLIP embedding vector. Set its corresponding category label; The server uses an adapter to embed and map the pre-trained visual language model CLIP into the visual feature space where the student model's backbone network output resides: ， in This represents the visual target features of the teacher obtained through semantic feature transformation. For the core feature dimensions of the student model, Indicates a semantic adapter. For adapter parameters; The server utilizes a global projector. and global classifier head The training aims to improve teachers' visual target features, with the following objectives: ， in, Represents the CLIP embedding vector. This indicates its corresponding label. Represents cross-entropy loss; Before each round of communication begins, the server freezes the core parameters of the student model and utilizes the semantic feature set. Training adapter Global Projector and global classifier head Then, the trained teacher semantic module is sent to the client. S3, The client uses the frozen adapter. With global projector Generate teacher goals: ， The client aligns the semantic features between student batches and teacher batches obtained from the server, with the loss defined as: ， in, Let represent the feature vectors to be aligned on the student side and the teacher side, respectively. and Let these represent the feature vector sets of the student side and the teacher side, respectively. For temperature parameters, "·" indicates inner product operation; The client applies the above alignment target at both the backbone and projection levels to obtain... ， The semantic alignment loss of the composition: ， in, These are non-negative weighting coefficients; S4. The client randomly samples CLIP embedding vectors from the semantic prototypes corresponding to the missing categories. And via the frozen adapter Generate pseudo-visual features: ， The client inputs pseudo-visual features into the projection module and classifier head of the student model, and freezes the backbone encoder. Under the premise of updating the projector With classifier head The semantic replay loss is: ， in, Corresponding category tags; S5, Building Clients The final local training objective function is: ， in, These are non-negative hyperparameters; The client performs model training based on local data and training objectives, and uploads the trained model parameters to the server. S6. The server performs weighted aggregation based on the client's model: ， in This represents the global model after aggregation. This represents the set of clients participating in the aggregation; After aggregation is complete, the server uses the global model. Classify the test images; S7. To meet the requirements of image classification tasks, firstly, select the model to be trained and collect the corresponding training data from public datasets; then, distribute the data to each client and complete the model parameter configuration; based on this, perform federated training on the model using the methods described in S1 to S6 respectively, and finally obtain the trained federated image classification model. S8. Use the trained federated image classification model to classify the target image.

Images