Federal chain type context optimization method for long-tail multi-label image classification
By introducing semantic correction vectors, hyperspecific label vocabularies, and asymmetric loss optimization methods into federated learning, the performance degradation caused by the long-tail distribution of multi-label image data is addressed, thereby improving classification accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HARBIN UNIV OF SCI & TECH
- Filing Date
- 2026-01-22
- Publication Date
- 2026-05-01
AI Technical Summary
Existing federated learning methods struggle to effectively handle the long-tail distribution of multi-label image data, leading to a decline in global model performance, especially in real-world scenarios.
A federated chain context optimization method for long-tail multi-label image classification is adopted. By initializing semantic correction vectors, upper-level label vocabulary and global model parameters, k-means++ clustering and asymmetric loss (ASL) are combined to optimize local model training, dynamically adjust class weights and loss weights, and achieve aggregation of the global model.
It improves the classification accuracy of the global model in long-tail multi-label image classification tasks, enhances the model's attention to and recognition of tail categories, and improves the robustness and practicality of federated learning.
Smart Images
Figure CN121962749A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a federated chain context optimization method for long-tail multi-label image classification, belonging to the field of federated learning. Background Technology
[0002] Federated Learning (FL), as an emerging distributed machine learning paradigm, has received widespread attention in recent years. Compared to traditional centralized learning methods, federated learning effectively alleviates the data silo problem by distributing the model training process across multiple data holders. FedAvg allows users to train models using local data and share updated model parameters with a central server for aggregation, thus achieving cross-device federated learning. However, in practical applications, due to significant differences in data volume, quality, and distribution among different devices or data sources, this data heterogeneity can lead to a decline in global model performance, thereby affecting the effectiveness and robustness of federated learning. Therefore, addressing the data heterogeneity problem has become one of the important research directions in the field of federated learning.
[0003] Existing methods primarily rely on class-balanced datasets, using sampling algorithms to distribute data across multiple clients, simulating data heterogeneity by ensuring that the data across clients exhibits a non-independent and identically distributed (Non-IID) pattern. Such datasets are often single-labeled with a uniformly distributed target label, such as CIFAR, MNIST, or Tiny-ImageNet. However, in the real world, data often has multiple labels, and the distribution of target labels generally follows a long-tailed distribution. Existing federated learning methods based on the class-balanced assumption struggle to effectively handle such problems, resulting in performance limitations in real-world scenarios. Summary of the Invention
[0004] This invention aims to address the challenge of declining global model performance in federated learning scenarios due to the dual characteristics of multi-label association and long-tail distribution of data. It proposes a federated chain-based context optimization method for long-tail multi-label image classification.
[0005] The technical solution of the present invention:
[0006] A federated chain context optimization method for long-tail multi-label image classification includes a federated learning training phase and a multi-label image classification phase.
[0007] The federated learning training phase includes the following steps: S1, the server initializes the semantic correction vector, the upper-level label vocabulary, and the global model parameters, and sends the initialized semantic correction vector, upper-level label vocabulary, and global model parameters to each client; S2, each client receives the upper-level label vocabulary and obtains the corresponding local cluster label from the upper-level label vocabulary according to the category label set, where the category label set includes image and category labels; S3, the client receives the semantic correction vector and the global model parameters, and obtains the context vector; S4, the client combines the local cluster label, semantic correction vector, context vector, and category label into a complete semantic descriptive vector; S5, the image and the semantic descriptive vector are used as training data to input the local model for training; S6, the client sends the trained semantic correction vector, context vector, and local model parameters to the server; S7, the server clusters... The semantic correction vector, context vector, and local model parameters received from each client are combined to obtain the aggregated global semantic correction vector, global context vector, and global model parameters; S8, the server sends the aggregated semantic correction vector, context vector, and global model parameters to each client; S9, steps S3 to S8 are repeated for a total of R-1 rounds to finally obtain the trained global model parameters; where R is the preset total number of communication rounds between the server and the client; the multi-label image classification stage includes the following steps: S10, for the image to be classified, the category label of its candidate category is input into a preset prompt word template to generate corresponding inference prompt words; S11, the global model parameters finally obtained in the training stage are loaded into the image classification model to form a trained global model; the image to be classified and the inference prompt words generated in step S10 are input into the trained global model to output the multi-label classification result.
[0008] Specifically, in step S1, the method for initializing the semantic correction vector and global model parameters is as follows: before federated training begins, the semantic correction vector and global model parameters on the server are randomly initialized.
[0009] The global model parameters are initialized by independently sampling each parameter value from a zero-mean Gaussian distribution with a variance of less than 0.01; the semantic correction vector is initialized by sampling from a zero-mean Gaussian distribution with a standard deviation of 0.02.
[0010] Specifically, in step S1, the method for initializing the upper-level label vocabulary is as follows: based on the hierarchical structure defined by the Open Images V7 dataset, the upper-level category names are selected to construct the upper-level label vocabulary; the upper-level label vocabulary contains 30 coarse-grained category labels.
[0011] Specifically, in step S2, the method for obtaining local cluster labels includes: S2.1, using a pre-trained text encoding model MiniLM-L6 to encode all category labels on the client and all labels in the superordinate label vocabulary to obtain corresponding label embedding vectors; S2.2, using the k-means++ clustering algorithm to cluster the embedding vectors of all category labels on the client; S2.3, for each cluster, calculating the cosine similarity between its cluster center and all label embedding vectors in the superordinate label vocabulary, and selecting the superordinate label with the highest cosine similarity as the local cluster label of that cluster; S2.4, automatically determining the optimal number of clusters k using the silhouette coefficient method: calculating the silhouette coefficient corresponding to k values from 2 to 10, and selecting the k value with the highest silhouette coefficient score as the number of clusters for the k-means++ algorithm.
[0012] Specifically, in step S3, the method for obtaining the context vector is as follows: during the first round of communication between the server and the client, the context vector is initialized by the client itself, and the initialization method is: sampling from a zero-mean Gaussian distribution with a standard deviation of 0.02; during the second to Rth rounds of communication, the context vector received by the client is the global context vector issued by the server in the previous round in step S7.
[0013] Specifically, in step S4, the method for combining semantic descriptive vectors is as follows:
[0014] S4.1 Semantic Descriptive Vector The implementation process is as follows:
[0015]
[0016] in, Indicates the local cluster label. Indicates length is semantic correction vector , Indicates length is context vector , S4.2. Based on the number of samples of each category in the training dataset, the categories are divided into head categories, middle categories, and tail categories, and the category weights are dynamically adjusted. , is represented as:
[0017]
[0018] in, These are learnable hyperparameters, representing the weights of the head, middle, and tail categories, respectively.
[0019] S4.3, The semantic descriptive vector is represented as follows:
[0020] .
[0021] Specifically, in step S5, the asymmetric loss ASL is introduced to optimize the local model and train the local model, based on the model's performance on the first... All samples Prediction results for each category The loss weight is dynamically adjusted, as follows:
[0022]
[0023] in, is a hyperparameter representing its contribution to the loss on both positive and negative samples; and These represent the positive loss portion and the negative loss portion, respectively. Represents the asymmetric probability shift, where is a hyperparameter representing the probability boundary.
[0024] Specifically, the method for aggregating the semantic correction vector, context vector, and local model parameters described in step S7 is as follows:
[0025] After local training is complete, the client Upload semantic correction vector Context vector and local model parameters On the server side, vector and model aggregation is achieved using an average aggregation strategy; the server then aggregates the vectors and models by... Semantic correction vector of communication round Context vector and local model parameters Aggregate to obtain the global vector and global model parameters, represented as:
[0026]
[0027]
[0028]
[0029] in, Represents the complete dataset. Indicates the first Datasets for each client.
[0030] Specifically, step S10 uses a manual prompt: "The photo of a..." "." serves as a clue template for inference.
[0031] The beneficial effects of this invention are:
[0032] This invention effectively mitigates the impact of sample imbalance and label association. Experiments show that this invention can improve the performance of the global model and achieve higher classification accuracy in federated long-tail multi-label image classification tasks. In particular, it enhances the model's attention to and recognition of tail categories through category-aware weights and asymmetric loss, thereby improving the practicality and robustness of federated learning in complex real-world data. Attached Figure Description
[0033] Figure 1 This is the overall framework diagram;
[0034] Figure 2 The average precision for each class on the VOC-LT dataset;
[0035] Figure 3 This represents the average precision for each category on the COCO-LT dataset. Detailed Implementation
[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. The specific embodiments described in this section are intended to explain the present invention, rather than to limit the scope of protection of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0037] This embodiment demonstrates the application process and effects of the method described in this invention on the public dataset VOC-LT.
[0038] The specific steps of the federated learning training phase are as follows:
[0039] S1. The server initializes the semantic correction vector, the upper-level tag vocabulary, and the global model parameters, and sends the initialized semantic correction vector, upper-level tag vocabulary, and global model parameters to each client:
[0040] Each parameter of the global model is independently sampled from a zero-mean Gaussian distribution with a variance of less than 0.01; the semantic correction vector is sampled from a zero-mean Gaussian distribution with a standard deviation of 0.02; the upper-level categories of the Open Images V7 dataset are selected to create the upper-level label vocabulary, thus completing the initialization of the upper-level label vocabulary. This upper-level category contains 30 coarse-grained labels, which can ensure comprehensive coverage of the real-world category space;
[0041] After the above processing, the server sends the semantic correction vector with a length of 8 and a dimension of 512, the upper-level tag vocabulary containing 30 tags, and the initialization parameters of the global model to the client.
[0042] S2. Each piece of data in the client contains an image and category label. After receiving the upper-level label vocabulary, it performs initialization and retrieves the local cluster label from the table.
[0043] Each client uses the lightweight pre-trained model MiniLM-L6 to encode its category labels and labels in the upper-level label vocabulary. Then, the k-means++ algorithm is used to cluster the encoded category label embeddings. Local cluster labels are selected from the vocabulary by calculating the cosine similarity between each cluster center and the encoded upper-level label vocabulary embedding. To automatically determine the optimal number of clusters, silhouette coefficients with k values from 2 to 10 are calculated, and the k value with the highest score is selected as the optimal number of clusters.
[0044] After the above processing, each client obtains a local cluster label corresponding to all category labels on its local machine.
[0045] S3. The client receives the semantic correction vector and global model parameters, and obtains the context vector;
[0046] When the server and client communicate for the first time, the context vector is obtained by the client for initialization. Specifically, the context vector is sampled from a zero-mean Gaussian distribution with a standard deviation of 0.02; otherwise, the context vector comes from the server's transmission.
[0047] After the above processing, the client obtains a context vector with a length of 8 and a dimension of 512.
[0048] S4. The client combines the local cluster label, semantic correction vector, context vector, and category label into a complete semantic descriptive vector;
[0049] To enable a step-by-step reasoning process from local cluster labels to specific category labels, semantic descriptive vectors... The implementation process can be represented as follows:
[0050]
[0051] in, Indicates the local cluster label. Indicates length is semantic correction vector , Indicates length is context vector , Indicates category label;
[0052] In long-tail multi-label classification tasks, the data distribution often exhibits severe class imbalance. To address this, a class-aware weighting method is proposed to enhance the model's attention to tail categories. Specifically, categories are divided into head categories based on the frequency of class samples. (More than 100 samples per category), Middle Category (20 to 100 samples per class) and tail categories (Less than 20 samples per class), then dynamically adjust the class weights. ; can be represented as:
[0053]
[0054] in, These are learnable hyperparameters, representing the weights of the head, middle, and tail categories, respectively; ultimately, the semantic descriptive vector can be represented as:
[0055]
[0056] Each client participating in the training will combine the local cluster labels, semantic correction vectors, context vectors, and category labels obtained in S1-S3 into a complete semantic descriptive vector.
[0057] S5. Input the image and semantic descriptive vector as training data into the local model for training;
[0058] Asymmetric Loss (ASL) is introduced to optimize the local model; specifically, ASL optimizes the model based on its performance on the first... All samples Prediction results for each category The loss weights are dynamically adjusted to improve the model's ability to identify tail categories; the entire process can be represented as:
[0059]
[0060] in, It is a hyperparameter that indicates its value in positive... burden Loss contribution on the sample; and These represent the positive loss portion and the negative loss portion, respectively. Represents the asymmetric probability shift, where It is a hyperparameter representing the probability boundary;
[0061] All participating clients work together to train the model, and through the above process, they obtain the trained semantic correction vector, context vector, and local model parameters.
[0062] S6. The client sends the trained semantic correction vector, context vector, and local model parameters to the server.
[0063] All clients participating in the training will send the trained semantic correction vector, context vector, and local model parameters to the server.
[0064] S7. Server-side aggregation of semantic correction vectors, context vectors, and local model parameters;
[0065] After local training is complete, the client uploads the learnable vectors and model parameters to the server; specifically, the client... Upload semantic correction vector Context vector and local model parameters The data is then sent to the server, where an average aggregation strategy is used to aggregate the vectors and the model. Therefore, the server aggregates the vectors and the model by... Semantic correction vector of communication round Context vector and local model parameters Aggregate to obtain global vectors and global model parameters:
[0066]
[0067]
[0068]
[0069] in, Represents the complete dataset. Indicates the first Data sets for each client;
[0070] After receiving the semantic correction vector, context vector, and local model parameters from each participating client in this round, the server performs the above process to complete the average aggregation of all semantic correction vectors, context vectors, and local model parameters.
[0071] S8. The server sends the aggregated semantic correction vector, context vector, and global model parameters to the client.
[0072] The server sends the aggregated semantic correction vector, context vector, and global model parameters to the client participating in the new round of training.
[0073] S9. Repeat steps S3-S8 for round R-1 to obtain the trained global model parameters, where R represents the number of rounds of communication between the server and the client.
[0074] Repeat the entire federated learning training process 50 times to obtain the trained global model parameters. The overall framework is as follows: Figure 1 As shown.
[0075] The multi-label image classification stage includes the following steps:
[0076] S10. Input the category label into the prompt word template to obtain reasoning prompt words;
[0077] Using manual prompts "The photo of a The "." serves as a template for inference clues; by filling in the category labels into the template, inference clues can be obtained.
[0078] Input all possible candidate category labels corresponding to image 2010_002373 into the inference prompt word template to obtain several inference prompt words.
[0079] S11. Load the trained global model parameters into the global model, and input the images and inference prompts as inference data into the global model for multi-label image classification.
[0080] All the reasoning prompts for the image are input together with the image into the trained global model, and the final categories corresponding to image 2010_002373 are dining table, bottle and person.
[0081] In this embodiment, VOC-LT contains 20 categories and a total of 6094 images, of which the training set contains 1142 images and the test set contains 4952 images. In VOC-LT, the distribution ratio of head, middle, and tail categories is 6:6:8, with the number of images in each category ranging from 4 to 775, and the number of categories per image ranging from 1 to 6. Additionally, COCO-LT contains 80 categories and a total of 6909 images, of which the training set contains 1909 images and the test set contains 5000 images. In COCO-LT, the distribution ratio of head, middle, and tail categories is 22:33:25, with the number of images in each category ranging from 6 to 1128, and the number of categories per image ranging from 1 to 14.
[0082] To demonstrate the effectiveness of this invention, the following comparisons are made:
[0083] 1. If the semantic descriptive vector is replaced with the manual prompt "The photo of a..." The classification accuracy is:
[0084] VOC-LT: 82.38%, a decrease of 1.99% compared to case 6;
[0085] COCO-LT: 59.81%, a decrease of 2.20% compared to case 6;
[0086] 2. If only the context vector and category label from the semantic descriptive vector are retained, the classification accuracy is:
[0087] VOC-LT: 83.15%, a decrease of 1.22% compared to case 6;
[0088] COCO-LT: 61.27%, a decrease of 0.74% compared to case 6;
[0089] 3. If the method of this invention is fully implemented for federated long-tail multi-label image classification, the classification accuracy is:
[0090] VOC-LT: 84.37%, average accuracy for each category as follows: Figure 2 As shown;
[0091] COCO-LT: 62.01%, average accuracy across categories as follows Figure 3 As shown.
[0092] As can be seen, the method proposed in this invention can improve the performance of the global model and achieve higher classification accuracy in federated long-tailed multi-label image classification tasks.
Claims
1. A federated chain context optimization method for long-tail multi-label image classification, characterized in that, This includes the federated learning training phase and the multi-label image classification phase; The federated learning training phase includes the following steps: S1, the server initializes the semantic correction vector, the upper-level label vocabulary, and the global model parameters, and sends the initialized semantic correction vector, upper-level label vocabulary, and global model parameters to each client; S2, each client receives the upper-level label vocabulary and obtains the corresponding local cluster label from the upper-level label vocabulary according to the category label set, where the category label set includes image and category labels; S3, the client receives the semantic correction vector and the global model parameters, and obtains the context vector; S4, the client combines the local cluster label, semantic correction vector, context vector, and category label into a complete semantic descriptive vector; S5, the image and the semantic descriptive vector are used as training data to input the local model for training; S6, the client sends the trained semantic correction vector, context vector, and local model parameters to the server; S7, the server clusters... The semantic correction vector, context vector, and local model parameters received from each client are combined to obtain the aggregated global semantic correction vector, global context vector, and global model parameters; S8, the server sends the aggregated semantic correction vector, context vector, and global model parameters to each client; S9, steps S3 to S8 are repeated for a total of R-1 rounds to finally obtain the trained global model parameters; where R is the preset total number of communication rounds between the server and the client; the multi-label image classification stage includes the following steps: S10, for the image to be classified, the category label of its candidate category is input into a preset prompt word template to generate corresponding inference prompt words; S11, the global model parameters finally obtained in the training stage are loaded into the image classification model to form a trained global model; the image to be classified and the inference prompt words generated in step S10 are input into the trained global model to output the multi-label classification result.
2. The federated chain context optimization method for long-tail multi-label image classification according to claim 1, characterized in that, In step S1, the method for initializing the semantic correction vector and global model parameters is as follows: before federated training begins, the semantic correction vector and global model parameters on the server are randomly initialized. The global model parameters are initialized by independently sampling each parameter value from a zero-mean Gaussian distribution with a variance of less than 0.01; the semantic correction vector is initialized by sampling from a zero-mean Gaussian distribution with a standard deviation of 0.
02.
3. The federated chain context optimization method for long-tail multi-label image classification according to claim 1, characterized in that, In step S1, the method for initializing the upper-level label vocabulary is as follows: based on the hierarchical structure defined by the Open Images V7 dataset, the upper-level category names are selected to construct the upper-level label vocabulary; the upper-level label vocabulary contains 30 coarse-grained category labels.
4. The federated chain context optimization method for long-tail multi-label image classification according to claim 1, characterized in that, In step S2, the method for obtaining local cluster labels specifically includes: S2.1, using the pre-trained text encoding model MiniLM-L6 to encode all category labels locally on the client and all labels in the superordinate label vocabulary to obtain corresponding label embedding vectors; S2.2, using the k-means++ clustering algorithm to cluster the embedding vectors of all category labels locally on the client; S2.3, for each cluster, calculating the cosine similarity between its cluster center and all label embedding vectors in the superordinate label vocabulary, and selecting the superordinate label with the highest cosine similarity as the local cluster label of that cluster; S2.4, automatically determining the optimal number of clusters k using the silhouette coefficient method: calculating the silhouette coefficient corresponding to k values from 2 to 10, and selecting the k value with the highest silhouette coefficient score as the number of clusters for the k-means++ algorithm.
5. The federated chain context optimization method for long-tail multi-label image classification according to claim 1, characterized in that, In step S3, the method for obtaining the context vector is as follows: during the first round of communication between the server and the client, the context vector is initialized by the client itself, and the initialization method is: sampling from a zero-mean Gaussian distribution with a standard deviation of 0.02; during the second to the Rth rounds of communication, the context vector received by the client is the global context vector issued by the server in the previous round step S7.
6. The federated chain context optimization method for long-tail multi-label image classification according to claim 1, characterized in that, In step S4, the method for combining semantic descriptive vectors is as follows: S4.1 Semantic Descriptive Vector The implementation process is as follows: ； in, Indicates the local cluster label. Indicates length is semantic correction vector , Indicates length is context vector , S4.
2. Based on the number of samples of each category in the training dataset, the categories are divided into head categories, middle categories, and tail categories, and the category weights are dynamically adjusted. , is represented as: ； in, These are learnable hyperparameters, representing the weights of the head, middle, and tail categories, respectively. S4.3, The semantic descriptive vector is represented as follows: 。 7. The federated chain context optimization method for long-tail multi-label image classification according to claim 1, characterized in that, In step S5, asymmetric loss (ASL) is introduced to optimize the local model and train the local model. Based on the model's performance on the first... All samples Prediction results for each category The loss weight is dynamically adjusted, as follows: ； in, is a hyperparameter representing its contribution to the loss on both positive and negative samples; and These represent the positive loss portion and the negative loss portion, respectively. Represents the asymmetric probability shift, where is a hyperparameter representing the probability boundary.
8. The federated chain context optimization method for long-tail multi-label image classification according to claim 1, characterized in that, The method for aggregating semantic correction vectors, context vectors, and local model parameters described in step S7 is as follows: After local training is complete, the client Upload semantic correction vector Context vector and local model parameters On the server side, vector and model aggregation is achieved using an average aggregation strategy; the server then aggregates the vectors and models by... Semantic correction vector of communication round Context vector and local model parameters Aggregate to obtain the global vector and global model parameters, represented as: ； ； ； in, Represents the complete dataset. Indicates the first Datasets for each client.
9. The federated chain context optimization method for long-tail multi-label image classification according to claim 1, characterized in that, Step S10 uses a manual prompt: "The photo of a" "." serves as a clue template for inference.