A federated image classification method and system based on visual guidance attribute correction

Abstract

This invention belongs to the technical field of federated learning, and more specifically, relates to a federated image classification method and system based on visually guided attribute correction. The method includes: a server constructing a global model for image classification, comprising an image encoder, a text encoder, a cue learning module, and a gated text correction module, and distributing trainable parameters to each client participating in the federated learning; each client updating the parameters in its initial local model based on the trainable parameters distributed by the server, and training the updated local model using its local image dataset; and uploading the parameters from the trained local model to the server for aggregation to obtain the target global model. This invention addresses the shortcomings of existing attribute-assisted federated learning methods, such as static and rigid text attributes, inability to adapt to heterogeneous data with non-independent and identically distributed characteristics, and lack of cross-modal dynamic interaction mechanisms.

Description

Technical Field

[0001] This invention belongs to the technical field of federated learning, and more specifically, relates to a federated image classification method and system based on visually guided attribute correction. Background Technology

[0002] With the rapid evolution of deep learning technology, AI models driven by massive amounts of data have made breakthroughs in fields such as image classification. However, in real-world applications such as healthcare and finance, high-quality data is often scattered across different user terminals, and due to data privacy regulations, raw data cannot be aggregated to a central server for centralized training. Federated learning, as an emerging distributed privacy-preserving computing paradigm, allows participants to collaboratively train models without leaving their local machines, effectively resolving the contradiction between data silos and privacy protection. Nevertheless, in scenarios involving non-independent and identically distributed data with significant differences among clients, federated learning still faces serious performance bottlenecks, as a single global model often struggles to adapt to the personalized data characteristics of all clients.

[0003] To enhance the generalization and few-shot learning capabilities of models in heterogeneous data environments, leveraging large-scale vision-language pre-trained models to assist federated learning has become a cutting-edge trend. These methods typically introduce fine-grained textual attribute descriptions generated by large language models as prior knowledge, aiming to enhance the discriminative power of visual features through cross-modal interaction. Existing methods guide the model to focus on key regions in images by injecting textual attributes such as "has stripes" and "long tails" into the visual encoder.

[0004] For example, Chinese patent document CN120635237A discloses a remote sensing image generation method based on a federated visual language model. The method includes: each client receiving a text instruction description; extracting multi-scale feature maps from private remote sensing image data through a visual encoder, while simultaneously generating semantic embedding vectors from the text instruction description through a language encoder; inputting the semantic embedding vectors and multi-scale feature maps into a dynamic attention mask generator to generate a pixel-level spatial weight distribution; performing a weighted fusion operation on the multi-scale feature maps to generate a text-conditional visual feature representation; generating a remote sensing image conforming to the text instruction description through an image decoder; the client uploading model parameter increments from the visual encoder, language encoder, and dynamic attention mask generator to a central server; and the central server aggregating the model parameter increments and distributing the updated global model parameters to each client.

[0005] However, existing attribute-assisted federated image classification methods still have significant shortcomings in their processing mechanisms. On the one hand, their textual knowledge injection is passive and static; that is, regardless of changes in the content of the input image, the model mechanically uses the same set of generic textual attributes for matching, lacking the ability to dynamically adjust the focus based on specific image instances. On the other hand, there is a natural distributional bias between generic textual attributes and the local image features of a specific client; unmodified generic descriptions often fail to accurately cover the unique visual patterns of local data. This one-way knowledge injection mechanism, lacking visual guidance, makes it difficult to achieve deep semantic alignment between visual and linguistic modalities. In a highly heterogeneous federated environment, it may even limit the model's local classification performance due to the introduction of noise. Summary of the Invention

[0006] The present invention aims to overcome at least one of the defects of the prior art and provide a federated image classification method based on visually guided attribute correction, so as to solve the defects of existing attribute-assisted federated learning methods, such as static and rigid text attributes, inability to adapt to heterogeneous data with non-independent and identical distribution, and lack of cross-modal dynamic interaction mechanism.

[0007] The present invention also discloses a system loaded with the aforementioned federated image classification method based on visually guided attribute correction.

[0008] The detailed technical solution of this invention is as follows:

[0009] A federated image classification method based on visually guided attribute correction, the method comprising:

[0010] S1. Construct a global model for federated learning on the server side. The global model is used for image classification and includes an image encoder, a text encoder, a cue learning module, and a gated text correction module. The parameters of the cue learning module and the gated text correction module are distributed as trainable parameters to each client participating in federated learning.

[0011] S2. Construct an initial local model on the client that has the same structure as the global model, and update the parameters of the prompt learning module and the gated text correction module in the initial local model based on the trainable parameters issued by the server. Then, train the local model with updated parameters using the client's local image dataset, including:

[0012] The text encoder based on the local model obtains the original text features of the local image dataset;

[0013] The image encoder based on the local model obtains image features from the local image dataset and constructs a batch-level visual context representation to correct the original text features;

[0014] A joint optimization objective is constructed based on the corrected text features to train the local model with updated parameters;

[0015] S3. Upload the parameters of the prompt learning module and the gated text correction module in the trained local model to the server for aggregation, in order to update the trainable parameters of the global model and obtain the target global model.

[0016] According to a preferred embodiment of the present invention, in step S1, the global model constructed by the server is:

[0017] (1)

[0018] in, This indicates a frozen image encoder. Indicates a frozen text encoder. This indicates a trainable cue learning module. This represents a trainable gated text correction module;

[0019] The initial trainable parameters are:

[0020] (2)

[0021] in, This indicates the initial parameters for the learning module. This represents the initial parameters of the gated text correction module.

[0022] According to a preferred embodiment of the present invention, in step S2, the text encoder based on the local model obtains the original text features of the local image dataset, including:

[0023] Obtain the image category set from the client's local image dataset;

[0024] For each category in the image category set, construct multiple corresponding attribute-level text descriptions;

[0025] The constructed attribute-level text description is encoded using the text encoder of the local model to obtain the original text features of the local image dataset.

[0026] According to a preferred embodiment of the present invention, the constructed attribute-level text description is encoded using the text encoder of the local model, specifically as follows:

[0027] (3)

[0028] in, This represents the set of original text features corresponding to the attribute-level text descriptions of all categories in the local image dataset; Indicates the first A subset of the original text features corresponding to each category; Represents a set of image categories; Represents the union;

[0029] and,

[0030] (4)

[0031] in, Indicates the current client's local number The corresponding category is the The raw text features obtained by mapping attribute-level text descriptions through a text encoder Indicates the first The total number of attribute-level text descriptions corresponding to the class. Indicates the text embedding dimension.

[0032] According to a preferred embodiment of the present invention, in step S2, the image encoder based on the local model obtains image features from the local image dataset and constructs a batch-level visual context representation, specifically as follows:

[0033] (5)

[0034] in, This represents a batch-level visual context representation; Indicates the first in the current training batch Feature representation of each image.

[0035] According to a preferred embodiment of the present invention, in step S2, the original text features are corrected using the batch-level visual context representation, specifically as follows:

[0036] The original text features and batch-level visual context representation are jointly input into the gated text correction module of the local model. The gated text correction module includes a correction subnetwork and a gated subnetwork.

[0037] The correction subnetwork takes the original text features as input and generates a correction vector for the original text features through nonlinear mapping. ;

[0038] The gating subnetwork takes the combination of the original text features and batch-level visual context representation as input, and outputs gating coefficients located in the interval [0,1]. ;

[0039] The original text features are corrected based on the correction vector and gating coefficients:

[0040] (6)

[0041] in, Indicates the current client's local number The corresponding category is the The original text features obtained by mapping attribute-level text descriptions through a text encoder. Representing the original text features The corresponding correction vector, Indicates the gating coefficient. Representing the original text features The corresponding corrected text features.

[0042] According to a preferred embodiment of the present invention, in step S2, constructing a joint optimization objective based on the corrected text features to train the local model with updated parameters includes:

[0043] Based on the corrected text features and the original images in the client-side local image dataset, the conditional image feature representation of the text is obtained using the prompting learning module of the client-side local model;

[0044] A classification loss is constructed based on the text-conditionalized image feature representation and the corrected text features;

[0045] A visual consistency loss is constructed based on the text-conditional image feature representation and the image features of the enhanced original image.

[0046] The classification loss and visual consistency loss are combined to train the local model with updated parameters.

[0047] According to a preferred embodiment of the present invention, the classification loss is obtained by calculating the similarity between the text-conditionalized image feature representation and the corrected text features of the same category, wherein the text features of the same category are aggregated from multiple attribute descriptions belonging to the same category to obtain the category prediction probability. Thus, the classification loss is obtained. :

[0048] (7)

[0049] (8)

[0050] (9)

[0051] in, This represents the temperature parameter and adjusts the smoothness of the probability distribution. Represents text-conditional image feature representation; Indicates aggregation operation; It is a text feature obtained by aggregating multiple attribute descriptions of the same category after correction. Indicates category label.

[0052] According to a preferred embodiment of the present invention, the visual consistency loss is:

[0053] (10)

[0054] in, Represents text-conditional image feature representation; This represents the image features of the enhanced original image.

[0055] In another aspect of the present invention, a federated image classification system based on visually guided attribute correction is provided, comprising a server participating in federated learning and multiple clients communicating with the server, wherein the clients participating in federated learning and the server perform federated learning using the above-described federated image classification method.

[0056] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0057] (1) This invention introduces a visually guided gating network and a residual correction mechanism, enabling the model to dynamically judge and correct the semantic expression of text attributes based on the current visual context of the input. This dynamic mechanism, which is "dependent on the specific image," allows general knowledge to adaptively migrate to the unique local data distribution of each client, thereby effectively solving the problem of model performance degradation caused by mismatch of general attributes in non-independent and identically distributed scenarios, and significantly improving the accuracy of local classification.

[0058] (2) This invention changes the traditional method of unidirectional matching of image features with original text. It innovatively proposes a symmetric semantic alignment strategy, which constructs a complete gradient closed loop by forcing the enhanced image features to perform classification matching with the corrected text features. This ensures that the text correction module can be directly supervised by the classification target, so that the learned text offset is always optimized in a direction that is conducive to classification decision, avoiding the semantic drift risk caused by blind correction, and ensuring the stability of model convergence and the discriminativeness of feature representation.

[0059] (3) This invention introduces a visual consistency constraint based on random erasure. By forcing the local model to fit the feature representation of the teacher model, it forces the model to learn the global contextual information and structured semantics of the image, rather than relying solely on local salient features. This effectively prevents catastrophic forgetting, so that the final aggregated global model not only performs well on known categories, but also has stronger robustness and generalization ability when facing complex testing environments such as occlusion and noise. Attached Figure Description

[0060] Figure 1 This is a flowchart of the federated image classification method based on visually guided attribute correction as described in this invention.

[0061] Figure 2 This is a schematic diagram of the implementation process of the federated image classification method described in Embodiment 1 of the present invention.

[0062] Figure 3 This is an experimental result figure of the local task effectiveness test of the gated text correction module on the Flowers102 dataset in the experimental example of this invention.

[0063] Figure 4 This is an experimental result figure showing the effectiveness test of random erasure regularization on the Flowerers102 dataset in the experimental examples of this invention.

[0064] Figure 5 This is an experimental result graph showing the generalization performance test of this invention on the Flowers102 dataset in the experimental examples of this invention. Detailed Implementation

[0065] This invention introduces visual context as a priori conditions to construct a cross-modal adaptive gating network, enabling instance-level dynamic residual correction of general text attributes generated by large language models. Simultaneously, by combining symmetric semantic alignment and random erasure consistency strategies, it significantly improves the local classification accuracy and robustness of the model on heterogeneous clients while protecting data privacy. The following description, in conjunction with the accompanying drawings and embodiments, further illustrates this disclosure.

[0066] Example 1

[0067] In this embodiment, the federated learning system consists of a server and multiple clients communicating with the server. Let the client set be... , This represents the total number of clients. Each client holds its own independent, non-shareable local dataset, and the data distribution varies significantly across different clients. Furthermore, the local dataset held by each client can be a local image dataset, such as the Flowers102 dataset.

[0068] The core idea of ​​this embodiment is to keep the parameters of the multimodal base model frozen, and only perform federated updates on the lightweight text prompts and text correction parameters, and suppress client model drift through visual consistency constraints.

[0069] like Figure 1 and Figure 2 As shown, this embodiment provides a federated image classification method based on visually guided attribute correction, the method specifically including:

[0070] S1. Construct a global model for federated learning on the server side. The global model is used for image classification and includes an image encoder, a text encoder, a cue learning module, and a gated text correction module. The parameters of the cue learning module and the gated text correction module are distributed as trainable parameters to each client participating in federated learning.

[0071] The server builds and initializes the global model for federated learning, and distributes the corresponding trainable parameters in the global model to each federated learning client to update the parameters in the local model of each client.

[0072] Specifically, the global model built on the server side comprises the following components: a frozen multimodal base model backbone, a trainable cue learning module, and a trainable gated text correction module; wherein, the multimodal base model backbone includes an image encoder and a text encoder. Based on this, the global model built on the server side can be formally represented as:

[0073] (1)

[0074] in, This indicates a frozen image encoder. Indicates a frozen text encoder. This indicates a trainable cue learning module. This represents a trainable gated text correction module.

[0075] During the initialization phase, the frozen encoder parameters remain unchanged throughout the federated training process, thus providing a unified semantic representation foundation for different clients. The server only initializes and updates the parameters of trainable modules, namely the parameters of the cue learning module and the gated text correction module. Let the initialized trainable parameters be denoted as... :

[0076] (2)

[0077] in, This indicates the initial parameters for the learning module. This represents the initial parameters of the gated text correction module.

[0078] Finally, initialize the trainable parameters. It was distributed to all clients participating in federated learning.

[0079] S2. Construct an initial local model on the client with the same structure as the global model, and update the parameters of the prompt learning module and the gated text correction module in the initial local model based on the trainable parameters issued by the server, and train the local model with updated parameters using the local image dataset of the client.

[0080] In this step, each client participating in federated learning has a local model structure consistent with the server. Each client receives the trainable parameters of the global model from the server and updates its local model parameters. That is, the client uses the trainable parameters from the server to overwrite the parameters of its local model's prompting learning module and gated text correction module, while the frozen multimodal backbone remains unchanged.

[0081] Understandably, in the first round of communication, i.e., r = 1, where r represents the communication round and is a positive integer, the trainable parameters sent by the server are: In the second and subsequent communication rounds, i.e., r ≥ 2, the trainable parameters sent by the server are: .

[0082] For the local model with updated parameters, it is trained using the client's local image dataset, including:

[0083] The text encoder based on the local model obtains the original text features of the local image dataset;

[0084] The image encoder based on the local model obtains image features from the local image dataset and constructs a batch-level visual context representation to correct the original text features;

[0085] A joint optimization objective is constructed based on the corrected text features to train the local model with updated parameters.

[0086] In this embodiment, the text encoder based on the local model obtains the original text features of the local image dataset, specifically including:

[0087] Attribute-level text description acquisition: Each federated learning client prepares features based on its local image dataset, which contains image data of multiple categories. The client determines the set of available image categories from its local image dataset. And for image category sets Each category loads multiple attribute-level text descriptions, and each category corresponds to multiple attribute perspectives, forming a subset of attribute text for each category. This subset is used to characterize the semantic features of the category under different visual semantic dimensions. The attribute text subsets of all categories are then combined to form an attribute text set. .

[0088] Text description encoding: The attribute-level text description is encoded using a text encoder frozen in the local model, mapping it to the original text feature set. :

[0089] (3)

[0090] in, Indicates the first A subset of the original text features corresponding to each category; It represents the union of sets.

[0091] The first The subset of attribute text corresponding to each category is denoted as ,in, Indicates the first The corresponding category is the Attribute-level text description, By using a frozen text encoder, the subset Mapped to a subset of original text features :

[0092] (4)

[0093] in, Indicates that the local number of the current client The corresponding category is the Attribute-level text description The original text feature vector obtained by the text encoder; Indicates the first The total number of attribute-level text descriptions corresponding to the class. Indicates the text embedding dimension.

[0094] The original text features are only encoded in this step and do not participate in the update, serving as the basis for subsequent visually guided corrections.

[0095] In this embodiment, the image encoder based on the local model acquires image features from the local image dataset and constructs a batch-level visual context representation to correct the original text features, including:

[0096] Visually Guided Text Attribute Correction: The client first extracts image features from the local image dataset of the current training batch using a frozen image encoder and computes a batch-level visual context representation. For a batch containing B images, the extracted batch-level image feature representation is as follows: .

[0097] By analyzing batch-level image features Take the average value to construct a batch-level visual context representation. :

[0098] (5)

[0099] in, Indicates the first in the current training batch Feature representation of each image.

[0100] Subsequently, the client inputs the original text feature set and the batch-level visual context representation into the gated text correction module of its local model. The gated text correction module dynamically adjusts the text features based on the visual context representation of the current training batch to achieve adaptive correction of the text features.

[0101] Specifically, the gated text correction module includes a correction subnetwork and a gated subnetwork:

[0102] The corrected subnetwork uses attribute-level text features. As input, a modified vector of text features is generated through nonlinear mapping. This is used to characterize the residual adjustment direction of text semantics in the feature space;

[0103] The gated subnetwork uses attribute-level text features Batch-level visual context representation The combination of these factors is used as input, and the output is the gating coefficient located in the interval [0,1]. ,in, This refers to the sigmoid function, which maps output values ​​to the interval [0,1]. Represents the gated network function, gating coefficients This is used to control the contribution ratio of the corresponding text correction vector in the final feature update.

[0104] Both the correction subnetwork and the gated subnetwork can be composed of multi-layer fully connected networks, and non-linear activation functions are introduced to realize feature mapping. In the module initialization stage, the correction subnetwork is configured to output a correction vector close to 0 in the initial state, so that the attribute-level text features remain consistent with the original text semantics in the early stage of training. Through this initialization constraint, excessive perturbation to the text semantic representation can be avoided in the early stage of model training, thereby improving the stability and convergence of the model training process.

[0105] Through the aforementioned gating mechanism, the gated text correction module can introduce dynamic corrections related to the current visual semantic distribution while maintaining the semantic stability of the original text. This improves the alignment between text features and image features, ultimately resulting in corrected text features. :

[0106] (6)

[0107] in, Indicates the current client's local number The corresponding category is the Attribute-level text description The original text feature vector obtained by the text encoder mapping, This represents the attribute-level text feature correction vector generated by the correction subnetwork. This represents the gating coefficients output by the gating subnetwork, used to characterize the correction strength of the corresponding attribute-level text in the current visual context.

[0108] Finally, the corrected text feature set for each of all categories is obtained as the corrected text feature set. Through the above mechanism, different clients can adaptively adjust the attribute-level text semantics based on their local visual distribution.

[0109] In this embodiment, a joint optimization objective is constructed based on the corrected text features to train the local model with updated parameters, specifically including:

[0110] Text-guided prompt injection and image feature generation: The client injects the modified set of text features as conditional information into the image feature modeling process. The prompt learning module guides the generation of image features, so that the image features are constrained and guided by the semantics of text attributes during the generation process, thereby obtaining an image feature representation that integrates text semantic information.

[0111] The above process is as follows: After obtaining the corrected text feature set... The client then uses this as conditional information and combines it with the input image. The injection prompt learning module is a lightweight core architecture based on Transformer, which mainly includes three components: linear projection layer, multi-head cross attention, and feedforward network. It achieves efficient fusion of visual and text features through a lightweight process of "projection-cross attention-FNN".

[0112] In simple terms, the prompting learning module uses image features as the query and text features as the key / value pair. Through a cross-modal attention fusion mechanism, it calculates cross-modal attention output to obtain enhanced image features guided by textual semantics. These enhanced image features are then concatenated with the original image token sequence to obtain the final text-conditionalized image feature representation. :

[0113] (7)

[0114] in, Indicates the input image. This represents the enhanced image feature generation function that incorporates the prompting learning module. This represents the corrected set of text features. This indicates the trainable parameters of the prompt learning module, from which the resulting image features not only contain raw visual information but also incorporate attribute-level textual semantic information.

[0115] Joint optimization objective construction: The client constructs a joint optimization objective based on the above features, including classification loss and visual consistency loss.

[0116] Classification loss The similarity between the final text-conditional image feature representation and the corrected text features of the same category is calculated, where the similarity is measured using normalized inner product or cosine similarity, and the text features of the same category are aggregated from multiple attribute descriptions belonging to the same category to obtain the category prediction probability. Thus, the classification loss is obtained. :

[0117] (8)

[0118] (9)

[0119] (10)

[0120] in, This represents the temperature parameter and adjusts the smoothness of the probability distribution. Represents text-conditional image feature representation; Indicates the first The corresponding category is the The text features of the attribute-level text description after being corrected by the gated text correction module. This represents an aggregation operation, used to combine information from multiple attribute perspectives. It is a text feature obtained by aggregating multiple attribute descriptions of the same category after correction. Indicates category label.

[0121] To suppress model drift under conditions of few samples and non-independent identically distributed (i.i.d.) images, this invention generates an enhanced view by randomly erasing the original image, thus constraining visual consistency. The visual consistency loss refers to the client generating an enhanced view from the input image. And extract corresponding features using a frozen image encoder. Then, by constraining the consistency of image features under different views, a visual consistency loss is constructed. :

[0122] (11)

[0123] The final joint loss function for the client is: ,in, These are the weighting coefficients.

[0124] The client updates the trainable parameters of the local model based on the above loss function, namely the parameters of the prompt learning module and the gated text correction module in the local model.

[0125] S3. Upload the parameters of the prompt learning module and the gated text correction module in the trained local model to the server for aggregation, in order to update the trainable parameters of the global model and obtain the target global model.

[0126] After the client completes local training of its local model, it uploads the updated trainable parameters to the server. These parameters, representing the prompting learning module and the gated text correction module in the trained local model, are as follows: ,in, Indicates the first The client in the first The set of trainable parameters for the cue learning module obtained after one round of local training. Indicates the first The client in the first The set of trainable parameters for the gated text correction module obtained after rounds of local training.

[0127] The server receives the updated trainable parameters of the local model uploaded by each client, and aggregates them to update the global model. Specifically:

[0128] The server collects the set of clients participating in this round of training. The uploaded parameters are then weighted or averaged to obtain new global model parameters. ,in, Indicates with the client The corresponding aggregate weights;

[0129] The updated global model parameters will be used as the initial parameters for the next round of communication and will be sent to each client again for the next round of updates. This update process will be repeated until the preset number of communication rounds or model convergence conditions are met. Finally, the server obtains a target global model for the image classification task. Using the global model based on the joint modeling results of the image and attribute-level text, the server performs category discrimination on the input image and outputs the corresponding image classification result.

[0130] Through the above steps, this embodiment achieves stable training of multimodal models in federated learning scenarios without sharing the original data, and effectively improves the model's generalization ability in heterogeneous data environments through attribute-level text modeling, visual guidance correction, and consistency constraints.

[0131] Experimental example

[0132] Regarding the experimental configuration and the experimental results of this invention, the following content is included:

[0133] A. Dataset and Model Setup

[0134] This invention was tested on the Flowers102 dataset, which contains approximately 8,000 images distributed across 102 classes with an imbalanced class distribution. The Flowers102 dataset is characterized by its fine-grained class structure and minimal visual differences, making it suitable for evaluating the classification performance of the proposed method in complex visual-semantic scenarios, particularly demonstrating its improved ability to align textual semantic modeling with visual features. The model used in this invention consists of a frozen multimodal base model backbone, a trainable cue learning module, and a trainable gated text correction module. The cue learning module employs a low-rank parameter structure, while the text correction module utilizes a multi-layer fully connected and residual structure.

[0135] B. Training Setup and Data Splitting

[0136] In the experiments, a federated learning training paradigm was adopted to optimize the model. The server randomly selected a subset of clients to participate in training during each round of global communication and updated the global model parameters after aggregation. The experiment set the number of global communication rounds to 100, with each client performing 3 local training epochs in each round. Mini-batch stochastic gradient descent (SGD) was used as the optimization method during the local training phase. During model training, the parameters of the pre-trained image-text coding backbone network were kept frozen, and only the cue learning module and the gated text correction module were updated, thereby reducing the overhead of federated communication and improving training stability. Regarding data partitioning, a non-independent and identically distributed (Non-IID) federated learning scenario was constructed based on the Flowers102 dataset. Image category sets were allocated to different clients by class partitioning, ensuring that each client contained only a subset of category samples, thus simulating the inconsistent data distribution among clients in a real federated environment. Simultaneously, a few-shot sampling strategy was used within each client to construct a local training set to verify the effectiveness of the proposed method under limited sample conditions.

[0137] C. Experimental Results

[0138] To verify the effectiveness of the gated text correction module and random erasure consistency strategy proposed in this invention in federated image classification tasks, comparative experiments were conducted with existing federated image classification methods. The existing federated classification method is a federated multimodal visual cue tuning method oriented towards a vision-language model, abbreviated as FedMVP, which does not introduce a gated text correction mechanism and adopts traditional data augmentation methods. The experimental results are as follows: Figure 3 , Figure 4As shown, both the gated text correction module and random erasure contribute to improving model performance. Furthermore, to further evaluate the generalization performance of the method of this invention, generalization performance tests were conducted, and the experimental results are as follows: Figure 5 As shown, this invention can improve the generalization performance of the model.

[0139] Example 2

[0140] This embodiment provides a federated image classification system based on visually guided attribute correction, including a server participating in federated learning and multiple clients communicating with the server. The clients participating in federated learning and the server use the federated image classification method described in Embodiment 1 to perform federated learning.

[0141] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A federated image classification method based on visual guidance attribute correction, characterized in that, The method comprises: S1, constructing a global model participating in federated learning on a server side, the global model being used for image classification and comprising an image encoder, a text encoder, a prompt learning module and a gated text revision module, and issuing parameters of the prompt learning module and the gated text revision module as trainable parameters to each client participating in federated learning; S2, constructing an initial local model consistent with the global model structure on the client, updating parameters of the prompt learning module and the gated text revision module in the initial local model based on the trainable parameters issued by the server side, and training the local model after parameter updating by using a local image data set of the client, comprising: obtaining original text features of the local image data set based on the text encoder of the local model; obtaining image features of the local image data set based on the image encoder of the local model, and constructing a batch-level visual context representation for revising the original text features, specifically as follows: First, obtaining image features of the local image data set based on the image encoder of the local model to construct a batch-level visual context representation, specifically as follows: （5） wherein, represents a batch-level visual context representation; represents the i-th image in the current training batch; represents a feature representation corresponding to the i-th image pair. Next, revising the original text features by using the batch-level visual context representation, specifically as follows: inputting the original text features and the batch-level visual context representation into the gated text revision module of the local model, the gated text revision module comprising a revision subnetwork and a gating subnetwork; The modified sub-network takes the original text features as input and generates a modified vector of the original text features through nonlinear mapping ; The gating sub-network takes as input the combination of the original text features and the batch-level visual context representation, and outputs gating coefficients located in the interval [0, 1] ; revising the original text features based on a revision vector of the original text features and a gating coefficient: （6） wherein, represents the first attribute-level text description corresponding to the current client local first category, the original text feature mapped by the text encoder, represents the original text feature corresponding to the correction vector, represents the gating coefficient, represents the original text feature corresponding to the corrected text feature; constructing a joint optimization target based on the revised text features for training the local model after parameter updating, specifically comprising: obtaining a text-conditioned image feature representation by using the prompt learning module of the local model based on the revised text features and original images in the local image data set of the client; constructing a classification loss based on the text-conditioned image feature representation and the revised text features; constructing a visual consistency loss based on the text-conditioned image feature representation and enhanced image features of the original images; jointly using the classification loss and the visual consistency loss for training the local model after parameter updating; S3, uploading parameters of the prompt learning module and the gated text revision module in the trained local model to the server side for aggregation, for updating trainable parameters of the global model to obtain a target global model.

2. The federated image classification method based on visual guidance attribute revision according to claim 1, wherein, In S1, the global model constructed on the server side is: （1） wherein, represents a frozen image encoder, represents a frozen text encoder, represents a trainable prompt learning module, represents a trainable gated text revision module; the initial trainable parameters are: （2） wherein, denote initial parameters of the prompt learning module, denote initial parameters of the gating text revision module.

3. The federated image classification method based on visual guidance attribute revision according to claim 1, wherein, In S2, obtaining original text features of the local image data set based on the text encoder of the local model comprises: obtaining an image category set from the local image data set of the client; constructing a plurality of attribute-level text descriptions corresponding to each category in the image category set; encoding the constructed attribute-level text descriptions by using the text encoder of the local model to obtain original text features of the local image data set.

4. The federated image classification method based on visual guidance attribute revision according to claim 3, characterized in that, Encoding the constructed attribute-level text descriptions by using the text encoder of the local model, specifically as follows: （3） wherein, represents a set of original text features corresponding to the attribute-level text description of all classes in the local image data set; represents a subset of original text features corresponding to the th class; represents a set of image classes; represents a union set; and （4） wherein, represents the current client local first category corresponding to the first attribute level text description mapped by the text encoder to obtain the original text features represents the total number of attribute level text descriptions corresponding to the first category, represents the text embedding dimension.

5. The federated image classification method based on visual guidance attribute revision according to claim 4, characterized in that, The classification loss is obtained by calculating the similarity between the text-conditioned image feature representation and the text feature of the same category after correction, wherein the text feature of the same category is aggregated by multiple attribute descriptions belonging to the same category to obtain a category prediction probability , and further obtaining the classification loss : （7） （8） （9） wherein, denotes a temperature parameter, adjusting the probability distribution smoothness; denotes a text-conditioned image feature representation; denotes an aggregation operation; is a text feature aggregated from multiple attribute descriptions of the same category after revision, denotes a category label.

6. The federated image classification method based on visual guidance attribute revision according to claim 4, characterized in that, the visual consistency loss is: （10） wherein, represents a text-conditioned image feature representation; represents an image feature of the enhanced original image.

7. A federated image classification system based on visual guidance attribute correction, characterized in that, The server end participating in federated learning and a plurality of client ends connected in communication with the server end, the client end participating in federated learning and the server end adopting the federated image classification method in any one of claims 1 to 6 to perform federated learning.

Images