A federated large model visual-language prompt tuning method
By introducing sample-level dynamic fusion of attribute-driven branches and stable anchor branches in federated learning, the problems of heterogeneous client data distribution and attribute text noise in federated learning are solved, improving the model's prediction stability and generalization ability under heterogeneous data conditions, while reducing communication and computational overhead.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGXI NORMAL UNIV
- Filing Date
- 2026-02-12
- Publication Date
- 2026-05-29
Smart Images
Figure CN122114082A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of federated learning, visual-language pre-trained models, and efficient parameter fine-tuning, specifically a method for optimizing visual-language cues in federated large models. Background Technology
[0002] In recent years, visual-language pre-trained models, represented by CLIP, have achieved good general visual representation capabilities through large-scale image-text contrastive learning, demonstrating strong transfer performance in zero-shot or few-shot scenarios. To adapt to specific downstream tasks or application scenarios, a common approach is to introduce efficient parameter fine-tuning methods such as prompt learning and low-rank adaptation (LoRA) while freezing or partially freezing the parameters of the pre-trained model, thereby reducing training and communication overhead.
[0003] In federated learning scenarios, multiple clients collaboratively train models without sharing raw data, satisfying privacy and compliance requirements. However, due to significant differences in acquisition devices, data sources, shooting environments, and category distributions among clients, client data typically exhibits non-independent identically distributed (Non-IID) characteristics. Existing efficient fine-tuning methods for federated parameters face the following problems in this scenario: Firstly, the update direction of lightweight parameters may deviate significantly across different clients, easily leading to model drift or convergence instability during federated aggregation; secondly, single-path cue learning lacks reliability assessment and backoff mechanisms under domain shift or difficult sample conditions, easily resulting in inflated confidence or error propagation.
[0004] To enhance the model's ability to perceive fine-grained differences, some technical solutions introduce attribute text or external knowledge to enhance visual features. However, in practical applications, attribute text often comes from diverse sources and may contain redundant, irrelevant, or even erroneous descriptions. Furthermore, the coverage of categories in attribute libraries varies across different clients or application domains, easily leading to missing category attributes. Under federated learning conditions, such attribute noise or omissions further amplify the differences between clients, causing overall instability in attribute-driven suggestion branches and thus affecting the federated aggregation effect.
[0005] Existing techniques typically filter attributes through gating or weighting, but most of these methods still rely on a single prediction path, lacking a stable reference branch unaffected by attribute noise, and also lacking a mechanism for adaptively fusing different branches based on sample-level prediction reliability. When the attribute branches as a whole are unreliable, relying solely on attribute filtering is insufficient to avoid the interference of erroneous predictions on model updates, especially in federated training where this interference can be amplified through multiple rounds of aggregation. Summary of the Invention
[0006] The purpose of this invention is to address the problems of heterogeneous client-side data distribution (Non-IID), noisy or missing attribute text, and unstable predictions and inflated confidence levels in single-branch cue learning under domain-biased or difficult sample conditions in federated learning environments. This invention proposes a visual-language cue tuning method for federated large models. This method, while freezing the parameters of the visual encoder and text encoder, trains only on the client side and aggregates relevant lightweight cue parameters on the server side. Through sample-level dynamic fusion of attribute-driven branches and stable anchor branches, it enhances attribute information when reliable and regresses when unreliable, thereby improving training stability and generalization ability in federated heterogeneous (Non-IID) and domain-biased scenarios.
[0007] The technical solution to achieve the objective of this invention is:
[0008] A method for optimizing a federated large-scale model using visual-language prompts is proposed. This method is executed within a federated learning system, which includes a server and multiple clients. The server-side architecture includes a visual encoder. With text encoder The method describes a vision-language pre-trained model; it performs efficient parameter tuning and communication training of the vision-language pre-trained model under a federated learning system, and the visual encoder... With text encoder The original model parameters remain frozen during training; the client only updates the lightweight set of trainable parameters used for parameter tuning communication training. The lightweight trainable parameter set The method includes at least one of prompt generation parameters and prompt injection parameters, and further includes one or more of attribute gating parameters, class bias parameters, sample-level fusion network parameters, and low-rank adaptation parameters (LoRA). The method comprises the following steps:
[0009] S3.1 Attribute Management: This involves processing input image samples... Attribute routing is performed based on dataset identifiers or application domain identifiers to complete category name normalization and alignment and missing category validation; the attribute text of each category is then input into the frozen text encoder. Get attribute text embedding The dataset identifier or application domain identifier is obtained from a client-side pre-configured configuration file, data acquisition device marker, business system-issued marker, or sample metadata field; the server or client obtains a set of candidate attribute texts by routing from a pre-configured attribute library / attribute template set based on the identifier.
[0010] S3.2 Attribute-driven branch: Input image samples Input the frozen visual encoder to obtain the basic image visual features The visual features of the base image are enhanced by the cue generator. With attribute text embedding Interactive generation of multimodal visual cues ,in Embed the cue vector sequence / cue token output by the cue generator; embed the cue... Injecting a visual encoder to obtain attribute-enhanced image features ; and through a gating network for the same category Assigning sample conditional weights to multiple attributes Output attribute branch prediction ;in, Indicates the input sample Under the conditions Class 1 The contribution weight of each attribute text to the sample is determined by the gating network based on the gating visual features. Embedded with corresponding attribute text The correlation / similarity calculation yielded And normalize the weights within the same category; Apply Top-k or Top-p sparsity before normalization; where, This represents the input visual features of the gating network, and is selected according to the dataset / domain configuration. or ,in To avoid injecting the aforementioned prompt Input image samples Input the clean image features obtained from the frozen visual encoder, and ;
[0011] S3.3 Stable Anchor Branch: Input image samples Without injecting the aforementioned prompt Under the condition of inputting a frozen visual encoder, the output of the clean image features is obtained. ; and by calculating the Embedded with category name template text The similarity score between them yields stable anchor branch predictions. The category name template text is embedded therein. Obtained by the frozen text encoder, specifically , To fill the category name into the preset text template obtained by the category name, the preset text template is either pre-set and distributed by the server or pre-set locally by the client; the similarity score is the inner product or cosine similarity, and the similarity scores of each category are combined into a vector as a stable anchor point for branch prediction. The stable anchor branch does not use attribute text and does not calculate weights through a gating network; when at least one of the cue generator or visual encoder contains low-rank adaptation parameters, the low-rank adaptation path associated with attribute branch cue injection is bypassed or disabled in the stable anchor branch to maintain prediction stability.
[0012] S3.4 Sample-level Dynamic Fusion: Generating Sample-level Basic Fusion Coefficients Based on the Uncertainty Difference Between Attribute Branch and Stable Anchor Point Branch Predictions At least one of entropy-gated suppression and divergence suppression is introduced to form the final fusion coefficient. and to Pruning restricts the values to [0,1]; the outputs of the attribute branch and the stable anchor branch are merged to obtain the final output:
[0013] ;
[0014] S3.5 Local Optimization: Based on the final output Construct the loss function and update the lightweight trainable parameter set ;
[0015] S3.6 Upload and Aggregation: Client uploads updated lightweight parameters On the server side, the global parameters are updated by weighted aggregation based on the local sample size of each client. Repeat steps S2–S3.6 until the preset number of communication rounds or convergence conditions are reached.
[0016] In step S3.1, attribute management involves processing the input image samples... Based on the dataset identifier or application domain identifier, the corresponding attribute template or attribute file is selected for routing. Normalized mapping alignment is performed on the category names, and missing categories in the attribute file are verified. When a missing category is detected, at least one of the following is executed: fall back to the default attribute template, execute the missing category filling strategy, execute the dynamic masking strategy, or pull the general global attribute template from the server for filling, so as to ensure that training can continue. The dynamic masking strategy includes: resetting the weights corresponding to missing categories to a preset minimum value or setting them to zero when calculating the gating weights, and renormalizing them within the same category.
[0017] In step S3.2, the prompt generator is PromptFormer, which uses a cross-attention mechanism to process the input image samples. Image visual features and attribute text embedding interactive generation of multimodal prompts and the prompt Inject at least one layer of the visual encoder to obtain attribute-enhanced image features. .
[0018] In step S3.2, the gating network for each category Attribute text collection Calculate the sample conditional weights and for the same category Normalization;
[0019] ,
[0020]
[0021] in All are learnable linear mapping matrices. Based on the base temperature, For learnable temperature scaling parameters, and for Apply upper and lower bound pruning to ensure numerical stability; where Select by dataset / domain configuration or ; and on Apply Top-k or Top-p sparsity and then normalize; and obtain the sample adaptive attribute prototype based on the gated weights:
[0022]
[0023] make If the logit scaling factor is the visual-language model, then the attribute branch output is:
[0024]
[0025] in For category Scalar bias; and derived from the attribute mean prototype:
[0026]
[0027] Generated by a multilayer perceptron:
[0028] ,
[0029] The bias It can be enabled or disabled by a configuration switch; when disabled, it commands... Or, the bias term may not be omitted in the calculation, thus
[0030]
[0031] To enhance stability.
[0032] In step S3.3, the stable anchor branch uses the clean image features. Embedded with category name template text calculate:
[0033]
[0034] in The category name template text is obtained by filling the category name into the preset text template; and if the visual encoder contains LoRA, the stable anchor branch bypasses or disables LoRA to maintain anchor prediction stability.
[0035] In step S3.4, sample-level dynamic fusion, the basic fusion coefficients... It is generated at least by the uncertainty difference between the attribute branch and the stable anchor branch prediction distribution, and the uncertainty difference includes at least the normalized entropy difference and the confidence boundary difference; Let:
[0036]
[0037]
[0038] in For the number of categories, and These are the predicted distributions. The probability values of Top-1 and Top-2 are obtained; and the fusion network then... Mapping The entropy-gated suppression term is:
[0039]
[0040] in The scaling factor is used; the divergence suppression term can be:
[0041]
[0042] in The inhibition coefficient and the divergence measure This includes any one or any combination of Jensen–Shannon divergence, KL divergence, symmetric KL divergence, predictive distribution cosine divergence, or Top-1 class consistency indicator function; the final fusion coefficient is:
[0043]
[0044] The loss function constructed in step S3.5 includes classification cross-entropy loss and consistency regularization term, as well as federated proximal regularization term:
[0045]
[0046] in For consistency regularization weights, The consistency regularization term is the weight of the near-end item. For the client to perform local training on input image samples The generated enhanced samples Distance or similarity constraints between the enhanced sample and the original sample in the visual encoder output feature space, wherein the enhanced sample The data is augmented locally on the client side, including at least horizontal flipping; and the global parameters are updated on the server side using weighted aggregation based on the number of samples.
[0047]
[0048] in For the client The local sample size; the client also employs at least one of two-stage training scheduling and preheating / freezing strategies to improve training stability; wherein, the two-stage training scheduling includes: in the first stage, during the preheating round, only updating at least one of the prompt generation parameters and prompt injection parameters while keeping the remaining optional parameters frozen; in the second stage, after the first stage ends, unfreezing a portion of the optional parameters and performing a joint update; the preheating / freezing strategy includes preheating the learning rate and keeping some module parameters frozen within a preset number of communication rounds.
[0049] The purpose of this technical solution is as follows:
[0050] (1) By introducing stable anchor branches that do not depend on attribute text and gated computation, a robust baseline path is provided for model prediction, suppressing attribute noise or indicating overfitting interference with the prediction results;
[0051] (2) By using attribute governance, attribute gating and sample-level dynamic fusion mechanism, attribute information can enhance model expression when reliable and automatically back off when unreliable, thereby improving the stability and generalization ability of federated training under cross-domain and heterogeneous data conditions.
[0052] (3) By training only on the client side and aggregating relevant lightweight parameters on the server side, communication and computing overhead are reduced while protecting data privacy.
[0053] Compared with the existing technology, the present technical solution has the following beneficial effects:
[0054] (1) Improve prediction stability under attribute noise and missing conditions:
[0055] This technical solution introduces a stable anchor branch that does not rely on attribute text and gated computation, which can still provide reliable baseline predictions even when the attribute text contains noise, errors, or missing data. Furthermore, through a sample-level dynamic fusion mechanism, the weight of the attribute-driven branch is automatically reduced when the uncertainty of the attribute-driven branch increases or the divergence with the anchor branch increases, thereby suppressing the misleading effect of noisy attributes on the prediction results.
[0056] (2) Enhance generalization ability under cross-domain and heterogeneous data conditions:
[0057] By using sample-level fusion based on prediction uncertainty difference and divergence metric, this technical solution can adaptively adjust the contribution ratio of the two branches according to the difficulty and domain offset of different samples, making the model more inclined to stabilize the anchor path under cross-domain or difficult sample conditions, and making full use of the fine-grained discrimination capability brought by attribute enhancement under typical sample conditions.
[0058] (3) Reduce the communication and computing overhead of federated training:
[0059] This technical solution freezes the visual encoder and text encoder of the vision-language pre-trained model, trains only on the client side and aggregates lightweight parameters such as prompt generation, gating, and fusion on the server side, significantly reducing the amount of parameters to be transmitted in each round of communication, and is suitable for federated learning scenarios with limited bandwidth or multi-terminal collaboration.
[0060] (4) Improve the convergence stability of the federated training process:
[0061] By employing stable anchor branching, sample-level fusion suppression mechanisms, and optional training scheduling and parameter freezing strategies, the cumulative propagation risk of unstable updates caused by Non-IID data distribution and attribute noise during the federated aggregation process is reduced. Attached Figure Description
[0062] Figure 1 This is an architecture diagram of the system in the embodiment;
[0063] Figure 2 This is a flowchart of the method in the embodiment;
[0064] Figure 3 This is a schematic diagram of the dynamic fusion structure in the embodiment;
[0065] Figure 4 This is a schematic diagram illustrating the phased training scheduling and preheating / freezing strategy adopted by the client in this embodiment. Detailed Implementation
[0066] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.
[0067] Example:
[0068] Reference Figure 1 , Figure 2 This paper presents a federated large-scale model visual-language prompting optimization method, applied to a federated learning system comprising a server and multiple clients. This method is particularly suitable for smart home device management systems, where each client represents a smart device (such as a smart camera, smart light bulb, smart thermostat, etc.), and the server is responsible for global model aggregation and optimization. Image data captured by smart home devices in the home environment (such as facial images of family members, activities within the room, etc.) is processed together with text information (such as device status, commands, etc.) to achieve device recognition, control, and status inference.
[0069] The server-side constructs a vision-language pre-trained model, which includes a frozen visual encoder. With frozen text encoder Visual encoder With frozen text encoder The original model parameters remain frozen during training; in a federated learning system, the server communicates with multiple clients for efficient parameter tuning. Clients only update and upload a lightweight set of trainable parameters to provide tuning hints. The lightweight trainable parameter set It includes at least one of prompt generation parameters and prompt injection parameters, and may also optionally include one or more of attribute gating parameters, class bias parameters, sample-level fusion network parameters, and low-rank adaptation parameters (LoRA).
[0070] Specific steps in smart home applications:
[0071] S1 Global Initialization: Server-side initialization of a lightweight set of trainable parameters used for parameter tuning training. To obtain the initial global parameters ;
[0072] S2 Client Sampling and Distribution: In the first In round-robin communication, the server samples the participating set from the set of available clients that meet the preset participation conditions. The available client set is a set of clients that meet preset participation conditions, which include one or a combination of the following: online status meets a threshold, communication link quality meets a threshold, computing resources meet a threshold, and local sample size meets a minimum threshold; the server sends a message to each client. Distribute current global lightweight parameters ;
[0073] S3 client local training: The client maintains... and With the original model parameters frozen, only the lightweight trainable parameter set is updated. and the input image samples Perform the following operations:
[0074] S3.1 (Attribute Governance):
[0075] Input image sample Attribute routing is performed based on dataset identifiers or application domain identifiers (e.g., living room, bedroom, kitchen, etc.) to complete category name normalization and alignment and missing category verification. In smart homes, image data may contain categories such as "family member," "guest," or "pet." The system needs to ensure the standardization of each category name through attribute governance and to fall back when missing categories exist in the attribute library. Attribute text is input into a frozen text encoder. Get attribute text embedding .
[0076] S3.2 (Attribute-Driven Branch)
[0077] Input image sample Input frozen visual encoder Obtain basic image visual features Then, the visual features of the base image are made possible by the cue generator. With attribute text embedding Interactive generation of multimodal prompts And will prompt Injecting a visual encoder to obtain attribute-enhanced image features For example, a smart camera can enhance image features based on attributes of the "family member" category, thereby identifying whether a face belongs to a family member. This is achieved by assigning conditional weights to multiple attributes within the same category using a gating network. Calculate the attribute branch prediction Furthermore, it can infer the control commands of the device, such as: if a family member is in the room, the system may instruct to automatically adjust the temperature or turn on the lights.
[0078] Among them, the gated network uses gated visual features With attribute text embedding The correlation / similarity calculation yielded The weights are normalized within the same category; Retrieve by dataset / domain configuration or ,in To avoid injecting prompts The input image is frozen by the visual encoder. The output clean image features, and .
[0079] ;
[0080] S3.3 (Stable Anchor Branch)
[0081] In stable anchor point branches, do not splice hints. Instead, it only uses the original input image samples. The corresponding image patch token serves as the visual encoder. The input, get For example, in smart home scenarios, stable anchor branch can be used for basic identification of categories such as "family member / guest / pet". When attribute branches are unreliable, it provides a stable baseline prediction, such as detecting whether someone is a family member, and outputs a stable anchor branch prediction. This branch does not rely on attribute text, therefore it does not calculate sample conditional weights through a gating network, but directly based on... Embedded with category name template text A prediction is obtained. If at least one of the cue generator or visual encoder contains a low-rank adaptation parameter (LoRA), the low-rank adaptation path related to the attribute branch cue injection is bypassed (i.e., the path is disabled) in the stable anchor branch to maintain prediction stability.
[0082] ;
[0083] S3.4 (Sample-level Dynamic Fusion), such as Figure 3 As shown,
[0084] Generate sample-level basic fusion coefficients based on the uncertainty difference between the two branches of prediction: attribute and stable anchor. It also introduces at least one of entropy gating suppression and divergence suppression to form the final fusion coefficient. ,right The cropping process restricts the values to [0,1]. For example, in smart home scenarios, if the image recognition result has high uncertainty (e.g., a "blurry face image" or a "dark environment"), the stable anchor branch will be preferentially relied upon to ensure prediction stability. The outputs of the attribute and stable anchor branches are fused to obtain the final output:
[0085] ;
[0086] S3.5 (Local Optimization)
[0087] According to the final output Construct the loss function and update the lightweight trainable parameter set. ;
[0088] .
[0089] in, For consistency regularization weights, For the weight of the proximal terms, This represents the visual consistency loss term.
[0090] The consistency regularization term For the client to perform local training on input image samples The generated enhanced samples Distance or similarity constraints between the enhanced sample and the original sample in the visual encoder output feature space, wherein the enhanced sample The data is augmented locally on the client side, including at least horizontal flipping; and the global parameters are updated on the server side using weighted aggregation based on the number of samples.
[0091]
[0092] in For the client The local sample size; the client also employs at least one of the following strategies: two-stage training scheduling and preheating / freezing. Figure 4 As shown, to improve training stability, the two-stage training scheduling includes: in the first stage, during the warm-up rounds, only at least one of the prompt generation parameters and prompt injection parameters is updated while the remaining optional parameters are frozen; in the second stage, after the first stage ends, a portion of the optional parameters are unfrozen and a joint update is performed; the warm-up and freezing strategy includes warming up the learning rate and keeping some module parameters frozen within a preset number of communication rounds.
[0093] S3.6 (Upload and Aggregation)
[0094] Upload updated lightweight parameters from the client On the server side, the global parameters are updated by weighted aggregation based on the local sample size of each client. Repeat steps S2–S3.6 until the preset number of communication rounds or convergence conditions are reached.
[0095] Through application examples in the smart home field, and combined with the technical framework of federated learning, this paper details how to optimize device identification, state reasoning, and control policy generation in a device management system by using image recognition and text prompts. This embodiment demonstrates how to share lightweight training parameters among devices and improve the system's generalization ability and stability in complex home environments through a sample-level dynamic fusion mechanism of attribute-driven branches and stable anchor branches.
[0096] Supplementary explanation of formula symbols:
[0097] Sample conditional weights are calculated using a gating network, based on the correlation between the visual encoder and attribute text embedding.
[0098] Stable anchor branch prediction calculates the similarity between image features and category template text.
[0099] The final output obtained by fusing predictions from the attribute and stable anchor branches determines the device control strategy.
[0100] Experimental and technical effect description
[0101] (I) Experimental Objective
[0102] To verify the generalization ability and prediction stability of the proposed federated multimodal visual cue tuning method under cross-domain distribution shift and federated heterogeneous data (Non-IID) conditions, this embodiment adopts the multi-source single-objective cross-domain generalization (DG) evaluation protocol, performs validation on multiple public cross-domain classification benchmark datasets, and compares and analyzes it with existing technologies in the context of federated heterogeneous (client constructed based on source domain data) scenarios.
[0103] The experiment focuses on examining the following technical effects:
[0104] 1) The stability and generalization ability of the model's prediction performance in target domains with significant domain shifts;
[0105] 2) The effectiveness of federated training under the premise of freezing the backbone parameters of the visual-language model and transmitting only lightweight cue parameters;
[0106] 3) The role of dual-branch structure and sample-level dynamic fusion mechanism in improving the performance of difficult samples and strong domain offset scenarios.
[0107] (II) Explanation of Experimental Setup and Fairness
[0108] 1) Federal Training Process
[0109] Multiple clients train using their own local data without sharing the original image data. Each client only uploads lightweight parameters related to prompt generation, attribute gating, fusion network, and optional low-rank adaptation to the server. The server then performs sample-weighted aggregation on the uploaded parameters to form global parameters, which are then sent to the clients for the next round of training.
[0110] 2) Freezing strategy
[0111] In the experiment, the original parameters of the visual encoder and text encoder of the vision-language pre-trained model were frozen, and only the lightweight parameters of the cue tuning module were trained and aggregated to ensure communication efficiency and privacy protection.
[0112] 3) Evaluation Indicators
[0113] The classification accuracy (%) on the target domain test set was used as the main evaluation metric.
[0114] 4) Fairness explanation
[0115] The control scheme and the example scheme use the same pre-trained backbone model, the same data preprocessing and augmentation strategies, the same cross-domain training / test partitioning protocol, and the same evaluation metrics to ensure the reproducibility and comparability of the experimental results.
[0116] 5) Key training and communication configurations (example implementation parameters)
[0117] The following training and communication configuration parameters are used in the example implementation (the following parameters are for example implementation and are used for reproduction and illustration, and do not constitute a limitation on the scope of protection of this invention):
[0118] (1) Number of communication rounds (Global Rounds): OfficeHome (Multi-source Single Objective) 120 rounds; VLCS (Multi-source Single Objective) 100 rounds; TerraIncognita (Multi-source Single Objective) 100 rounds.
[0119] (2) Client participation rate: 100% (all source domain clients participate in training and parameter uploading in each round).
[0120] (3) Local training rounds: Each client trains locally for 1 epoch per round.
[0121] (4) Local batch size: 128.
[0122] (5) few-shot sample size (shots): 8.
[0123] (6) Learning rate / optimizer: learning rate 0.003, SGD + momentum 0.9, cosine learning rate scheduling.
[0124] (7) Prompt token number :4 (PromptFormer multimodal hint length).
[0125] (8) Class division / client granularity: OfficeHome / VLCS: num_cls_per_client=20; TerraIncognita: num_cls_per_client=10.
[0126] (9) Attribute selection (example default): Top-p=0.9, ATTR_MIN_K=1, ATTR_MAX_K=8.
[0127] (10) Fusion and safety valve (consistent with the "correct baseline" run): Enable sample-level fusion gating (e.g., enable DG_LEARNED_GATE), and disable the safety valve mechanism (e.g., FUSION_SAFETY_ENABLE=0). Learned from the gating network; in the statistical analysis of the experimental results in this embodiment, the branching / distance suppression term related to the safety valve is not enabled, which is equivalent to... (or let) Therefore, no additional suppression term based on JS distance / divergence is introduced. Optionally, the safety valve mechanism can be configured as an optional feature if it is not enabled (e.g., (11) LoRA configuration (example implementation): rank=8; the insertion point is the q / k / v low-rank bypass of PromptFormer cross-attention; optionally, LoRA can also be deployed in the q / k / v bypass of the visual encoder attention layer; bypass or disable the LoRA path related to attribute branch cue injection in the stable anchor branch to maintain the stability of anchor prediction.
[0128] (12) Aggregation strategy: The server uses FedAvg to aggregate based on the number of client samples.
[0129] (III) Dataset and Task Division
[0130] This embodiment selects a cross-domain classification benchmark dataset covering different domain offset types for validation:
[0131] 1) OfficeHome dataset: Contains four domains: Product, RealWorld, Clipart, and Art;
[0132] 2) VLCS dataset: contains four domains: Caltech, LabelMe, Pascal, and SUN;
[0133] 3) TerraIncognita dataset: contains four domains: L100, L38, L43, and L46.
[0134] A cross-domain generalization protocol of "multi-source domain training and single target domain testing" is adopted: each experiment selects a target domain, and the target domain test set is used only in the testing phase to evaluate the performance; in the training phase, the target domain is removed, and only the remaining domains are used as source domain data to participate in federated training.
[0135] (iv) Explanation of the comparison plan
[0136] The control scheme is constructed based on multimodal or cue-based optimization methods using federated learning in publicly available literature, employing the same visual-language pre-training backbone and federated training process. Since engineering implementation details (such as training scheduling, module configuration, or hyperparameter values) may differ, this example is described as a "pre-existing technology control scheme" for repeatable comparison and verification, and does not constitute a limitation on the implementation details of publicly available literature.
[0137] (V) Experimental Results
[0138] Table 1 shows the comparison results (Accuracy, %) of the classification accuracy of the existing technology implementation and the proposed solution on three types of cross-domain datasets.
[0139]
[0140] Note: The average is the arithmetic mean of the accuracy of all target domains in this dataset; Improvement Δ = (Solution in this example) - (Existing technology comparison implementation).
[0141] (vi) Supplementary verification: Attribute noise stress test and ablation experiment settings (DG caliber)
[0142] 1) Attribute noise stress test settings
[0143] To verify the fallback stability of this example under conditions where the attribute text contains noise or is missing, an attribute noise stress test is constructed:
[0144] Noise injection method: Randomly replace the proportion of attribute phrases .
[0145] Evaluation sets: OfficeHome, VLCS, TerraIncognita test sets for each target domain; training configuration is consistent with the example implementation parameters in "(II) Experimental Setup and Fairness Description".
[0146] Record metrics: Top-1 accuracy for attr-only, class-only, and fused metrics; Mean and quantiles (e.g.) ); safety valve trigger rate (e.g.) (Suppression trigger ratio, entropy gating suppression trigger ratio).
[0147] Expected verification point: with the noise ratio The performance degradation of the fused output is less than that of attribute-only branches, and... The mean value decreases as noise increases, while the safety valve trigger rate increases, thus proving the effectiveness of "disagreement / uncertainty driven backoff".
[0148] 2) Ablation experiment setup
[0149] To verify the effect attribution of the distinguishing features in this example, the following ablation comparison terms were set:
[0150] Remove stable anchor branches: retain only attribute-driven branches for prediction and training;
[0151] De-divergence suppression: Disable the suppression term based on the divergence metric (e.g., do not enable it). Or not enabled );
[0152] Deentropy gating: Turn off the entropy gating suppression term (e.g., set the entropy gating scaling factor s=0);
[0153] Attribute removal governance: Excludes missing category validation / fault tolerance, attribute deduplication / diversity filtering and fixing. Constraints, etc.
[0154] Record metrics: Accuracy and average value for each target domain across the three datasets; also record attr-only, class-only, and fused metrics. Quantile statistics are used to form an attribution chain of "effect – cause – module correspondence".
[0155] (vii) Explanation of technical effects and beneficial effects
[0156] As shown in Table 1, under the same experimental conditions, the proposed scheme achieved better or more stable classification performance on multiple cross-domain generalization tasks.
[0157] On the TerraIncognita dataset, the average accuracy of this approach improved from 40.118% to 46.808%, an average improvement of approximately 6.690 percentage points. In particular, the accuracy improved by approximately 12.598 and 8.202 percentage points respectively in the L38 and L46 target domains with large domain offsets, indicating that this approach has a significant generalization advantage under conditions of strong domain offsets and federated heterogeneous data.
[0158] On the VLCS dataset, the average accuracy of this solution improved from 80.583% to 81.805%, with an improvement of approximately 2.991 percentage points in the LabelMe domain, demonstrating improved stability under conditions of different data distributions across scenarios.
[0159] In the OfficeHome dataset, there is a slight decrease in the Product and RealWorld target domains, but an improvement is achieved in the Clipet and Art target domains, keeping the overall average stable and slightly improved. The improvement is approximately 1.013 and 1.232 percentage points in the Clipet and Art target domains, which have significant style differences, respectively. This further illustrates that this example can suppress the performance fluctuations caused by attribute noise and unstable prompts in complex style change scenarios.
[0160] The results above demonstrate that, through the stable anchor branch and sample-level dynamic fusion mechanism, this example can automatically reduce the weight of the attribute branch and fall back to the stable prediction path when the uncertainty of the attribute-driven branch increases or the divergence with the anchor branch increases, thereby effectively improving the stability and generalization ability of the model prediction in the federated training environment.
[0161] (viii) Communication parameters and client division
[0162] To ensure consistency between the cross-domain generalization protocol and the federated training process at the implementation level, this embodiment adopts a multi-source, single-target (MPS) configuration: each experiment selects one target domain for testing only, without participating in federated training and aggregation; the remaining domains serve as source domains for federated training. Thus, the target domain does not generate clients; clients are constructed solely from source domain data, forming a federated heterogeneous training scenario without altering the original domain partitioning of the dataset.
[0163] 1) Client segmentation method
[0164] In this embodiment, clients are constructed for each source domain based on a "category set": For a given source domain dataset, the available category list for that domain is first obtained, and then the dataset is evenly split according to the number of categories per client, num_cls_per_client. Each client holds a subset of categories for that domain. By default, a random and balanced category splitting strategy is enabled (i.e., the category list is first shuffled, and then distributed as evenly as possible among the clients) to avoid extreme imbalances in category distribution among clients.
[0165] Therefore, for the number of categories is The source domain typically has the following number of clients: .
[0166] Taking a commonly used dataset as an example, under the "3 sources 1 target" setting:
[0167] OfficeHome: domain collection During training, the target domain is removed, and the remaining three source domains are split into clients according to their categories. Since each source domain has approximately 65 classes and num_cls_per_client=20, each source domain is typically split into approximately 3 to 4 clients.
[0168] VLCS: Domain Collection During training, the target domain is removed, and clients are generated from the remaining three source domains. Since each source domain has a small number of categories and num_cls_per_client=20, each source domain typically corresponds to one client (no further splitting).
[0169] TerraIncognita: Set of Domains During training, the target domain is removed, and clients are generated from the remaining three source domains; when num_cls_per_client=10, typically one client corresponds to each source domain.
[0170] 2) Client engagement rate
[0171] In each round of communication, the client participation rate is set to 100%, meaning that all clients generated by the source domain in the current experiment participate in the local training and parameter upload in this round. If it is necessary to simulate actual disconnection or online fluctuations, the participating clients in each round can be changed to a subset randomly sampled from the source domain client set. However, this embodiment maintains full participation by default to ensure the reproducibility of the experiment and the fairness of the comparison.
[0172] 3) Aggregation Strategy
[0173] The server-side uses standard FedAvg aggregation to fine-tune the lightweight parameters uploaded by participating clients: a weighted average is calculated based on the number of local samples on each client to obtain the global lightweight parameters, which are then distributed to each client for the next round of training. This strategy achieves cross-client knowledge aggregation without uploading the original data and allows clients with larger sample sizes to contribute more to the global update.
[0174] 4) Example implementation of the hyper-participation training switch (consistent with the running method)
[0175] OfficeHome: num_cls_per_client=20, num_epoch=120, batch_size=128, num_shots=8;
[0176] VLCS: num_cls_per_client=20, num_epoch=100, batch_size=128, num_shots=8;
[0177] TerraIncognita: num_cls_per_client=10, num_epoch=100, batch_size=128, num_shots=8.
[0178] In addition, sample-level fusion coefficient learning is enabled in this round of experiments (e.g., enabling --dg_learned_gate to learn sample-level fusion coefficients). And disable fusion safety valve suppression (e.g., set --fusion_safety_enable=0) to avoid early training when the system has not yet converged. This generates excessive inhibition, thereby maintaining a stable and controllable training process.
Claims
1. A method for optimizing visual-language cues in a federated large model, characterized in that, The method is executed in a federated learning system, which includes a server and multiple clients. The server-side architecture includes a visual encoder. With text encoder The method describes a vision-language pre-trained model, in which the parameters of the pre-trained model are tuned and trained via communication within a federated learning system, and the visual encoder is... With text encoder The original model parameters remain frozen during training; the client only updates the lightweight set of trainable parameters used for parameter tuning communication training. The lightweight trainable parameter set The method includes at least one of prompt generation parameters and prompt injection parameters, and further includes one or more of attribute gating parameters, class bias parameters, sample-level fusion network parameters, and low-rank adaptation parameters (LoRA). The method comprises the following steps: S1 Global Initialization: Server-side initialization of a lightweight set of trainable parameters used for parameter tuning training. To obtain the initial global parameters ; S2 Client Sampling and Distribution: In the... In round-robin communication, the server samples the participating set from the set of available clients. The available client set is a set of clients that meet preset participation conditions, which include one or a combination of the following: online status meets a threshold, communication link quality meets a threshold, computing resources meet a threshold, and local sample size meets a minimum threshold; the server sends a message to each client. Distribute current global lightweight parameters ; S3 client local training: The client maintains... and With the original model parameters frozen, only the lightweight trainable parameter set is updated. and the input image samples Perform the following operations: S3.1 Attribute Management: This involves processing input image samples... Attribute routing is performed based on dataset identifiers or application domain identifiers to complete category name normalization alignment and missing category verification. Input the attribute text of each category into the frozen text encoder. Get attribute text embedding The dataset identifier or application domain identifier is obtained from a client-side pre-configured configuration file, data acquisition device marker, business system-issued marker, or sample metadata field; the server or client obtains a set of candidate attribute texts by routing from a pre-configured attribute library / attribute template set based on the identifier. S3.2 Attribute-driven branch: Input image samples Input the frozen visual encoder to obtain the basic image visual features The visual features of the base image are enhanced by the cue generator. With attribute text embedding Interactive generation of multimodal visual cues ,in Embed the cue vector sequence / cue token output by the cue generator; embed the cue... Injecting a visual encoder to obtain attribute-enhanced image features ; And through a gating network, the same category Assigning sample conditional weights to multiple attributes Output attribute branch prediction ;in, Indicates the input sample Under the conditions Class 1 The contribution weight of each attribute text to the sample is determined by the gating network based on the gating visual features. Embedded with corresponding attribute text The correlation / similarity calculation yielded And normalize the weights within the same category; Apply Top-k or Top-p sparsity before normalization; where, This represents the input visual features of the gating network, and is selected according to the dataset / domain configuration. or ,in To avoid injecting the aforementioned prompt Input image samples Input the clean image features obtained from the frozen visual encoder, and ; S3.3 Stable Anchor Branch: Input image samples Without injecting the aforementioned prompt Under the condition of inputting a frozen visual encoder, the output of the clean image features is obtained. ; and by calculating the Embedded with category name template text The similarity score between them yields stable anchor branch predictions. The category name template text is embedded therein. Obtained by the frozen text encoder, specifically , To fill the category name into the preset text template obtained by the category name, the preset text template is either pre-set and distributed by the server or pre-set locally by the client; the similarity score is the inner product or cosine similarity, and the similarity scores of each category are combined into a vector as a stable anchor point for branch prediction. The stable anchor branch does not use attribute text and does not calculate weights through a gating network; when at least one of the cue generator or visual encoder contains low-rank adaptation parameters, the low-rank adaptation path associated with attribute branch cue injection is bypassed or disabled in the stable anchor branch to maintain prediction stability. S3.4 Sample-level Dynamic Fusion: Generating Sample-level Basic Fusion Coefficients Based on the Uncertainty Difference Between Attribute Branch and Stable Anchor Point Branch Predictions At least one of entropy-gated suppression and divergence suppression is introduced to form the final fusion coefficient. and to Pruning restricts the values to [0,1]; the outputs of the attribute branch and the stable anchor branch are merged to obtain the final output: ； S3.5 Local Optimization: Based on the final output Construct the loss function and update the lightweight trainable parameter set ; S3.6 Upload and Aggregation: Client uploads updated lightweight parameters On the server side, the global parameters are updated by weighted aggregation based on the local sample size of each client. Repeat steps S2–S3.6 until the preset number of communication rounds or convergence conditions are reached.
2. The method for optimizing visual-linguistic cues in a federated large model according to claim 1, characterized in that, In step S3.1, attribute management includes: processing the input image samples... Based on the dataset identifier or application domain identifier, the corresponding attribute template or attribute file is selected for routing. Normalized mapping alignment is performed on the category names, and missing categories in the attribute file are validated. When a missing category is detected, at least one of the following fault-tolerance strategies is executed to ensure training can continue: reverting to the default attribute template, executing a missing category imputation strategy, executing a dynamic masking strategy, or pulling a general global attribute template from the server for imputation. The dynamic masking strategy includes: calculating the gate weights... When the missing category is found, the candidate attribute weights are reset to a preset minimum value or set to zero, and the remaining weights are renormalized within the same category.
3. The method for optimizing visual-linguistic cues in a federated large model according to claim 1, characterized in that, In step S3.2, the gating network for each category Attribute text collection Calculate the sample conditional weights and for the same category The weights within are normalized; in: ， ， , All are learnable linear mapping matrices. Based on the base temperature, For learnable temperature scaling parameters, and for Apply upper and lower bound pruning to ensure numerical stability; where Select as per dataset / domain configuration or ; and on Apply Top-k or Top-p sparsity and then normalize; and obtain the sample adaptive attribute prototype based on the gated weights: ， make If the logit scaling factor is the visual-language model, then the attribute branch output is: ， in For category Scalar bias; and derived from the attribute mean prototype: ， Generated by multilayer perceptron And the bias Disabled; when disabled, it will cause... Alternatively, a bias term may not be introduced to enhance stability.
4. The method for optimizing visual-linguistic cues in a federated large model according to claim 1, characterized in that, In step S3.3, the stable anchor branch uses clean image features. Embedded with category name template text calculate: ， in The category name template text is obtained by filling the category name into the preset text template; and if the visual encoder contains LoRA, the stable anchor branch bypasses or disables LoRA to maintain anchor prediction stability.
5. The method for optimizing visual-linguistic cues in a federated large model according to claim 1, characterized in that, In step S3.4, sample-level dynamic fusion, the basic fusion coefficients... The uncertainty difference is generated by the prediction distribution of the attribute and stable anchor points. The uncertainty difference includes the normalized entropy difference and the confidence boundary difference. make: ， ， By converged network Mapping The entropy-gated suppression term is: ， in The scaling factor is used; the divergence suppression term is: ， in The inhibition coefficient and the divergence measure This includes any one or any combination of Jensen–Shannon divergence, KL divergence, symmetric KL divergence, predictive distribution cosine divergence, or Top-1 class consistency indicator function; the final fusion coefficient is: 。 6. The method for optimizing visual-linguistic cues in a federated large model according to claim 1, characterized in that, The loss function constructed in step S3.5 includes classification cross-entropy loss and consistency regularization term, as well as federated near-end regularization term: ， in For consistency regularization weights, The consistency regularization term is the weight of the near-end term. For the client to perform local training on input image samples The generated enhanced samples Distance or similarity constraints between the enhanced sample and the original sample in the visual encoder output feature space, wherein the enhanced sample The data is augmented locally on the client side, including at least horizontal flipping; and the global parameters are updated on the server side using weighted aggregation based on the number of samples. ， in For the client The local sample size; the client also employs at least one of two-stage training scheduling and preheating / freezing strategies to improve training stability.
7. The method for optimizing visual-linguistic cues in a federated large model according to claim 6, characterized in that, The client also employs at least one of two-stage training scheduling and preheating / freezing strategies. The two-stage training scheduling includes: in the first stage, only at least one of the prompt generation parameters and prompt injection parameters is updated in the first few communication rounds, while keeping the remaining lightweight parameters frozen; in the second stage, while keeping the original parameters of the visual encoder and text encoder frozen, at least one of the sample-level fusion network parameters and low-rank adaptation parameters is further unfrozen and jointly updated in subsequent communication rounds.