Controllable federated prompt learning method, device and equipment
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2026-04-20
- Publication Date
- 2026-07-14
Smart Images
Figure CN122390001A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of federated learning technology for visual language systems, and relates to a test-time controllable federated prompting learning method, apparatus and device. Background Technology
[0002] Federated Learning (FL) has emerged as a scalable and privacy-preserving solution for training models across decentralized edge devices in recent years. Unlike traditional centralized learning, which aggregates raw data onto a single server, FL allows local devices to collaboratively optimize a global model while keeping the data on the devices themselves. This paradigm is highly valuable in IoT-driven applications, such as smart cameras, autonomous vehicles, and data-sensitive fields like industrial sensing. By leveraging distributed computing and reducing reliance on data transmission, FL provides a viable path to deploying intelligent systems on a large scale.
[0003] However, fundamental limitations remain: once deployed, models typically remain static, struggling to adapt to distribution shifts in local data even if the task and label spaces remain unchanged. Such shifts are frequent in real-world vision, where factors like lighting, background, or viewpoint changes alter data features over time. To address the rigidity of federated learning (FL), research has shifted towards Vision-Language Models (VLMs) such as CLIP. Their powerful generalization capabilities and prompt-based modular adaptation have laid a new foundation for research in federated scenarios, a direction termed Federated Prompt Learning (FPL).
[0004] While FPL methods achieve robust cross-client generalization for the global model, they still struggle to adapt to client-specific local data distributions. To address this deficiency, Personalized Federated Prompt Learning (PFPL) emerged, aiming to adapt prompt-based models to individual client data distributions in the post-training phase before deployment. Although PFPL effectively mitigates heterogeneity issues between clients, it relies solely on historical training data and lacks the ability to adapt to newly emerging, unseen domain variations during inference. Therefore, existing FPL and PFPL methods still have limitations in handling domain offsets during testing, highlighting the urgent need for a federated prompt learning framework with dynamic post-deployment adaptation capabilities. Summary of the Invention
[0005] To address the problems existing in the above-mentioned traditional methods, this invention proposes a test-time controllable federated prompt learning method, a test-time controllable federated prompt learning device, and a computer device, which can effectively achieve efficient adaptation after dynamic deployment of the federated visual language system.
[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a test-time controllable federated prompting learning method is provided, including the following steps: Obtain the client's pre-trained cue set and unlabeled test set; after the federated training process, the client holds pre-trained cue sets including global cue sets, personalized local cue sets, and irrelevant cue sets; irrelevant cue sets are inherited from the federated visual language model and serve as domain-neutral priors; The alignment score is obtained by quantifying the consistency between model predictions and observed data distributions using a model-data alignment metric based on the unlabeled test set. Based on the alignment score, the federated visual language model is dynamically adjusted using a controlled test time adaptation framework to make the federated visual language model suitable for the client.
[0007] On the other hand, a test-controlled federated learning device is also provided, including: The data acquisition module is used to acquire the client's pre-trained cue set and unlabeled test set; after the federated training process, the pre-trained cue held by the client includes global cue, personalized local cue, and irrelevant cue; irrelevant cue is inherited from the federated visual language model and serves as a domain-neutral prior; The model-data alignment module is used to quantify the consistency between model predictions and observed data distributions based on the unlabeled test set using a model-data alignment metric to obtain an alignment score; The test-time adaptation module is used to dynamically adjust the federated visual language model based on the alignment score using a controllable test-time adaptation framework, so that the federated visual language model is adapted to the client.
[0008] In another aspect, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described test-controllable federated prompting learning method.
[0009] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned test-controlled federated prompting learning method, apparatus, and device propose a three-prompt controllable TTFPL framework (COTE), integrating the complementary advantages of original prompts, global prompts, and local prompts to achieve adaptive utilization of general-level, shared-level, and personalized knowledge. It introduces the model-data alignment metric MoDA, a unified standard that quantifies the consistency between model predictions and local data distribution on each client, supporting dynamic, data-aware prompts and sample selection during adaptation. This addresses the domain offset and generalization challenges in federated visual language systems, achieving efficient adaptation after deployment. Extensive experiments on multiple benchmark datasets demonstrate that the method of this invention continuously improves target domain performance, establishing a new paradigm for adaptive, highly generalized federated prompting learning. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is a flowchart illustrating a test-time controllable federated prompting learning method in one embodiment; Figure 2 This is a schematic diagram of three federated learning paradigms in one embodiment; Figure 3 This is a schematic diagram of a federated prompting learning framework during controlled testing in one embodiment; Figure 4 This is a schematic diagram illustrating the influence of coefficient factors in one embodiment, wherein, Figure 4 (a) is the stability coefficient The impact, Figure 4 (b) is the weighting factor The impact; Figure 5 This is a schematic diagram of the modular architecture of a federated prompting learning device that is controllable during testing in one embodiment. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention.
[0013] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.
[0014] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0015] Federated learning supports decentralized model training without sharing raw data, providing a privacy-preserving paradigm for collaborative learning. Cue learning has become an effective method for adapting large-scale Vision-Language Models (VLMs) such as CLIP to downstream tasks. This idea has been extended to decentralized scenarios, giving rise to Federated Cue Learning, which allows clients to exchange and optimize cues, rather than the full model weights. Existing techniques enhance global consistency through category-aware cue generators guided by global label embeddings. Although existing FPL methods achieve strong generalization across different clients by utilizing well-trained cue initialization, they still struggle to adapt to dynamically changing local data features.
[0016] To enable federated learning (FL) models to preferentially adapt to local data features, Personalized Federated Learning (PFL) adapts shared global knowledge to various clients. Existing research addresses this issue from different perspectives: achieving independent local optimization through parameter decoupling, utilizing knowledge distillation to complete cross-client information transfer, and balancing global generalization and local specialization through adaptive model interpolation.
[0017] To enhance generalization capabilities while preserving personalization, recent research on personalized federated learning has incorporated prompt learning, giving rise to Personalized Federated Prompt Learning (PFPL) methods. For example, pFedPrompt learns language consensus while adapting to the client's specific visual distribution; pFedPG generates personalized visual cues on the server side; FedOTP jointly learns global and local cues through optimal transport alignment; and pFedMoAP utilizes an attention-based gating mechanism to filter multiple prompt experts, optimizing the personalization effect. Although existing PFPL methods improve generalization capabilities during training on heterogeneous local data, they still have limitations in deployment scenarios: when only unlabeled test data is available, these methods cannot adapt to unseen domain offsets.
[0018] Test-time adaptation (TTA) improves the robustness of a model to distribution shifts by adapting it to unlabeled target data during the inference phase. Early methods (such as TENT) achieved adaptation by minimizing prediction entropy, while methods like CoTTA, MEMO, and EATA enhance stability through consistency regularization or selective regularization. Recent research has further expanded this paradigm, including contrastive self-regularization (such as SAR and AdaContrast), cue-based fine-tuning (TPT) for visual language models (VLMs), and continuous or energy-based adaptation (such as TTAC and ECoTTA).
[0019] While these methods perform well in centralized scenarios, they all assume access to all data and a single, shared model. These assumptions conflict with the constraints of federated learning regarding privacy, communication, and heterogeneity. Therefore, extending TTA to federated learning scenarios requires lightweight, modular adaptation strategies, and cue-based representations provide a natural and privacy-preserving interface. One existing approach attempts to explore test-time personalization in federated learning by fine-tuning or integrating classification heads using unlabeled client data. However, this approach requires modification of the backbone network, making it incompatible with the deployment-optimized federated learning model. It also relies on explicit classification heads and batch-based updates, which contradicts the working mechanism of CLIP models, which operate by freezing the graph-text alignment of the backbone network without explicit classification heads. These limitations highlight the need for headless, cue-level adaptation during deployment, which is the core motivation behind the Test-Time Federated Cue Learning (TTFPL) framework proposed in this invention.
[0020] Federated prompting learning (FPL) has attracted increasing attention due to its ability to leverage large-scale visual language models such as CLIP within a federated learning framework. Although existing research has driven the development of FPL through personalized strategies to improve client-specific task performance, personalized models often suffer severe performance degradation when deployed in unknown cross-domain scenarios due to distribution offset.
[0021] This invention explores Test-Time Federated Prompt Learning (TTFPL) for the first time, aiming to bridge cross-domain performance gaps with minimal cost and requiring only unlabeled target domain data. To this end, a Controllable Test-Time Federated Prompt Learning (COTE) framework is proposed, a three-prompt controllable TTFPL method that dynamically balances three complementary prompts: a global prompt from the standard FPL, a local prompt from the personalized FPL, and a frozen CLIP original prompt. Specifically, COTE introduces a novel confidence-guided Model-Data Alignment (MoDA) metric to quantify the degree of alignment at both macro and micro levels, capturing the consistency between model predictions and data distribution. By combining MoDA with model confidence, COTE adaptively adjusts the contribution weights of each prompt during the testing phase, achieving robust generalization to heterogeneous clients and unknown domains without labeled data. Extensive experiments on multiple benchmark datasets demonstrate that the method of this invention continuously improves target domain performance, opening up new directions for adaptive federated prompt learning.
[0022] In one embodiment, such as Figure 1 As shown, a test-time controllable federated prompting learning method is provided, which may include the following steps S12 to S16: S12, obtain the client's pre-trained hint set and unlabeled test set; after the federated training process, the client holds pre-trained hints including global hints, personalized local hints and irrelevant hints; irrelevant hints are inherited from the federated visual language model and serve as domain-neutral priors; S14, Based on the unlabeled test set, the alignment score is obtained by quantifying the consistency between the model prediction and the observed data distribution using the model-data alignment index. S16, based on the alignment score, dynamically adjusts the federated visual language model using a controllable test time adaptation framework to adapt the federated visual language model to the client.
[0023] To address the domain transfer problem after deployment in federated visual language systems, the COTE framework combines three complementary cue types. This three-cue design balances local personalization with global knowledge preservation. The cue types include original cues, global cues, and local cues. Original cues come from pre-trained visual language models (such as CLIP) and are used for data-independent generalization. Global cues are aggregated cues on federated clients and are used for task-specific cross-domain semantic sharing. Local cues are personalized for each client and are used for fine-grained perception of local visual patterns.
[0024] COTE dynamically adjusts the use of prompts based on real-time data characteristics, rather than statically combining prompts, forming a dynamic adjustment mechanism that embodies the core of controllable adaptation and opens up a new direction for prompt-based test-time adaptation in federated settings. COTE also introduces a new MoDA (Model-Data Alignment) metric to quantify the consistency between model predictions and the underlying data distribution. This metric measures the consistency between model predictions and the underlying data distribution on each client, laying the foundation for controllable adaptation. COTE also designs a controllable test-time adaptation framework that adjusts model behavior based on client-specific alignment, adapting prompts through three steps: confidence-based sample splitting, high-confidence adaptation, and low-confidence prompt fine-tuning, achieving the goal of a controllable test-time adaptation workflow.
[0025] The aforementioned test-time-controllable federated prompting learning method proposes a three-prompt controllable TTFPL framework, integrating the complementary advantages of original prompts, global prompts, and local prompts to achieve adaptive utilization of general, shared, and personalized knowledge. It introduces the Model-Data Alignment Metric (MoDA), a unified metric that quantifies the consistency between model predictions and local data distribution on each client, supporting dynamic, data-aware prompts and sample selection during adaptation. This addresses the domain offset and generalization challenges in federated visual language systems, achieving efficient adaptation after deployment. Extensive experiments on multiple benchmark datasets demonstrate that the method continuously improves target domain performance, establishing a new paradigm for adaptive, highly generalized federated prompting learning.
[0026] Three federated learning paradigms, such as Figure 2As shown, Test-Time Federated Hints Learning (TTFPL) is a novel paradigm that enables federated hint learning models to dynamically adapt to distribution shifts using only unlabeled test data. FPL leverages the strong generalization of the Vision Language Model (VLM), while PFPL fine-tunes hints using local training data before deployment. TTFPL focuses on enabling the model to adapt to the test data distribution of each client while preserving global knowledge across clients. Unlike Personalized Federated Hints Learning (PFPL), which fine-tunes hints using local training data, TTFPL allows edge devices to adapt hints during the inference phase. This feature is crucial for IoT vision scenarios, where post-deployment labeled data is often not feasible. This overlooked task presents two inherent challenges to the deployment of federated vision language models (VLM): (1) how to adapt local personalized hints to visual distribution shifts in unlabeled test data (WD) without accessing historical training samples; and (2) how to preserve the global knowledge encoded in the federated hints across clients, ensuring that the adaptation process does not reduce the model's generalization ability across different clients.
[0027] This dual challenge constitutes the core dilemma: how to strategically utilize different types of cues during the adaptation process to balance local distribution alignment with global knowledge preservation—a problem that has not yet been solved by existing cue learning or Test-Time Adaptation (TTA) frameworks.
[0028] In one embodiment, regarding step S14 above, the process of quantifying the consistency between model predictions and observed data distributions using a model-data alignment metric may specifically include the following processing: Calculate the empirical category frequency vector based on the pseudo-label distribution predicted by the federated visual language model on the unlabeled test set; Macro-level statistics are extracted based on empirical category frequency vectors; macro-level statistics include the normalized Gini coefficient and normalized entropy. Micro-level statistics are calculated based on empirical category frequency vectors; micro-level statistics include local regularity indices. The model-data alignment index integrates macro-level and micro-level statistics; the model-data alignment index adaptively controls the weights of macro-level and micro-level statistics through weighting factors. The alignment score is calculated based on the model-data alignment metric; the alignment score measures the consistency between the model's predictions and the observed data distribution.
[0029] It is understood that, in order to address the above problems, this invention proposes a controllable test-time federated prompting learning framework, COTE. This framework adopts a three-prompt design and introduces a new model-data alignment metric, MoDA, to achieve controllable, data-aware adaptation.
[0030] To effectively regulate the use of three-way prompts, MoDA, a unified metric, is introduced to quantify the consistency between model predictions and underlying data distribution on each client, laying the foundation for controllable adaptation. MoDA captures alignment at both the macro and micro levels: at the macro level, it evaluates how the model's prediction distribution is organized in the global semantic space, reflecting overall sparsity and diversity; at the micro level, it measures the local stability and balance among activation categories. Unlike traditional single-dimensional metrics, MoDA can stably and sensitively evaluate client distribution heterogeneity, supporting dynamic, data-aware prompt selection during the adaptation process in testing.
[0031] In one embodiment, regarding step S16 above, the process of dynamically adjusting the federated visual language model based on the alignment score using a controllable test-time adaptation framework may specifically include the following processing: Based on the confidence scores of the samples obtained from the federated visual language model on the client, the samples are divided into a high-confidence sample set and a low-confidence sample set; For high-confidence samples, pseudo-label supervision is used for cue optimization guided by the alignment score; if the alignment score is positive, global cue is used, otherwise local cue is used. For low-confidence samples, a sample-level testing-based optimization strategy is employed, guided by the alignment score, to provide suggestions for optimization.
[0032] Specifically, MoDA, combined with pseudo-label confidence, enables controllable cue selection: First, using confidence scores generated by the model based on local cueing, samples are divided into high-confidence and low-confidence sets. Within each set, MoDA evaluates the alignment between the model's prior knowledge and the local data structure, thus guiding decision-making. For highly aligned samples, global cueing is used to leverage shared knowledge across clients, while raw cueing is used for low-confidence samples to improve generalization under uncertainty. Conversely, for imbalanced or weakly aligned samples, local cueing is selected by default to better capture domain-specific details. This data-aware, controllable cue selection mechanism allows each client to achieve robust self-supervised adaptation without external labels, achieving a controllable balance between local personalization and global generalization.
[0033] Specifically, regarding Test-Time Federated Hints Learning (TTFPL), the problem is as follows: Consider a federated hints learning system built upon a pre-trained visual language model (VLM), whose image encoder... and text encoder All are currently frozen. After the federated training process, each client... It holds three pre-trained hints: a global hint aggregated across clients and used to capture shared semantics. ,based on Personalized local hints initialized and optimized on local data Irrelevant hints inherited from the original VLM model This serves as a domain-neutral a priori. During the deployment phase, the client... i Receive unlabeled test sets from an unknown distribution This distribution may be related to its training data. There are differences. It should be noted that training datasets, test data labels, and cross-client communication resources are unavailable at this stage.
[0034] Definition: Given each client Pre-training cue set = and its unlabeled test set During testing, Federated Hints Learning (TTFPL) aims to improve learning through local adaptation operators. Get adaptive prompts during testing The definition is as follows: (1) Due to the lack of test labels, a general unsupervised test-time objective function acting on the prediction distribution needs to be utilized. To optimize local adaptation operators : (2) in, Indicates the input sample. This represents the model's predicted labels. To obtain this distribution, a similarity-based visual-language matching framework is used. For each category... Text input Adaptive prompts during testing Formed by combining the category name. The corresponding log odds and predictive probabilities are defined as follows: (3) (4) in, It is a temperature parameter.
[0035] The designed controllable test-time federated hint learning framework COTE, such as Figure 3As shown, the process begins with visual and textual input encoded by the CLIP model to predict domain-neutral labels (references). Its distribution is then analyzed using MoDA scores and compared to the client's ideal reference to calculate an alignment score, which guides a controlled-time adaptation workflow. The model adjusts its cues in two branches, with sample partitioning based on CLIP confidence: batch adaptation is performed on high-confidence samples, refining the cues using high-confidence pseudo-labels; sample adaptation is applied to low-confidence samples through rapid refinement by selecting a high-entropy augmented view. Figure 3 A checkmark indicates acceptance, an X indicates rejection, and a dashed arrow indicates backpropagation.
[0036] COTE enables models to adapt to changing unlabeled data. The framework consists of two main modules: the Model-Data Alignment Metric (MoDA) and the Controlled Test-Time Adaptation Workflow. MoDA quantifies the consistency between model predictions and observed data distributions, while the Controlled Test-Time Adaptation Workflow adjusts model cues based on the alignment score. These two modules work together to dynamically balance global generalization and local specialization, ensuring effective adaptation to real-world data variations.
[0037] Components of the MoDA metric: For each client, analyze its unlabeled test set. The pseudo-label distribution obtained from the above prediction. Let the global label space be... S , This represents the subset of categories actually predicted by the deployed federated visual language model on this client. The empirical category frequency vector is defined as: (5) in, n k Indicates being assigned to a category k The number of samples.
[0038] First, macro-level statistics are extracted to describe the model's predictions in the global label space that includes unobserved categories. S The distribution of data on the model. These statistics are used to measure the global equilibrium or sparsity of the prediction model: (6) Normalized Gini coefficient G It emphasizes the degree of concentration on the dominant category, while normalized entropy E This reflects the degree of dispersion among the tail categories. (Through |) S Normalization ensures that unobserved categories will influence these metrics, making... G and E It can reflect the global coverage of the semantic space.
[0039] The calculation is limited to the actual active subset.S obs Microscopic statistics within the model characterize the uniformity of the distribution of the model's predictions across the categories actually used by the client. The deviation from local uniformity is defined as: (7) The local regularity index is defined as follows: (8) smaller R A value indicating that predictions are concentrated in a few active categories implies bias or instability; while a larger value indicates that predictions are concentrated in a few active categories. R The value reflects that the model adapts more smoothly within the local support domain.
[0040] The unified MoDA formula integrates macro- and micro-level statistics into a unified model-data alignment (MoDA) metric, or MoDA score, which quantifies the consistency between model predictions and the client's local data structure.
[0041] (9) Among them, weighting factors It adaptively controls the contributions of statistics at both the macro and micro levels: (10) Intuitively speaking, This reflects the confidence level in the observed predicted structure. When only a few categories are activated... , It depends on global regularities; however, as more categories are observed, It will focus on local regularity R This allows the model to smoothly transition between global calibration and local specialization. Empirically, a higher... A value of [value] implies stable general domain prediction; while a lower value indicates [prediction]. The value indicates a deviation in the prediction, suggesting a need for stronger personalized adaptation.
[0042] To interpret the MoDA metric in conjunction with an ideal configuration, a client-specific reference is introduced: the model's performance on the observed subset of labels. S obs The prediction results are uniformly distributed to reflect the maximum alignment with the available data.
[0043] (11) Based on this definition, the corresponding macro-level and micro-level statistics become: (12) This reflects a relatively ideal calibrated model-data relationship, exhibiting neither overconfidence nor bias towards any class subset. The corresponding ideal MoDA metric is defined as: (13) To measure the deviation between the client's predicted state and the ideal state, an alignment signal (i.e., alignment score) is calculated: (14) in, It is the stability coefficient. A higher value indicates a better fit between the model's predictions and those trained on diverse data, meaning the current data distribution is more compatible with the characteristics of this type of model. Conversely, a lower value indicates a lower value. The lower the value, the better the data distribution fits the model that is "more biased towards specific data features," indicating that further adaptation is needed. Therefore, It can serve as a soft guide for a controllable and adaptable workflow, enabling the model to adjust its strategy based on its alignment with the underlying data structure.
[0044] To align the signal For practical implementation, a controllable test-time adaptation framework was designed, which adjusts the model behavior based on client-specific alignment. Guided by this framework, the framework completes the prompt adaptation in three steps: confidence-based sample segmentation, high-confidence adaptation, and low-confidence prompt fine-tuning.
[0045] For each client, pseudo-labels and confidence scores are obtained from the visual language model CLIP. The predictions of this CLIP model serve as frozen, domain-neutral references for evaluating prediction reliability. According to the definitions in equations (3) to (4), the samples... The prediction confidence level is calculated as follows: (15) in, This represents the normalized probability predicted by the model.
[0046] The samples will be based on the confidence threshold. The samples were divided into two subsets: a high-confidence sample set and a low-confidence sample set. (16) High confidence samples To provide reliable pseudo-supervision for optimization, while low-confidence samples This will be reserved for adaptation purposes in subsequent processes to detect uncertainties.
[0047] Fitting high-confidence samples: For high-confidence samples, in aligning the signal... Guided by this approach, pseudo-label supervision is used to optimize the most reliable suggestions. Let the selected suggestion be: (17) Among them, positive A value indicates global alignment behavior, favoring the use of global hints; while a negative value indicates... The value corresponds to local tilting behavior, and therefore depends on local cues. Under a selected cue, the category... k The predicted probability follows formula (4), and the pseudo-label is defined as: (18) Subsequently, high-confidence samples were used. Optimize the prompts using a loss function: (19) This update enhances the cues that align with client-side conditions, ensuring stable adaptation through reliable pseudo-label supervision.
[0048] For adaptation of low-confidence samples, direct pseudo-label optimization is unreliable. Therefore, a method based on the alignment signal is used. The guided sample-level test prompt tuning (TPT) strategy has the following initialization options for adaptive prompts: (20) in, The hints generated by the CLIP model provide domain-independent initialization, while personalized local hints... This captures client-specific prior information. For each uncertain sample ,generate Enhanced view And calculate their prediction entropy: (twenty one) Top- A high-entropy view is denoted as And optimize the hints by maximizing their average entropy: (twenty two) This process allows each uncertain sample to adjust its representation toward the most suitable initialization method (CLIP or local), thereby enhancing robustness to distribution shifts during inference.
[0049] In some implementations, to more clearly and intuitively demonstrate the implementation of the above-described test-time controllable federated prompting learning method, the following experimental examples are also provided: Datasets: The proposed method is evaluated on five commonly used benchmark datasets: CIFAR100, ImageNet and its variants (such as ImageNet-A, ImageNet-V2 and ImageNet-R), Flowers-102, Caltech-101, and Food-101. These datasets cover different domains and granularities, including general object recognition and fine-grained recognition tasks.
[0050] Following the BRFL benchmark protocol, various test-time distribution offsets were simulated to evaluate robustness. Four test scenarios were constructed for the benchmark datasets CIFAR100, Flowers102, Caltech101, and Food101: (1) Ori (original test set); (2) Corr (adversarial perturbation); (3) Ooc (out-of-client data from other users); and (4) Mix (a combination of all the above scenarios to simulate real-world deployments). For the benchmark dataset ImageNet, ImageNet-A (A in the table below), ImageNet-V2 (V2 in the table below), and ImageNet-R (R in the table below) were used to represent cross-domain offsets. In the table below, Avg refers to the mean.
[0051] Settings: To simulate a real-world scenario-driven environment, using concentration parameters... The Dirichlet partition constructs a strongly non-independent, identically distributed (non-IID) federated environment, which leads to extreme label heterogeneity among clients.
[0052] Benchmark Methods: The proposed method of this invention is compared with two representative methods, covering two paradigms: Federated Prompt Learning (FPL) and Test-Time Adaptive (TTA): (1) Federated Prompt Learning methods include PromptFL (i.e., Prompt-based Federated Learning), PromptFL+FT (i.e., PromptFL with Fine-Tuning), pFedMoAP (i.e., Personalized Federated Mixture of Adaptive Prompts), and FedOTP (i.e., Federated Prompts Cooperation via Optimal Transport). Among them, PromptFL learns global prompts through client collaboration; PromptFL+FT fine-tunes local data to achieve personalization; pFedMoAP and FedOTP further enhance the personalization capability and robustness of the federated system. (2) Test-time adaptive methods include existing methods such as PL (Pseudo-Labeling), TENT (Test-Time Entropy Minimization), and TPT (Test-Time Prompt Tuning). These methods are adapted to federated scenarios: test-time optimization is performed on the personalized model of PromptFL+FT to evaluate the adaptive capability under distribution shift.
[0053] Implementation details: All methods are implemented using PyTorch (an open-source deep learning framework based on Python), employing a frozen CLIP ViT-B / 16 backbone network. Only the cue parameters are optimized, and each cue contains 16 learnable context labels. Local cue training uses the SGD optimizer, with one local training epoch per round, followed by aggregation using FedAvg (Federated Averaging). Global cues are retained after aggregation. Local prompts With CLIP prompt It is used for adaptive testing.
[0054] During the inference phase, the cue is the only learnable component; samples pass the confidence threshold. Partitioning: Low-confidence samples are divided using an adaptive strategy. An enhanced view and the front The high-entropy samples are optimized using a 1-step SGD method; the stability coefficient in equation (14) is set to... .
[0055] Table 1 shows the test accuracy of the CIFAR100 and ImageNet datasets under different test distributions, assuming non-uniform client-side data partitioning. The best results in each column are highlighted in bold, while the second-best results are highlighted in underline.
[0056] Table 1
[0057] Results on CIFAR100 and ImageNet: Table 1 shows the experimental results on the CIFAR100 and ImageNet datasets under the condition of non-uniform partitioning of client data. The method of this invention (COTE) achieved the highest average accuracy on both benchmark datasets: 71.08% on CIFAR100 and 81.44% on ImageNet, outperforming the strongest benchmark method (TPT) by more than 6 percentage points on average.
[0058] This performance improvement is particularly significant in the most challenging OOC and Mix scenarios, where existing fine-tuning or pseudo-labeling strategies struggle to maintain robustness. Specifically, the method of this invention improves OOC accuracy on CIFAR100 from 19.2% to 42.3% and on ImageNet from 6.9% to 36.6%, demonstrating a significant enhancement in generalization ability on unseen client distributions. Furthermore, its performance on in-distribution and natural offset variants (such as ImageNet-A, V2, and R) is comparable to or even better than all benchmark methods, proving that the proposed alignment guidance mechanism achieves both stability and adaptability under diverse distribution offsets. It is worth noting that representative personalized federated cue learning (PFPL) methods perform poorly because their personalization mechanism highly aligns the cue with a biased local training distribution, which has a low match with the test data distribution in the experimental setup.
[0059] Results on Caltech101, Flowers102, and Food101: Table 2 reports the experimental results on the fine-grained recognition datasets. On these three benchmark datasets, the method of this invention consistently achieves the best overall performance. This advantage is most pronounced in the OOC configuration, where the data distribution deviates significantly from the distribution observed during training. Compared to the strongest benchmark method, the method of this invention achieves absolute accuracy improvements of 2.7%, 5.7%, and 23.4% on these three datasets, respectively. These results highlight the effectiveness of the proposed COTE in promoting stable prompt optimization for clients with unique or long-tailed data distributions, while maintaining excellent performance in both in-distribution and data-impaired scenarios.
[0060] Table 2
[0061] Ablation experiment results: (1) Impact of Alignment Indicators: Table 3 compares the proposed MoDA indicator with single-factor indicators (such as Gini coefficient, entropy, and regularity) in the same adaptive process of CIFAR100. Among these alignment options, MoDA achieved the best overall average (71.08% on CIFAR100), while the entropy indicator was the second best choice. This indicates that in large-scale, high-cardinality label spaces, pure uncertainty cues (entropy) perform better; while MoDA integrates macroscopic ( G , E ) and micro ( R This statistic is therefore more stable and reliable in heterogeneous scenarios (including situations where the client distribution is smaller or more unbalanced).
[0062] Table 3
[0063] (2) Impact of Sample Splitting Strategy: Table 4 illustrates the role of confidence-based sample splitting in the adaptive process. When all samples are processed using only high-confidence or low-confidence methods, overall performance significantly decreases, especially in OOC and Mix distribution shift scenarios. On CIFAR100, removing the splitting mechanism reduces accuracy from 71.08% to 65.04%; on ImageNet, it decreases from 81.44% to 75.21%. These observations demonstrate the critical importance of distinguishing between high-confidence and low-confidence samples. This splitting facilitates accurate pseudo-label adaptation for high-confidence predictions while enabling more conservative entropy-based updates for uncertain situations, thus improving both stability and generalization.
[0064] Table 4
[0065] Further analysis of stability coefficients Impact: Figure 4 (a) The stability coefficient was evaluated. Its function is to control the calculation of the alignment signal in equation (14). The influence weight of the ideal alignment reference. Increasing the value from 0.7 to 0.9 resulted in a stable performance improvement on both CIFAR100 and ImageNet; when The optimal average accuracy was achieved at a value of 0.9 (71.08% and 81.44%, respectively). This is a moderate level. A good balance is achieved between sensitivity to client-specific offsets and noise immunity stability.
[0066] Weighting factors Impact: Figure 4 (b) shows the weighting factors. The role of this factor is to provide global and local statistics for the MoDA index in the balanced equation (10). Extreme settings ( The value of 0 or 1 results in a significant performance drop, indicating that relying solely on global or local information can lead to overgeneralization or overfitting. The value in the middle range ( The optimal result ([0.25, 0.50]) highlights the importance of jointly utilizing global calibration and local consistency. The adaptive strategy adjusts based on the number of observed categories for each client. Its performance is comparable to or slightly better than the optimal fixed value. This indicates that the adaptive mechanism effectively balances versatility and specificity, ensuring stable performance across diverse client distributions.
[0067] The above experimental examples, conducted extensively on five benchmark datasets, demonstrate that COTE consistently improves performance across different federated scenarios, establishing a new direction for adaptive and general federated cue learning.
[0068] It should be understood that, although Figure 1 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0069] In one embodiment, such as Figure 5As shown, a test-time controllable federated cue learning device is provided, including a data acquisition module 11, a model data alignment module 13, and a test-time adaptation module 15. The data acquisition module 11 acquires the client's pre-trained cue set and unlabeled test set. After federated training, the client's pre-trained cue set includes global cue, personalized local cue, and irrelevant cue. The irrelevant cue is inherited from the federated visual language model and serves as a domain-neutral prior. The model data alignment module 13 quantifies the consistency between the model's predictions and the observed data distribution using a model-data alignment metric based on the unlabeled test set, obtaining an alignment score. The test-time adaptation module 15 dynamically adjusts the cue settings of the federated visual language model using a controllable test-time adaptation framework based on the alignment score, adapting the federated visual language model to the client.
[0070] The aforementioned test-controlled federated prompting learning device proposes a three-prompt controllable TTFPL framework (COTE), integrating the complementary advantages of original prompts, global prompts, and local prompts to achieve adaptive utilization of general, shared, and personalized knowledge. It introduces the model-data alignment metric MoDA, a unified standard that quantifies the consistency between model predictions and local data distribution on each client, supporting dynamic, data-aware prompts and sample selection during adaptation. This addresses the domain offset and generalization challenges in federated visual language systems, achieving efficient adaptation after deployment. Extensive experiments on multiple benchmark datasets demonstrate that the method of this invention continuously improves target domain performance, establishing a new paradigm for adaptive, highly generalized federated prompting learning.
[0071] In one embodiment, the model data alignment module 13 may specifically include: The class vector submodule calculates the empirical class frequency vector based on the pseudo-label distribution predicted by the federated visual language model on the unlabeled test set. The macro-statistics submodule extracts macro-level statistics from the empirical class frequency vector; these statistics include the normalized Gini coefficient and normalized entropy. The micro-statistics submodule calculates micro-level statistics from the empirical class frequency vector; these statistics include the local regularity index. The index integration submodule integrates the macro-level and micro-level statistics into a model-data alignment index; this index adaptively controls the weights of the macro-level and micro-level statistics through weighting factors. The alignment calculation submodule calculates the alignment score based on the model-data alignment index; this score measures the consistency between the model's predictions and the observed data distribution.
[0072] In one embodiment, the test adaptation module 15 may specifically include: The sample partitioning submodule divides samples into high-confidence and low-confidence sample sets based on the confidence scores obtained from the federated visual language model on the client side. The high-hint optimization submodule optimizes hints for high-confidence samples using pseudo-label supervision, guided by the alignment score; if the alignment score is positive, global hints are used; otherwise, local hints are used. The low-hint optimization submodule optimizes hints for low-confidence samples using a sample-level testing-based hint tuning strategy, guided by the alignment score.
[0073] It is understood that the specific limitations of the test-time controllable federated prompting learning device mentioned above can be found in the corresponding limitations of the test-time controllable federated prompting learning method mentioned above, and will not be repeated here.
[0074] Each module in the aforementioned controllable federated learning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of a device with data processing capabilities, or stored in software within the memory of the aforementioned device, so that the processor can invoke and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of data processing devices already existing in the art.
[0075] In one embodiment, a computer device is also provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following processing steps: acquiring a pre-trained cue set and an unlabeled test set for the client; the pre-trained cue set held by the client after a federated training process includes global cue, personalized local cue, and irrelevant cue; the irrelevant cue is inherited from the federated visual language model and serves as a domain-neutral prior; based on the unlabeled test set, a model-data alignment metric is used to quantify the consistency between the model's predictions and the observed data distributions to obtain an alignment score; based on the alignment score, the federated visual language model is dynamically adjusted using a controllable test-time adaptation framework to adapt the federated visual language model to the client.
[0076] In one embodiment, when the processor executes the computer program, it can also implement the steps or sub-steps added to the various embodiments of the test-time controllable federated prompting learning method described above.
[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus DRAM (RDRAM), and interface DRAM (DRDRAM), etc.
[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0079] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of protection of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and all such modifications and improvements fall within the scope of protection of the present invention.
Claims
1. A test-time controllable federated prompting learning method, characterized in that, Including the following steps: Obtain the client's pre-trained hint set and unlabeled test set; after the federated training process, the client holds pre-trained hints including global hints, personalized local hints, and irrelevant hints; Irrelevant hints are inherited from the federated visual language model and serve as domain-neutral priors; The alignment score is obtained by quantifying the consistency between model predictions and observed data distributions using a model-data alignment metric based on the unlabeled test set. Based on the alignment score, the federated visual language model is dynamically adjusted using a controlled test time adaptation framework to make the federated visual language model suitable for the client.
2. The test-time controllable federated prompting learning method according to claim 1, characterized in that, The process of quantifying the consistency between model predictions and observed data distributions using model-data alignment metrics includes: Calculate the empirical category frequency vector based on the pseudo-label distribution predicted by the federated visual language model on the unlabeled test set; Macro-level statistics are extracted based on empirical category frequency vectors; macro-level statistics include the normalized Gini coefficient and normalized entropy. Micro-level statistics are calculated based on empirical category frequency vectors; micro-level statistics include local regularity indices. The model-data alignment index integrates macro-level and micro-level statistics; the model-data alignment index adaptively controls the weights of macro-level and micro-level statistics through weighting factors. The alignment score is calculated based on the model-data alignment metric; the alignment score measures the consistency between the model's predictions and the observed data distribution.
3. The test-time controllable federated prompting learning method according to claim 1 or 2, characterized in that, The process of dynamically adjusting the federated visual language model based on alignment scores using a controlled test-time adaptation framework includes: Based on the confidence scores of the samples obtained from the federated visual language model on the client, the samples are divided into a high-confidence sample set and a low-confidence sample set; For high-confidence samples, pseudo-label supervision is used for cue optimization guided by the alignment score; if the alignment score is positive, global cue is used, otherwise local cue is used. For low-confidence samples, a sample-level testing-based optimization strategy is employed, guided by the alignment score, to provide suggestions for optimization.
4. A test-controllable federated prompting learning device, characterized in that, include: The data acquisition module is used to acquire the client's pre-training cue set and unlabeled test set; After the federated training process, the pre-trained cues held by the client include global cues, personalized local cues, and irrelevant cues; Irrelevant hints are inherited from the federated visual language model and serve as domain-neutral priors; The model-data alignment module is used to quantify the consistency between model predictions and observed data distributions based on the unlabeled test set using a model-data alignment metric to obtain an alignment score; The test-time adaptation module is used to dynamically adjust the federated visual language model based on the alignment score using a controllable test-time adaptation framework, so that the federated visual language model is adapted to the client.
5. The test-time controllable federated prompting learning device according to claim 4, characterized in that, The model data alignment module includes: The category vector submodule is used to calculate the empirical category frequency vector based on the pseudo-label distribution predicted by the federated visual language model on the unlabeled test set. The macro-statistics submodule is used to extract macro-level statistics based on empirical category frequency vectors; macro-level statistics include the normalized Gini coefficient and the normalized entropy. The microstatistics submodule is used to calculate micro-level statistics based on empirical category frequency vectors; micro-level statistics include local regularity indices. The indicator integration submodule is used to integrate macro-level and micro-level statistics into a model-data alignment indicator; the model-data alignment indicator adaptively controls the weights of macro-level and micro-level statistics through weight factors. The alignment calculation submodule is used to calculate the alignment score based on the model-data alignment metric; the alignment score is used to measure the consistency between the model prediction and the observed data distribution.
6. The test-time controllable federated learning device according to claim 4 or 5, characterized in that, The testing adaptation module includes: The sample partitioning submodule is used to divide the samples into high-confidence sample sets and low-confidence sample sets based on the confidence scores of the samples obtained from the federated visual language model on the client. The high-hint optimization submodule is used to perform hint optimization for high-confidence samples using pseudo-label supervision under the guidance of alignment scores; if the alignment score is positive, global hints are used, otherwise local hints are used. The low-hint optimization submodule is used to optimize hints for low-confidence samples by employing a sample-level test hint tuning strategy guided by the alignment score.
7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the test-time controllable federated prompting learning method as described in any one of claims 1 to 3.