A test-time adaptive method and apparatus based on sparse supervision explicit modeling of domain variables

Abstract

The application provides a kind of test time self-adaptive method and device based on sparse supervision explicit modeling field variable, comprising: first, for the setting that test stage only contains untagged stream data, the exclusive field representation of each sample is explicitly constructed;Then, the dense and continuous multimodal features are extracted using a visual-language pre-training model, and the field parameter of each sample is parameterized as a learnable distribution variable;Then, a momentum-updated sparse field library is designed, which provides structured prior for the field variable through decoupling supervision mechanism;On this basis, the learned explicit field clues are injected into the downstream model, and efficient adaptation can be achieved by using only the basic entropy minimization criterion without relying on complex online optimization strategy;Finally, through the system experiment verification on multiple standard robustness benchmarks, by introducing explicit and structured field representation mechanism, the application reveals that the key of robust adaptation does not come from complex adaptation algorithm.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a test-time adaptive method and apparatus based on sparse supervised explicit modeling of domain variables, applicable to scenarios that address training and testing distribution shifts in unlabeled test data streams. Background Technology

[0002] Test-Time Adaptation (TTA) is an important but highly challenging task with wide applications in open and dynamic environments such as autonomous driving, medical image analysis, and remote sensing monitoring. Its core objective is to enable the model to adapt to distribution shifts between training and testing by utilizing only unlabeled test data streams during the inference phase. In recent years, several deep learning-based TTA methods have been proposed, but they generally rely on strong assumptions about the test domain or are susceptible to prediction noise, limiting their robustness in complex real-world scenarios.

[0003] To improve adaptability, recent studies have begun to introduce structured modeling and pre-training priors. These methods are mainly divided into three categories: (1) Entropy minimization methods: These methods optimize model parameters to reduce the entropy of the predictions of test samples, motivated by the fact that correct predictions usually correspond to a sharper output distribution. However, this strategy is prone to falling into false confidence when there is a severe domain shift, leading to error accumulation. (2) Prototype-guided methods: These methods maintain class or domain prototypes for momentum updates to guide feature space alignment. However, these methods require the assumption that class semantics are stable and are difficult to cope with unknown or open domain changes. (3) Self-supervised consistency methods: These methods apply enhancements to test samples and force multi-view prediction consistency, utilizing pre-defined invariants for adaptation. However, when enhancements destroy key domain-specific cues, they can actually impair performance.

[0004] Despite the progress made by the aforementioned methods, their common limitation lies in the lack of explicit modeling of the domain itself. They either completely ignore the domain structure or implicitly infer domain information from potentially unreliable predictions. This makes existing TTA methods vulnerable to complex, multi-dimensional, sample-level domain shifts. Therefore, constructing a general, decoupled, and zero-shot generalization mechanism for domain representation is crucial to overcoming the current performance bottleneck of TTA.

[0005] (1) Implicit domain modeling leads to the problem of adaptation vulnerability: Existing test-time adaptation (TTA) methods usually assume that all test samples follow a single global domain distribution, or implicitly infer domain information only through indirect signals such as prediction confidence and feature statistics. However, the domain shift in the real world is multidimensional and sample-specific (such as independent changes in factors such as illumination, style, and weather). This implicit modeling method is difficult to accurately characterize the real domain attributes of each sample, resulting in instability of the adaptation process under conditions of large domain shift or blurred vision, and even performance degradation.

[0006] (2) Lack of explicit, decoupled, and general domain representation mechanisms: Current TTA methods rarely introduce structured representations of the "domain" itself. On the one hand, standard datasets only provide discrete, coarse-grained domain labels (such as "rainy day" and "cartoon"), which cannot support continuous, multi-dimensional, general domain modeling; on the other hand, domain cues extracted directly from image features are often highly entangled with semantic information, making it difficult to generalize to new domains. This means that even with advanced adaptive strategies, the model is still limited by a vague understanding of the nature of the domain. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a test-time adaptive method and apparatus based on sparse supervised explicit modeling of domain variables. Specifically, this invention constructs a general domain encoder (DOME) that utilizes a vision-language pre-trained model (such as CLIP) to generate dense, continuous domain distribution variables for any test sample in a zero-shot manner, achieving accurate characterization of multi-dimensional domain shifts. Secondly, this invention designs a momentum-updated sparse domain library, decoupling domain information from semantic content through a sparse activation mechanism, and using this as a supervisory signal to guide domain representation learning. Furthermore, this invention employs an explicit domain injection mechanism, injecting the learned domain variables into the downstream model through a lightweight MLP adapter. With only a few parameters optimized, the basic entropy minimization strategy achieves robust and efficient test-time adaptation, significantly outperforming existing complex methods.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A test-time adaptive method based on sparse supervised explicit modeling of domain variables includes the following steps:

[0010] Step S1: Train the domain encoder DOME using a sparse-to-dense learning framework. This framework contains two collaborative components: (a) a sparse label domain decoupling module, which separates domain-specific factors that are independent of category semantics from the input samples; and (b) a dense domain representation learning module, which extrapolates sparse discrete domain labels into a continuous, multi-dimensional domain distribution space to achieve fine-grained domain modeling for any sample.

[0011] Step S2: During the inference deployment phase, unlabeled test samples are input into the trained DOME encoder, and zero-sample explicit domain variables are generated.

[0012] Step S3: Inject the generated display domain variables into the downstream backbone model through a lightweight MLP adapter, fine-tune only the adapter and normalization layer, and adapt during testing in combination with the self-entropy minimization objective;

[0013] Step S4: Dynamically adjust the feature representation using the guided model with injected explicit domain clues to achieve robust adaptation to the unknown target domain and output the final prediction result.

[0014] A test-time adaptive device based on sparse supervised explicit modeling of domain variables includes the following steps:

[0015] The modeling module employs a sparse-to-dense learning framework to train the domain encoder DOME, which includes two collaborative components: (a) a sparse label domain decoupling module, used to separate domain-specific factors that are independent of category semantics from input samples; and (b) a dense domain representation learning module, which extrapolates sparse discrete domain labels into a continuous, multi-dimensional domain distribution space, enabling fine-grained domain modeling for any sample.

[0016] In the inference deployment module, during the inference deployment phase, unlabeled test samples are input into the pre-trained DOME encoder, and zero samples are used to generate the corresponding explicit domain variables.

[0017] The adaptive module injects the generated display domain variables into the downstream backbone model through a lightweight MLP adapter, only fine-tuning the adapter and normalization layer, and adapts during testing in combination with the self-entropy minimization objective;

[0018] The output module dynamically adjusts the feature representation using the guided model guided by the injected explicit domain clues, achieving robust adaptation to the unknown target domain and outputting the final prediction result.

[0019] An electronic device includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method.

[0020] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the method described thereon.

[0021] The beneficial effects of this invention: The technical solution proposed in this invention, by constructing a general domain encoder DOME and a sparse-to-dense (S2D) learning framework, achieves for the first time explicit, decoupled, and continuous modeling of the domain attributes of test samples. This solution effectively overcomes the limitations of existing adaptive methods during testing that rely on implicit domain inference or global domain assumptions. It significantly improves the model's generalization ability under large domain shifts without requiring class labels or complex online optimization. Experiments show that even when combined with the most basic entropy minimization strategy, this method achieves leading performance on multiple standard benchmarks. Overall, this technical solution provides an efficient, general, and scalable new paradigm for robust model deployment in open and dynamic environments, possessing significant theoretical value and practical application prospects. Attached Figure Description

[0022] Figure 1 This is a flowchart of the test-time adaptive method for domain variables based on sparse supervised explicit modeling, as described in this invention.

[0023] Figure 2 This is a schematic diagram of the model framework of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above objectives, this invention adopts the following technical solution.

[0025] This invention provides a test-time adaptive method based on sparse supervised explicit modeling of domain variables, such as... Figure 1 As shown, it includes the following steps:

[0026] Step S1: Train the domain encoder DOME using a sparse-to-dense learning framework. This framework contains two collaborative components: (a) a sparse label domain decoupling module, which separates domain-specific factors that are independent of category semantics from the input samples; and (b) a dense domain representation learning module, which extrapolates sparse discrete domain labels into a continuous, multi-dimensional domain distribution space to achieve fine-grained domain modeling for any sample.

[0027] Step S2: During the inference deployment phase, unlabeled test samples are input into the trained DOME encoder, and zero samples generate their corresponding explicit domain variables (in the form of a distribution parameterized by mean and variance).

[0028] Step S3: Inject the generated display domain variables into the downstream backbone model (such as ViT) through a lightweight MLP adapter, fine-tune only the adapter and normalization layer, and adapt during testing in combination with the self-entropy minimization objective;

[0029] Step S4: Dynamically adjust the feature representation using the guided model with injected explicit domain clues to achieve robust adaptation to the unknown target domain and output the final prediction result.

[0030] The sparse-to-dense learning framework disclosed in this embodiment aims to learn decoupled, transferable domain representations by leveraging sparse domain label supervision.

[0031] In step S1(a) sparse labeling neighborhood decoupling module, firstly, for the neighborhood encoder DOME as shown in Figure 2, image sequence features are fused. Domain label text sequence features Capture the stylistic features of the domain:

[0032] ;

[0033] fusion characteristics of DOME output After passing through multiple transformers, including attention, residual, normalization, and linear layers, it is represented as a compact channel statistic (mean). and standard deviation ), to capture explicit domain variables:

[0034] ;

[0035] Then, sparse labeling is used for domain decoupling to help the model learn domain-discriminative yet class-invariant representations. Encoded terms extracted from the CLIP visual encoder, after removing positional encodings, are used to calculate the original style statistics through channel statistics and then reparameterized into latent vectors. :

[0036] ;

[0037] The parameters obtained from sparse label domain decoupling include the mean. and standard deviation Reparameterize it to obtain ,in These are the parameters obtained by sampling from a Gaussian distribution.

[0038] After projecting onto a high-dimensional sparse space, the projection is obtained. The projection is updated via momentum ( Update to the domain repository prototype :

[0039] ;

[0040] Then apply Sparse regularization Constraint Prototype Force the domain repository to retain only consistent, duplicate domain signals, where It represents the total number of domain types.

[0041] ;

[0042] Prototypes retrieved from the domain repository Activated by JumpReLU (threshold) ), thus obtaining a pure sparse domain characterization. .

[0043] In step S1(b) dense domain representation learning module, dense domain prototype learning is performed to force DOME to learn dense features. Sparse high-fidelity signals generated by decoupling from sparse labeling domain Alignment. This invention designs a series of regularization loss functions. and :

[0044] ;

[0045] ;

[0046] And add multi-positive-example InfoNCE loss. This encourages compact features within a domain and separation of features between domains. Overall, the framework of this invention achieves structured learning by jointly optimizing all loss functions.

[0047] ;

[0048] in, , , , These are the weighting coefficients.

[0049] Steps S2-S4 are the DOME-based test-time adaptive process.

[0050] This embodiment integrates the frozen DOME into the online test-time adaptive (TTA) framework (Figure 2) to dynamically generate domain condition normalization parameters in real-time step S2.

[0051] To achieve adaptive modulation, the domain statistics output by the DOME are... The input is fed into the MLP adapter to perform step S3, and the affine parameters are output. Finally, referring to step S4, the domain statistical parameters are used to modulate the features after the normalization layer in the ViT backbone network. :

[0052] ;

[0053] This operation enables instance-level adaptation of features to the input style. During inference, only the affine terms of the MLP adapter and normalization layer are updated; the DOME and other ViT weights remain frozen. Unsupervised conditional entropy loss is used during testing. and marginal entropy loss Adaptive mechanisms are employed, with marginal loss used to prevent predictions from collapsing.

[0054] ;

[0055] , These are the weighting coefficients.

[0056] This method requires no additional supervision, and the performance improvement comes entirely from the high-quality domain representations learned.

[0057] Figure 2 This paper showcases a deep learning framework combining a Domain Adaptive Visual-Language Model (DOME) with a lightweight MLP, based on a multimodal encoder-decoder architecture, for achieving high-precision image classification tasks. The overall structure consists of three main parts: the left side is an image and text joint encoding module, the middle is the core domain adaptation mechanism (DOME), and the right side is a patch-based feature extraction and classification head. In the left module, images are encoded into image sequence embeddings by an image encoder, and text is encoded into text sequence embeddings by a text encoder. After positional encoding, both are input into a multi-head attention layer for interactive fusion. High-level semantic information is then progressively extracted through residual connections, normalization, and a forward propagation network. Finally, a linear layer outputs the latent representation of the image. and It also introduces a domain memory mechanism to store the reparameterized image representation. Together with the domain label as input, a domain-aware latent space representation is generated by the domain encoder. and utilizes multiple loss functions (including , , The model is optimized to enhance cross-domain generalization ability. Simultaneously, the DOME module works in conjunction with the MLP to further improve feature representation capabilities. During testing, the adaptive module extracts local features from the input image, processes them through linear projection, normalization, multi-head attention, and MLP, and then feeds them into the classification head to generate predicted logits. The entire model effectively integrates textual context and domain prior knowledge while preserving original image information, making it suitable for cross-domain visual recognition tasks with complex distributional differences. Furthermore, the model exhibits good scalability and robustness, enabling it to adapt to transfer learning needs across different datasets.

[0058] Implementation effect

[0059] The proposed adaptive method based on explicit domain modeling was evaluated on three widely used Test-Time Adaptation (TTA) benchmark datasets: ImageNet-C, ImageNet-R, and ImageNet-Sketch. The ImageNet-C dataset, by applying 15 different types of synthetic perturbations (such as noise, blur, and weather effects) to the original ImageNet images, simulates common real-world image degradation scenarios and serves as a core benchmark for measuring model robustness. ImageNet-R contains rendered images from ImageNet categories (such as paintings, cartoons, and sculptures) and is used to evaluate the model's generalization ability under significant style transfer. ImageNet-Sketch, on the other hand, consists entirely of hand-drawn sketches, completely removing color and texture cues and retaining only the object outline structure, posing a significant challenge to the model's semantic understanding capabilities.

[0060] The experiments strictly followed the standard online TTA protocol: the model performed forward-backward propagation only once for each test batch in a streaming setting, and the overall Top-1 accuracy was reported. We compared the method proposed in this patent with six representative TTA baseline models, including: TENT (a parameter fine-tuning method based on entropy minimization), CoTTA (a continuous adaptive framework introducing a teacher-student mechanism), SAR (dynamically adjusting the learning rate through sharpness perception), ActMAD (an adaptive strategy based on activation distribution alignment), DeYO (a debiasing optimization method guided by output confidence), and FOA (a newly proposed feature orthogonal alignment method). These methods represent the cutting-edge ideas in the current TTA field in terms of loss design, memory mechanism, confidence calibration, and feature decoupling.

[0061] As shown in Table 1, this invention achieves optimal or near-optimal performance on all three benchmarks. Specifically, on ImageNet-C, this method outperforms all comparable methods with an average accuracy of 66.9%, especially excelling in highly distorted scenarios such as Pixelate (75.3%) and Zoom Blur (64.2%), and maintaining high stability under Contrast perturbation (66.8%), while CoTTA suffers severe degradation in this category (only 9.7%). On ImageNet-R, this method achieves a Top-1 accuracy of 68.3%, close to the current best, DeYO (68.7%), demonstrating its effective handling of cross-modal appearance drastic changes. On the most challenging ImageNet-Sketch, this method breaks the existing record with an accuracy of 53.7%, significantly outperforming DeYO (50.3%), fully validating that explicit domain modeling can still guide the model to focus on robust structural semantics even when color and texture cues are completely missing.

[0062] The above results demonstrate that this invention, through the high-quality explicit domain variables provided by DOME, enables the most basic entropy minimization strategy to achieve efficient and stable adaptation under diverse and extreme domain offsets, without the need for complex architectures or additional supervision, exhibiting excellent versatility and practicality.

[0063] Table 1

[0064]

[0065] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any transformations or substitutions that can be conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A test-time adaptive method based on sparse supervised explicit modeling of domain variables, characterized in that, Includes the following steps: Step S1: Train the domain encoder DOME using a sparse-to-dense learning framework. This framework contains two collaborative components: (a) a sparse label domain decoupling module, which separates domain-specific factors that are independent of category semantics from the input samples; and (b) a dense domain representation learning module, which extrapolates sparse discrete domain labels into a continuous, multi-dimensional domain distribution space to achieve fine-grained domain modeling for any sample. Step S2: During the inference deployment phase, unlabeled test samples are input into the trained DOME encoder, and zero-sample explicit domain variables are generated. Step S3: Inject the generated display domain variables into the downstream backbone model through a lightweight MLP adapter, fine-tune only the adapter and normalization layer, and adapt during testing in combination with the self-entropy minimization objective; Step S4: Dynamically adjust the feature representation using the guided model with injected explicit domain clues to achieve robust adaptation to the unknown target domain and output the final prediction result.

2. The test-time adaptive method based on sparse supervised explicit modeling of domain variables according to claim 1, characterized in that, Explicit domain variables are presented as distributions with mean and variance parameterized.

3. The test-time adaptive method based on sparse supervised explicit modeling of domain variables according to claim 1, characterized in that, The downstream backbone model adopts ViT.

4. The test-time adaptive method based on sparse supervised explicit modeling of domain variables according to claim 1, characterized in that, In step S1(a) sparse labeling neighborhood decoupling module, firstly for the neighborhood encoder DOME, image sequence features are fused. Domain label text sequence features Capture the stylistic features of the domain: ； fusion characteristics of DOME output After passing through multiple transformers, including attention, residual, normalization, and linear layers, the channel statistics are represented as a compact representation including the mean. and standard deviation To capture explicit domain variables: ； The encoded tokens extracted from the CLIP visual encoder, after removing positional encodings, are used to calculate the original style statistics through channel statistics and then reparameterized into latent vectors. : ； The parameters obtained from sparse label domain decoupling include the mean. and standard deviation Reparameterize it to obtain ,in These are parameters obtained by sampling from a Gaussian distribution; After projecting onto a high-dimensional sparse space, the projection is obtained. This projection is updated to the domain repository prototype via momentum update. : ； Then apply Sparse regularization Constraint Prototype ,in It represents the total number of domain types; ； Prototypes retrieved from the domain repository After JumpReLU activation, a pure sparse neighborhood representation is obtained. .

5. The test-time adaptive method based on sparse supervised explicit modeling of domain variables according to claim 4, characterized in that, In step S1(b) dense domain representation learning module, dense domain prototype learning is performed, forcing DOME learning of dense features. Sparse high-fidelity signals generated by decoupling from sparse labeling domain Alignment is achieved using the following regularization loss function. and : ； ； And add multi-positive-example InfoNCE loss. This invention encourages compact features within a domain and separation of features between domains. Overall, the framework of this invention achieves structured learning by jointly optimizing all loss functions. ； in, , , , These are the weighting coefficients.

6. The test-time adaptive method based on sparse supervised explicit modeling of domain variables according to claim 5, characterized in that, Domain statistics output by DOME Input to the MLP adapter, output affine parameters Finally, referring to step S4, the domain statistical parameters are used to modulate the ViT backbone network to obtain the features after the normalization layer. : 。 7. The test-time adaptive method based on sparse supervised explicit modeling of domain variables according to claim 6, characterized in that, Use unsupervised conditional entropy loss during testing. and marginal entropy loss Perform adaptive behavior: ； , These are the weighting coefficients.

8. A test-time adaptive device based on sparse supervised explicit modeling of domain variables, characterized in that, Includes the following modules: The modeling module employs a sparse-to-dense learning framework to train the domain encoder DOME, which includes two collaborative components: (a) a sparse label domain decoupling module, used to separate domain-specific factors that are independent of category semantics from input samples; and (b) a dense domain representation learning module, which extrapolates sparse discrete domain labels into a continuous, multi-dimensional domain distribution space, enabling fine-grained domain modeling for any sample. In the inference deployment module, during the inference deployment phase, unlabeled test samples are input into the pre-trained DOME encoder, and zero samples are used to generate the corresponding explicit domain variables. The adaptive module injects the generated display domain variables into the downstream backbone model through a lightweight MLP adapter, only fine-tuning the adapter and normalization layer, and adapts during testing in combination with the self-entropy minimization objective; The output module dynamically adjusts the feature representation using the guided model guided by the injected explicit domain clues, achieving robust adaptation to the unknown target domain and outputting the final prediction result.

9. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1 to 7.

Images