Progressive spectral enhancement large model unsupervised domain adaptation method

Abstract

The application discloses a progressive spectrum enhancement large model passive domain self-adaptive method, and has the characteristics that a progressive cross-modal double-blind consensus screening is designed to cross-verify heterogeneous features of a visual source model and a visual-language large model, high-purity target domain consensus samples are dynamically extracted, low-frequency components of the samples are extracted to drive dynamic prompt generation, feature space topology regularization is supplemented, a specialized teacher model is constructed, high-frequency feature disturbance is applied to target domain images, student models are forced to focus on domain-invariant features, cross-modal knowledge distillation is carried out in combination with soft labels of the specialized teacher model, and self-adaption is completed through closed-loop iteration. The application aims to solve the problems of pseudo-label deviation in existing passive domain adaptation, lack of target perception of a large model and insufficient anti-interference, and through double-blind consensus, low-frequency prior specialization and high-frequency anti-interference distillation, the recognition precision and robustness in a shift environment are enhanced.

Description

Technical Field

[0001] This invention belongs to the field of machine learning and pattern recognition technology, specifically relating to a progressively spectrally enhanced passive domain adaptive method for large models. Background Technology

[0002] Deep learning has achieved tremendous success in computer vision, but this often relies on the premise that training and test data are independently and identically distributed. In real-world open-world applications, models are often deployed to new scenes with different distributions than the training environment. This "domain shift" leads to a sharp decline in model performance. To address this issue, Unsupervised Domain Adaptation (UDA) is developed. [1] Technology emerged as a result. (Long et al.) [2] A conditional adversarial domain adaptation method was proposed, which mainly utilizes a conditional discriminator to align the feature distributions of the source and target domains; He et al. [3] An unsupervised domain adaptation method based on difficult sample partitioning and processing is proposed, which effectively alleviates the negative transfer problem through fine-grained separation and adaptive weighting. However, traditional UDA methods require simultaneous access to data in both the source and target domains. With increasingly stringent data privacy regulations (such as GDPR) and rising costs for massive data transmission and storage, source domain data is often unavailable. Therefore, source-free domain adaptation (SFDA) has gradually become a research hotspot and focus in the fields of artificial intelligence and machine vision in recent years. [4][5] The SFDA provides only a pre-trained model in the source domain and unlabeled target domain data, requiring the target domain to be adapted without accessing any source data.

[0003] Currently, mainstream passive domain adaptation methods mainly rely on self-training and information maximization (such as SHOT). [6] The core idea is to use a pre-trained source model to generate pseudo-labels for unlabeled target domain samples, and then fine-tune the network parameters through cross-entropy or contrastive learning. (Xu et al.) [7] A pseudo-label screening mechanism based on uncertainty control was proposed to remove high-risk samples; Xia et al. [8] They proposed a discriminative pattern calibration mechanism to align features. Tang et al. [9]A proxy denoising network was designed to refine high-confidence samples by building a prototype; however, this type of method has a fatal flaw: due to the significant distributional difference between the source and target domains, the source model inevitably generates a large number of erroneous and high-confidence pseudo-labels in the target domain. According to Mitsuzumi et al.

[10] Theoretical analysis shows that when a model iterates on these pseudo-labels with inherent structural biases, it is highly susceptible to a vicious cycle of "confirmation bias," regardless of the confidence threshold or uncertainty filtering method used. Furthermore, Zhu et al.

[11] Recent theoretical insights also point out that this over-reliance on a single-source model's pre-defined feature space severely undermines the representativeness and diversity of the target domain samples during the adaptation process. With these two theoretical flaws combined, the model will ultimately overfit repeatedly at incorrect classification boundaries, severely limiting its generalization ability.

[0004] To overcome the cognitive limitations of single-source models, recent research has begun to introduce pre-trained large-scale vision-language models (VLMs, such as CLIP).

[12] As external prior knowledge, it assists in passive domain adaptation. Current SFDA methods combining VLM mainly focus on multi-view pseudo-label fusion, cross-modal knowledge interaction, and prompting engineering. For example, Tang et al.

[13] This study explored using the model's general priors to handle passive domain tasks by fully freezing the VLM; Zhan et al.

[14] A dual transferable knowledge interaction mechanism was proposed to establish knowledge communication between the source visual model and the VLM to calibrate features; Lee et al.

[15] The DUET framework was proposed, which refines the adaptive signal in the target domain by introducing dual-perspective pseudo-labeling and an uncertainty-aware exploration and utilization strategy. In addition, some studies have explored dynamic-prompting collaboration mechanisms.

[16] This approach attempts to alleviate the semantic rigidity problem caused by traditional static text templates. However, despite the introduction of VLM and the extensive explorations described above, existing methods still face two significant challenges.

[0005] First, simple fusion of visual and linguistic priors easily introduces new noise and lacks a rigorous consensus mechanism. Existing methods often use simple soft label fusion or confidence weighting to combine the predictions of the source model and CLIP. However, the source model is a "visual structure expert" of the target domain, while CLIP is a "global semantic expert," and their feature spaces are orthogonal. Simple soft fusion often masks the true category of high-difficulty samples and fails to establish a rigorous screening mechanism to obtain truly reliable anchor samples.

[0006] Secondly, VLM lacks the ability to perceive the specific style of the target domain. While the native CLIP is powerful, its text prompts are often static and lack the ability to perceive the specific environmental style of the target domain. If the native CLIP is used directly to guide the target model, domain misalignment is likely to occur; while if the CLIP is fine-tuned directly on the target domain, it is very easy to destroy its originally well-generalized cross-modal semantic space structure, leading to catastrophic forgetting.

[14] .

[0007] Furthermore, in terms of representation learning in the target domain feature space, relying solely on pseudo-labels is often insufficient to cope with extreme environmental interference. Recent studies have indicated that data augmentation and consistency regularization of target domain samples can improve the robustness of the model.

[17] However, existing data perturbation techniques (such as global noise addition or simple spatial transformation) often destroy the semantic structure of the image core. The frequency domain decoupling theory of images shows...

[18] Low-frequency components typically contain domain-specific lighting, color, and style information, while high-frequency components contain semantic boundary information closely related to classification. Existing methods fail to fully utilize the frequency domain decoupling mechanism at the global original image level to implement targeted high-frequency anti-interference training, making it difficult for the model to learn truly discriminative domain-invariant features when faced with complex shifts in the target domain.

[0008] In summary, existing passive domain adaptation methods generally face technical bottlenecks when dealing with severe domain shifts, including significant pseudo-label confirmation bias, lack of domain-aware specialization in visual-linguistic knowledge transfer, and insufficient robustness against interference in the feature space. Therefore, a novel framework is urgently needed that can rigorously coordinate traditional visual models with VLMs, achieve domain-driven specialization of large models while ensuring absolute purity of pseudo-labels, and enhance the model's robustness against interference by utilizing global frequency domain decoupling techniques, ultimately achieving efficient and robust passive domain adaptation.

[0009] [1] Ganin, Yaroslav, et al. "Domain-adversarial training of neuralnetworks." Journal of machine learning research 17.59 (2016): 1-35.

[0010] [2] Long, Mingsheng, et al. "Conditional adversarial domainadaptation." Advances in neural information processing systems 31 (2018).

[0011] [3] He, Chunmei, et al. "Unsupervised domain adaptation with hard-sample dividing and processing strategy." Information Sciences 680 (2024):121152.

[0012] [4] Li, Jingjing, et al. "A comprehensive survey on source-freedomain adaptation." IEEE Transactions on Pattern Analysis and MachineIntelligence 46.8 (2024): 5743-5762.

[0013] [5] Fang, Yuqi, et al. "Source-free unsupervised domain adaptation: Asurvey." Neural Networks 174 (2024): 106230.

[0014] [6] Liang, Jian, Dapeng Hu, and Jiashi Feng. "Do we really need toaccess the source data? source hypothesis transfer for unsupervised domainadaptation." International conference on machine learning. PMLR, 2020.

[0015] [7] Xu, Gezheng, et al. "Revisiting source-free domain adaptation: Anew perspective via uncertainty control." The Thirteenth InternationalConference on Learning Representations. 2024.

[0016] [8] Xia, Haifeng, Siyu Xia, and Zhengming Ding. "Discriminativepattern calibration mechanism for source-free domain adaptation." Proceedingsof the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024.

[0017] [9] Tang, Song, Wenxin Su, Mao Ye, and Xiatian Zhu. "Proxy denoisingfor source-free domain adaptation." International Conference on LearningRepresentations. 2025.

[0018]

[10] Mitsuzumi, Yu, Akisato Kimura, and Hisashi Kashima. "Understanding and improving source-free domain adaptation from a theoreticalperspective." Proceedings of the IEEE / CVF conference on computer vision andpattern recognition. 2024.

[0019]

[11] Zhu, Ronghang, et al. "Revisiting source-free domain adaptation:Insights into representativeness, generalization, and variety." Proceedingsof the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2025.

[0020]

[12] Radford, Alec, et al. "Learning transferable visual models fromnatural language supervision." International conference on machine learning.PmLR, 2021.

[0021]

[13] Tang, Song, et al. "Source-free domain adaptation with frozenmultimodal foundation model." Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition. 2024.

[0022]

[14] Zhan, Mengmeng, et al. "Dual transferable knowledge interactionfor source-free domain adaptation." Information Processing & Management 63.1(2026): 104302.

[0023]

[15] Lee, Jae Yun, et al. "DUET: Dual-Perspective Pseudo Labeling andUncertainty-aware Exploration & Exploitation Training for Source-Free DomainAdaptation." The Thirty-ninth Annual Conference on Neural InformationProcessing Systems. 2025.

[0024]

[16] Zhan, Mengmeng, et al. "Towards Dynamic-Prompting Collaborationfor Source-Free Domain Adaptation." IJCAI. 2024.

[0025]

[17] Wang, Jing, et al. "Vicinal Gaussian Transform: RethinkingSource-Free Domain Adaptation through Source-Informed Label Consistency."IEEE Transactions on Pattern Analysis and Machine Intelligence (2025).

[0026]

[18] Huang, Yuyang, Yabo Chen, Junyu Zhou, Wenrui Dai, Xiaopeng Zhang, Junni Zou, Hongkai Xiong, and Qi Tian. "Diffusion-Driven ProgressiveTarget Manipulation for Source-Free Domain Adaptation." NeurIPS. 2025. Summary of the Invention

[0027] The purpose of this invention is to address the problems of severe false label confirmation bias, lack of target domain perception capability in existing passive domain adaptation methods, and insufficient anti-interference capability of the feature space, which restricts the model's generalization ability. This invention proposes a progressively enhanced spectrum-enhanced large-model passive domain adaptation method. This method extracts high-purity anchor point samples by constructing a rigorous double-blind consensus mechanism that dynamically evolves with the model's capabilities, dynamically specializes the VLM using low-frequency visual priors in the target domain, and combines high-frequency perturbations in the frequency domain with cross-modal distillation to achieve efficient and robust adaptation of the source model in the target domain through closed-loop iteration.

[0028] The technical solution adopted in this invention is

[0029] A progressively spectrally enhanced large-model passive domain adaptation method is characterized by the following steps: First, a progressive cross-modal double-blind consensus mechanism is constructed, utilizing rigorous cross-validation between the visual source model and the orthogonal feature space of the frozen VLM (Virtual Model Library). High-purity target domain consensus samples are dynamically extracted as the model's capabilities improve. Second, low-frequency visual priors from the consensus samples are extracted to drive a dynamic cue generation network to generate target domain-aware textual cues for the VLM. Feature space topology regularization is then applied to construct a semantically structure-preserving specialized teacher model. Finally, high-frequency perturbations in the frequency domain are applied to the target domain image, forcing the student model to focus on domain-invariant semantic boundary features. This is combined with soft labels from the specialized teacher model for cross-modal knowledge distillation, achieving robust adaptation of the target domain through multiple rounds of closed-loop iterations.

[0030] The beneficial effects of this invention are

[0031] 1. This invention overcomes the "confirmation bias" bottleneck of traditional pseudo-labels. Compared to existing soft label fusion strategies, this invention innovatively introduces a "double-blind consensus" mechanism, rigorously utilizing the orthogonality of visual local structure experts (source models) and global semantic experts (VLMs) for cross-validation. This extremely stringent hard-voting screening mechanism can eliminate highly difficult confusing samples, obtaining absolutely pure pseudo-labels, laying a solid and correct optimization foundation for subsequent domain adaptation.

[0032] 2. Achieved "non-destructive" domain-aware specialization of VLM. This invention avoids the catastrophic forgetting caused by directly fine-tuning VLM and overcomes the semantic rigidity of static cues. By extracting low-frequency components (lighting, style information) of the target domain image to guide the generation of dynamic cues, and supplemented by topological regularization, VLM can keenly capture the specific environmental features of the target domain, achieving a seamless transformation from a "general practitioner" to a "target domain specialist".

[0033] 3. Significantly enhances the target model's resistance to environmental interference and its generalization ability. This invention introduces the concept of frequency domain decoupling at the global original image level, constructing consistency constraints through targeted high-frequency perturbations. This forces the model to no longer rely on volatile backgrounds and colors during the adaptation process, but instead learns truly core, domain-invariant semantic boundary features; combined with cross-modal soft label distillation of specialized large models, the final trained model exhibits excellent robustness when facing extreme domain shifts.

[0034] 4. It breaks through the limitations of traditional passive domain adaptive single-time static pseudo-label filtering, enabling the double-blind consensus mechanism to dynamically evolve as the model's capabilities improve. In the initial adaptive phase, the most stringent consensus threshold is used to lock in the absolutely correct optimization direction, preventing early noise from causing model collapse. As the capabilities of teacher and student models in the target domain are collaboratively enhanced, the consensus boundary gradually expands towards difficult samples. This dynamic iterative closed loop, "from easy to difficult," not only ensures the extreme stability of the knowledge transfer process but also deeply explores the distribution characteristics of complex samples, approaching the true performance upper limit of the target domain. Attached Figure Description

[0035] Figure 1 A general framework diagram of the method of this invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0037] The core of this invention lies in constructing a progressive spectrum enhancement large model passive domain adaptive method framework, which mainly includes three stages: progressive cross-modal double-blind consensus screening, large model dynamic specialization based on low-frequency priors, and frequency domain high-frequency anti-interference optimization and cross-modal distillation.

[0038] (1) First, a progressive cross-modal double-blind consensus screening is performed. For the input unlabeled target domain image, it is input into the pre-trained source visual model and the frozen VLM respectively. Since the feature spaces of the two are orthogonal to each other, we use them to independently generate the predicted probability distribution of the target domain samples. In order to eliminate the "confirmation bias" of single-modal pseudo-labels, we designed a strict double-blind hard voting mechanism: only when the prediction category of the source model and VLM for the same image is completely consistent, and the prediction confidence of both is higher than the preset stringent threshold, is the sample accepted as a reliable "consensus sample". In particular, the screening process adopts a progressive evolution strategy: in the early stage of adaptive iteration, an extremely high confidence threshold is set to strictly screen simple samples; as multiple rounds of adaptive training are advanced and the model's feature extraction capability is enhanced, the confidence threshold is dynamically and smoothly reduced, and marginal difficult samples are gradually included in the consensus sample pool, thereby maximizing the utilization rate of target domain data while ensuring the absolute purity of pseudo-labels.

[0039] (2) Secondly, dynamic specialization of the large model based on low-frequency priors is implemented. For the target domain consensus samples selected above, we first perform global frequency domain decoupling to extract low-frequency components containing specific lighting, color, and environmental style features of the target domain. This low-frequency visual feature is used as a priori clue and input into a learnable dynamic prompt generation network to generate dynamic text prompts for the VLM that incorporate target domain environmental perception. In this way, the text feature space of the VLM is specialized by the target domain style. At the same time, in order to avoid the large model from destroying its original well-generalized cross-modal semantic manifold structure during the specialization fine-tuning process, we introduce a feature space topology regularization mechanism to constrain the relative geometric distance and topological relationship of semantic vectors of various categories in space before and after specialization to remain consistent, and finally construct a "specialized teacher model" that has both general semantic priors and target domain perception capabilities.

[0040] (3) Finally, high-frequency anti-interference optimization and cross-modal distillation are performed in the frequency domain. For the target domain image, we extract its high-frequency components containing core semantic boundaries and shape information, and apply feature perturbation in the frequency domain to construct an enhanced view with extremely high environmental challenges. These samples subjected to high-frequency perturbation are input into the source vision model, forcing the student model to ignore the variable background noise of the target domain, actively learn and focus on the domain-invariant features that are truly discriminative. At the same time, the specialized teacher model constructed in the previous step is used to reason about the original target domain image, output high-quality soft labels, and use this to perform cross-modal knowledge distillation on the source vision model. The network parameters of the source vision model are updated by jointly optimizing the frequency domain consistency loss and the cross-modal distillation loss. The above three steps of "consensus screening - teacher specialization - student distillation" constitute a complete progressive closed loop. The teacher and student models co-evolve in multiple rounds of interactive iteration, directly

[0041] The network converges, ultimately achieving efficient and robust adaptation of the target domain.

Claims

1. A progressively spectrally enhanced large-model passive domain adaptive method, characterized in that... The design employs a progressive cross-modal double-blind consensus screening mechanism. For input unlabeled target domain images, they are fed into a pre-trained source visual model and a frozen visual-language large model, respectively. The heterogeneous feature spaces of both models are used to independently generate the predicted probability distribution of target domain samples. A strict double-blind hard-voting mechanism is established, accepting a sample as a reliable consensus sample only when the source visual model and the visual-language large model predict the same category for the same image, and both prediction confidence levels are higher than a preset confidence threshold. A progressive evolution strategy is adopted during the adaptive iteration process. Initially, an extremely high confidence threshold is set to rigorously screen simple samples. As multiple rounds of adaptive training progress and the model's feature extraction capabilities improve, the confidence threshold is dynamically and smoothly reduced, gradually incorporating marginal and difficult samples into the consensus sample pool.

2. A progressively spectral-enhanced large-model passive-domain adaptive method, characterized in that... A dynamic specialization of the large model based on low-frequency priors is proposed. For the target domain consensus samples selected above, the frequency domain is decoupled globally to extract low-frequency components containing the style features of the target domain's specific environment. This low-frequency visual feature is used as a priori cue and input into the dynamic prompt generation network to generate dynamic text prompts that integrate target domain environment perception for the visual-language large model, thereby utilizing the text feature space of the target domain style specialization large model. At the same time, a feature space topology regularization mechanism is introduced to constrain the relative geometric distance and topological relationship of semantic vectors of various categories in space before and after specialization to remain consistent. Finally, while maintaining the integrity of the original cross-modal representation of the large model, a specialization teacher model with both general semantic priors and target domain perception capabilities is constructed.

3. A progressively spectral-enhanced large-model passive-domain adaptive method, characterized in that... Construct high-frequency anti-interference optimization and cross-modal distillation in the frequency domain; for the target domain image, extract its high-frequency components containing core semantic boundaries and shape information, and apply feature perturbation in the frequency domain to construct an enhanced view; input the samples subjected to high-frequency perturbation into the source visual model as the student model, forcing the student model to ignore the variable background noise of the target domain and focus on the discriminative domain-invariant features; Simultaneously, the constructed specialized teacher model is used to infer the original target domain image, outputting high-quality soft labels, which are then used to perform cross-modal knowledge distillation on the source visual model. By jointly optimizing the frequency domain consistency loss and the cross-modal distillation loss, the network parameters of the source visual model are updated. The above consensus screening, teacher specialization, and student distillation constitute a progressive closed-loop optimization process. The teacher and student models co-evolve in multiple rounds of interactive iterations until the network converges and completes the adaptation of the target domain.

Images