Plug-and-play feature perception dynamic gating vision enhancement system
By designing a plug-and-play feature-aware dynamic gating vision enhancement system, the system addresses the issues of insufficient domain adaptability and integration complexity in existing remote sensing and low-altitude UAV image processing technologies. It achieves accurate adaptive feature enhancement and low-cost model adaptation, making it suitable for UAV-borne or edge computing platforms.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHONGKE XINGTU DIGITAL EARTH HEFEI CO LTD
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-17
AI Technical Summary
Existing visual enhancement technologies lack plug-and-play features, have complex and invasive integration processes, and cannot effectively adapt to remote sensing and low-altitude UAV image processing. They also suffer from insufficient domain adaptability, differences in feature distribution, architectural differences, low computational efficiency, and multi-task conflicts.
Design a plug-and-play feature-aware dynamic gating visual enhancement system, including a pre-trained model backbone network, an image encoder, a multimodal encoder, a multi-scale feature extraction layer module, a multimodal feature extraction layer module, and a plug-and-play feature enhancement module FADGM. Enhancement weights are dynamically generated through channel and spatial gating paths to achieve accurate adaptive feature enhancement and support unified adaptation of heterogeneous models.
It significantly improves the model's accuracy in perceiving and understanding professional ground features, reduces computing and storage costs, supports UAV-borne or edge computing platforms, alleviates multi-task conflicts, and has plug-and-play characteristics.
Smart Images

Figure CN121884073A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and deep learning technology, and more specifically, to a plug-and-play feature-aware dynamic gating vision enhancement system. Background Technology
[0002] With the rapid development of deep learning technology, pre-trained large-scale visual models such as DINOv3 and Qwen3-VL have demonstrated powerful feature representation capabilities in the field of computer vision. However, in the specific field of remote sensing and low-altitude UAV image processing, directly applying these general-purpose pre-trained models still faces severe technical challenges.
[0003] Currently, remote sensing and low-altitude UAV image processing technologies based on pre-trained large-scale visual models face a series of technical bottlenecks. First, there are significant differences in feature distribution between general-purpose pre-trained models and specialized remote sensing domains. The feature representations learned by models such as DINOv3 and Qwen3-VL on natural images are difficult to directly adapt to the unique ground cover characteristics and imaging conditions of remote sensing images, resulting in limited performance in specialized tasks. This insufficient domain adaptability severely restricts the effective application of advanced large-scale visual models in the remote sensing field. Second, existing multimodal models lack specialized knowledge encoding in the remote sensing domain, making it unable to accurately understand and process specialized terminology and ground cover descriptions, resulting in poor performance in tasks requiring specialized knowledge, such as visual question answering and index expression segmentation. The models also exhibit biases in their understanding of spatial relationships and specialized concepts unique to remote sensing, making it difficult to meet the demands for accurate understanding and reasoning in practical applications. Third, the architectural differences between different pre-trained models pose a significant challenge to feature enhancement. The ViT architecture of DINOv3 and the multimodal architecture of Qwen3-VL differ fundamentally in feature representation and processing procedures. Existing enhancement methods cannot achieve unified adaptation across architectures, leading to high integration costs and low efficiency. Fourth, existing feature enhancement methods have significant shortcomings in fine-grained feature selectivity. Traditional attention mechanisms struggle to accurately focus on key feature regions of specific ground features, resulting in a disconnect between the enhancement process and downstream task objectives. This prevents task-oriented, precise feature enhancement, particularly when dealing with complex remote sensing ground features. Fifth, the balance between computational efficiency and performance is a significant challenge. In resource-constrained scenarios such as UAV edge computing, traditional enhancement methods often introduce excessive computational overhead, failing to meet real-time processing requirements. Parameter inflation and inference latency severely restrict the deployment and application of these technologies in real-world scenarios. Sixth, feature conflicts and catastrophic forgetting in multi-task learning urgently need to be addressed. When continuously learning multiple remote sensing tasks, existing methods struggle to avoid mutual interference between tasks. Learning new tasks can disrupt already acquired knowledge representations, leading to a decline in overall model performance.
[0004] Therefore, existing visual enhancement technologies lack true plug-and-play functionality, and the integration process is complex and highly invasive. Users need to perform extensive customized development and optimization for different models, resulting in high technical barriers and maintenance costs, which seriously affects the popularization and application efficiency of advanced technologies. Summary of the Invention
[0005] This invention aims to address the problem that existing visual enhancement technologies lack true plug-and-play characteristics and have complex and invasive integration processes.
[0006] To address the aforementioned issues, this invention provides a plug-and-play feature-aware dynamic gating vision enhancement system, comprising a pre-trained model backbone network, an image encoder, a multimodal encoder, a multi-scale feature extraction layer module, a multimodal feature extraction layer module, a plug-and-play feature enhancement module FADGM, and a task output head; The pre-trained model backbone network is connected to the multi-scale feature extraction layer module via an image encoder. The pre-trained model backbone network communicates with the multimodal feature extraction layer module through a multimodal encoder; Both the multi-scale feature extraction layer module and the multi-modal feature extraction layer module are communicatively connected to the input of the plug-and-play feature enhancement module FADGM. The output of the Plug and Play Feature Enhancement Module (FADGM) is connected to the task output header for communication. The plug-and-play feature enhancement module FADGM includes an input feature normalization unit, a core enhancement engine unit, an adapter component unit, and an output feature standardization unit; the input feature normalization unit, the core enhancement engine unit, the adapter component unit, and the output feature standardization unit are connected in sequence. The core enhancement engine unit includes a channel gated path unit, a spatial gated path unit, and a residual fusion unit; the channel gated path unit, the spatial gated path unit, and the residual fusion unit are connected in sequence.
[0007] The present invention provides a plug-and-play feature-aware dynamic gating vision enhancement system, which, compared with the prior art, has, but is not limited to, the following beneficial effects: The pre-trained model backbone network described in this invention is used to select and load the corresponding pre-trained model according to the input data type. The multi-scale feature extraction layer module is used to extract and organize visual features at different levels from models based on the ViT architecture, such as DINOv3. The plug-and-play feature enhancement module FADGM dynamically generates enhancement weights related to the input feature content through parallel channel-gated paths and spatial-gated paths, and fuses them with the original features in a residual manner to achieve accurate and adaptive feature enhancement.
[0008] This invention has strong domain adaptability: through the FADGM module, it effectively bridges the feature distribution differences between general pre-trained models and remote sensing / UAV imagery, significantly improving the model's accuracy in perceiving and understanding specialized ground features.
[0009] This invention features outstanding model versatility: through a design of a general core and optional adapters, it achieves for the first time unified support for heterogeneous models such as DINOv3 (ViT architecture) and Qwen3-VL (multimodal architecture) with a single enhancement module, possessing true plug-and-play characteristics.
[0010] This invention features high computational performance: the number of module parameters is extremely small, typically increasing the overall model parameter count by less than 1%, and the inference latency is extremely low. It supports a fine-tuning paradigm of freezing the backbone network and training only enhancement modules, significantly reducing computation and storage costs. It is suitable for UAV-borne or edge computing platforms.
[0011] This invention features precise feature selectivity enhancement: the dual-path dynamic gating mechanism can simultaneously and adaptively focus on the feature components most relevant to the task from both channel and spatial dimensions, effectively enhancing the representation of fine-grained ground property attributes.
[0012] This invention alleviates multi-task conflicts: by introducing independent gating weight initialization or adding orthogonal constraint loss with gating parameters for different tasks, feature conflicts and catastrophic forgetting problems can be effectively reduced when continuously learning multiple remote sensing tasks.
[0013] Furthermore, the pre-trained model backbone network includes an input source unit, a model selection unit, and a feature extraction unit; the input source unit is communicatively connected to the input end of the model selection unit, and the output end of the model selection unit is communicatively connected to the feature extraction unit.
[0014] Furthermore, the image encoder includes an image input module, an architecture module, and a multi-layer feature output module; the image input module, the architecture module, and the multi-layer feature output module are sequentially connected in communication.
[0015] Furthermore, the multimodal encoder includes an image text input module, an architecture and language model module, and a multi-scale visual feature and text embedding output module; the image text input module, the architecture and language model module, and the multi-scale visual feature and text embedding output module are sequentially connected in communication.
[0016] Furthermore, the multi-scale feature extraction layer module includes a shallow detail feature unit, a mid-level structural feature unit, and a deep semantic feature unit; the shallow detail feature unit, the mid-level structural feature unit, and the deep semantic feature unit are sequentially connected in communication.
[0017] Furthermore, the multimodal feature extraction layer module includes a visual feature projection unit, a text-guided attention unit, and a feature dimension alignment unit; the visual feature projection unit, the text-guided attention unit, and the feature dimension alignment unit are sequentially connected in communication.
[0018] Furthermore, the adapter component unit includes a DINOv3 adapter and a Qwen3-VL adapter.
[0019] Furthermore, the task output header includes a task-specific output header and a multimodal task output header.
[0020] Furthermore, the task-specific output header includes a classifier, a segmenter, and a detector; the classifier, segmenter, and detector are sequentially connected in communication.
[0021] Furthermore, the multimodal task output head includes a VQA answer generation unit, a pointer representation segmentation unit, and an image description generation unit; the VQA answer generation unit, the pointer representation segmentation unit, and the image description generation unit are sequentially connected in communication. Attached Figure Description
[0022] Figure 1 This is a system architecture diagram of the plug-and-play feature-aware dynamic gating vision enhancement system according to an embodiment of the present invention. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings showing multiple embodiments according to this application. It should be understood that the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments described in this application without creative effort will fall within the scope of protection of this application.
[0024] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used in the description of this application is for the purpose of describing specific embodiments only and is not intended to limit this application; the terms "comprising," "including," "having," "containing," etc., in the description, claims, and accompanying drawings of this application are open-ended terms. Therefore, "comprising," "including," or "having" refers to, for example, a method or apparatus having one or more steps or elements, but is not limited to having only these one or more elements. The terms "first," "second," etc., in the description, claims, or accompanying drawings of this application are used to distinguish different objects, not to describe a specific order or hierarchy. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0025] In the description of this invention, it should be understood that the terms "upper", "lower", "left", "right", "front", "rear", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0026] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "attachment" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.
[0027] It should be emphasized that when the term "comprising / including" is used in this specification, it is used to explicitly indicate the presence of the stated feature, integer, step, or component, but does not exclude the presence or addition of one or more other features, integers, steps, components, or groups of features, integers, steps, or components.
[0028] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0029] See Figure 1 An embodiment of the present invention provides a plug-and-play feature-aware dynamic gating visual enhancement system, comprising a pre-trained model backbone network, an image encoder, a multimodal encoder, a multi-scale feature extraction layer module, a multimodal feature extraction layer module, a plug-and-play feature enhancement module FADGM, and a task output head; The pre-trained model backbone network is connected to the multi-scale feature extraction layer module via an image encoder. The pre-trained model backbone network communicates with the multimodal feature extraction layer module through a multimodal encoder; Both the multi-scale feature extraction layer module and the multi-modal feature extraction layer module are communicatively connected to the input of the plug-and-play feature enhancement module FADGM. The output of the Plug and Play Feature Enhancement Module (FADGM) is connected to the task output header for communication. The plug-and-play feature enhancement module FADGM includes an input feature normalization unit, a core enhancement engine unit, an adapter component unit, and an output feature standardization unit; the input feature normalization unit, the core enhancement engine unit, the adapter component unit, and the output feature standardization unit are connected in sequence. The core enhancement engine unit includes a channel gated path unit, a spatial gated path unit, and a residual fusion unit; the channel gated path unit, the spatial gated path unit, and the residual fusion unit are connected in sequence.
[0030] The pre-trained model backbone network of this invention is used to select and load corresponding pre-trained models according to the input data type. These pre-trained models include, but are not limited to, DINOv3 and Qwen3-VL. The pre-trained model backbone network is communicatively connected to the multi-scale feature extraction layer module through the image encoder for processing purely visual input; the pre-trained model backbone network is also communicatively connected to the multi-modal feature extraction layer module through the multi-modal encoder for processing image-text pair input.
[0031] The multi-scale feature extraction layer module is used to extract and organize visual features at different levels from ViT-based models such as DINOv3, forming a feature pyramid containing details, structure, and semantic information. The multimodal feature extraction layer module is used to separate and process visual features from text embeddings in multimodal models such as Qwen3-VL, and to guide the visual features through text.
[0032] The plug-and-play feature enhancement module FADGM is the core of this invention. Its input is communicatively connected to the output of the multi-scale feature extraction layer module and the multi-modal feature extraction layer module, respectively. This module includes a general core enhancement engine and a set of optional model-specific adapter components. Its core engine dynamically generates enhancement weights related to the input feature content through parallel channel-gated paths and spatial-gated paths, and fuses them with the original features in a residual manner to achieve accurate and adaptive feature enhancement.
[0033] The task output head is communicatively connected to the output of the Plug and Play Feature Enhancement Module (FADGM).
[0034] The key innovation of this invention lies in the design of the Plug and Play Feature Enhancement Module (FADGM), the technical details of which include: Input Feature Normalization Unit: Performs layer-level or batch normalization on the input features to ensure numerical stability. The core enhancement engine internally contains parallel channel-gated path units, spatial-gated path units, and residual fusion units. The channel-gated path generates channel weight vectors using a GAP-MLP structure; the spatial-gated path generates a spatial weight matrix using a convolution-normalization-activation structure. The residual fusion unit sums the original features with the two weighted features.
[0035] The Adapter Component Unit is a pluggable component library containing lightweight adapters designed for different pre-trained models, such as the DINOv3 adapter and the Qwen3-VL adapter. The system automatically loads the appropriate adapter based on the type of model currently running. The adapters do not alter the core engine's basic data flow; instead, they optimize the augmentation process by injecting additional computations such as token-level gating, cross-modal attention, or adjusting core engine parameters such as the weights of conditional MLPs.
[0036] The output feature normalization unit performs a simple normalization process on the enhanced features again, and then outputs them.
[0037] The core enhancement engine unit adopts a dual-path parallel structure.
[0038] Channel gating path: global channel statistics are captured through global average pooling, then channel dependencies are learned through a lightweight multilayer perceptron (MLP) bottleneck layer, and finally, the enhanced weights of each channel are generated through the sigmoid function.
[0039] Spatial gating paths use small-scale convolutions, such as 3x3, to capture local spatial context. Combined with layer normalization and the GELU activation function, they generate spatial attention maps that highlight important spatial regions.
[0040] The residual fusion unit performs a weighted summation of the original input features and the features that have undergone channel weighting and spatial weighting. The fusion weights α and β for the channel and spatial augmented features are learnable scalar parameters, enabling adaptive adjustment of the augmentation intensity.
[0041] To achieve true plug-and-play functionality, the adapter component unit is designed with lightweight adapters for different pre-trained model architectures.
[0042] Dynamic and perceptual characteristics: All gating weights of the module are dynamically generated based on the current input features, rather than fixed parameters. Therefore, it has feature perception capabilities and can adaptively adjust the enhancement strategy according to different image content (such as land cover type, scale, texture).
[0043] This invention has strong domain adaptability: through the FADGM module, it effectively bridges the feature distribution differences between general pre-trained models and remote sensing / UAV imagery, significantly improving the model's accuracy in perceiving and understanding specialized ground features.
[0044] This invention features outstanding model versatility: through a design of a general core and optional adapters, it achieves for the first time unified support for heterogeneous models such as DINOv3 (ViT architecture) and Qwen3-VL (multimodal architecture) with a single enhancement module, possessing true plug-and-play characteristics.
[0045] This invention features high computational performance: the number of module parameters is extremely small, typically increasing the overall model parameter count by less than 1%, and the inference latency is extremely low. It supports a fine-tuning paradigm of freezing the backbone network and training only enhancement modules, significantly reducing computation and storage costs. It is suitable for UAV-borne or edge computing platforms.
[0046] This invention features precise feature selectivity enhancement: the dual-path dynamic gating mechanism can simultaneously and adaptively focus on the feature components most relevant to the task from both channel and spatial dimensions, effectively enhancing the representation of fine-grained ground property attributes.
[0047] This invention alleviates multi-task conflicts: by introducing independent gating weight initialization or adding orthogonal constraint loss with gating parameters for different tasks, feature conflicts and catastrophic forgetting problems can be effectively reduced when continuously learning multiple remote sensing tasks.
[0048] Furthermore, the pre-trained model backbone network includes an input source unit, a model selection unit, and a feature extraction unit; the input source unit is communicatively connected to the input end of the model selection unit, and the output end of the model selection unit is communicatively connected to the feature extraction unit.
[0049] The model selection unit automatically selects the appropriate model based on the input data type, such as whether it contains text, or loads the corresponding pre-trained model weights as specified by the user. For example, DINOv3 is selected for pure image input, and Qwen3-VL is selected for image-text pair input. The feature extraction unit then calls the forward computation graph of the selected model.
[0050] Furthermore, the image encoder includes an image input module, an architecture module, and a multi-layer feature output module; the image input module, the architecture module, and the multi-layer feature output module are sequentially connected in communication.
[0051] This is enabled when the backbone network is a pure vision model such as DINOv3. It includes an image input module, an architecture module, and a multi-layer feature output module. It is responsible for standardizing the input image and performing forward propagation through architectures such as ViT, outputting features from multiple Transformer layers, especially the CLS token and patch token sequences.
[0052] Furthermore, the multimodal encoder includes an image text input module, an architecture and language model module, and a multi-scale visual feature and text embedding output module; the image text input module, the architecture and language model module, and the multi-scale visual feature and text embedding output module are sequentially connected in communication.
[0053] This is enabled when the backbone network is a multimodal model such as Qwen3-VL. It includes an image and text input module, an architecture and language model module, and a multi-scale visual feature and text embedding output module. It processes both images and text simultaneously, interacting through a cross-modal Transformer architecture, and ultimately outputs multi-scale visual features before deep fusion and the corresponding text embedding vectors.
[0054] Furthermore, the multi-scale feature extraction layer module includes a shallow detail feature unit, a mid-level structural feature unit, and a deep semantic feature unit; the shallow detail feature unit, the mid-level structural feature unit, and the deep semantic feature unit are sequentially connected in communication.
[0055] Connected after the image encoder, this module comprises shallow units for detail features, mid-level units for structural features, and deep units for semantic features. It extracts feature maps or CLS tokens from different depths of the ViT layer (shallow, mid-, and deep), potentially using simple upsampling, downsampling, or linear projection for resolution alignment and channel unification, constructing a feature set with rich hierarchical information for use by subsequent enhancement modules.
[0056] Furthermore, the multimodal feature extraction layer module includes a visual feature projection unit, a text-guided attention unit, and a feature dimension alignment unit; the visual feature projection unit, the text-guided attention unit, and the feature dimension alignment unit are sequentially connected in communication.
[0057] This module is connected after the multimodal encoder. It includes a visual feature projection unit, a text-guided attention unit, and a feature dimension alignment unit. The visual feature projection unit maps the visual features output by the encoder to a uniform dimension. The text-guided attention unit uses text embeddings to initially focus attention on the visual features. The feature dimension alignment unit ensures that the dimensions of the visual features are consistent with the expected input of the FADGM module.
[0058] Furthermore, the adapter component unit includes a DINOv3 adapter and a Qwen3-VL adapter.
[0059] The DINOv3 adapter includes a token-level gating mechanism specifically for handling the relationship between CLS tokens and patch tokens in ViT. It dynamically fuses global information of CLS tokens into patch features through an additional gating network. It also includes a position encoding preserver, which ensures that critical position information of ViT is not corrupted during the enhancement process through residual connections.
[0060] The Qwen3-VL adapter includes a cross-modal guided attention mechanism that uses the vector representation of the input text embedding, such as a query statement, as the Query, and visual features as the Key and Value to calculate attention weights, so that the enhancement process of visual features is guided by the semantics of the text. It also includes task-conditional gating, which dynamically adjusts the parameters of the gating path in the core engine according to the input instructions or task type.
[0061] Furthermore, the task output header includes a task-specific output header and a multimodal task output header; The task output head is configured as a task-specific output head, such as a classification, detection, or segmentation head, or a multimodal task output head, such as a VQA or index delivery segmentation head, according to the requirements of the downstream task.
[0062] Furthermore, the task-specific output header includes a classifier, a segmenter, and a detector; the classifier, segmenter, and detector are sequentially connected in communication.
[0063] The classifier determines the category at the image level; the segmenter performs semantic parsing at the pixel level; and the detector locates and identifies the target at the object level. The output head receives the features enhanced by FADGM and performs the final task prediction.
[0064] Furthermore, the multimodal task output head includes a VQA answer generation unit, a pointer representation segmentation unit, and an image description generation unit; the VQA answer generation unit, the pointer representation segmentation unit, and the image description generation unit are sequentially connected in communication. The VQA answer generation unit is used for classification or generative algorithms, the representation segmentation unit is used for encoder-decoder architecture, and the image description generation unit is used to produce textual descriptions of images. These output heads receive FADGM-enhanced features for final task prediction.
[0065] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.
Claims
1. A plug-and-play feature-aware dynamic gating vision enhancement system, characterized in that, It includes a pre-trained model backbone network, an image encoder, a multimodal encoder, a multi-scale feature extraction layer module, a plug-and-play feature enhancement module (FADGM), and a task output header; The pre-trained model backbone network is connected to the multi-scale feature extraction layer module via an image encoder; the pre-trained model backbone network is connected to the multi-modal feature extraction layer module via a multi-modal encoder; both the multi-scale feature extraction layer module and the multi-modal feature extraction layer module are connected to the input of the plug-and-play feature enhancement module FADGM; the output of the plug-and-play feature enhancement module FADGM is connected to the task output head. The plug-and-play feature enhancement module FADGM includes an input feature normalization unit, a core enhancement engine unit, an adapter component unit, and an output feature normalization unit. The input feature normalization unit, the core enhancement engine unit, the adapter component unit, and the output feature normalization unit are connected in sequence. The core enhancement engine unit includes a channel gated path unit, a spatial gated path unit, and a residual fusion unit; the channel gated path unit, the spatial gated path unit, and the residual fusion unit are connected in sequence.
2. The plug-and-play feature-aware dynamic gating vision enhancement system according to claim 1, characterized in that, The pre-trained model backbone network includes an input source unit, a model selection unit, and a feature extraction unit; the input source unit is communicatively connected to the input end of the model selection unit, and the output end of the model selection unit is communicatively connected to the feature extraction unit.
3. The plug-and-play feature-aware dynamic gating vision enhancement system according to claim 2, characterized in that, The image encoder includes an image input module, an architecture module, and a multi-layer feature output module; the image input module, architecture module, and multi-layer feature output module are connected in sequence.
4. The plug-and-play feature-aware dynamic gating vision enhancement system according to claim 3, characterized in that, The multimodal encoder includes an image text input module, an architecture and language model module, and a multi-scale visual feature and text embedding output module; the image text input module, the architecture and language model module, and the multi-scale visual feature and text embedding output module are connected in sequence.
5. The plug-and-play feature-aware dynamic gating vision enhancement system according to claim 4, characterized in that, The multi-scale feature extraction layer module includes a shallow unit for detailed features, a medium unit for structural features, and a deep unit for semantic features. The shallow units of detailed features, the middle units of structural features, and the deep units of semantic features are connected in sequence.
6. The plug-and-play feature-aware dynamic gating vision enhancement system according to claim 5, characterized in that, The multimodal feature extraction layer module includes a visual feature projection unit, a text-guided attention unit, and a feature dimension alignment unit; the visual feature projection unit, the text-guided attention unit, and the feature dimension alignment unit are connected in sequence.
7. The plug-and-play feature-aware dynamic gating vision enhancement system according to claim 6, characterized in that, The adapter component unit includes a DINOv3 adapter and a Qwen3-VL adapter.
8. The plug-and-play feature-aware dynamic gating vision enhancement system according to claim 7, characterized in that, The task output header includes a task-specific output header and a multimodal task output header.
9. The plug-and-play feature-aware dynamic gating vision enhancement system according to claim 8, characterized in that, The task-specific output header includes a classifier, a segmenter, and a detector; the classifier, segmenter, and detector are connected in sequence.
10. The plug-and-play feature-aware dynamic gating vision enhancement system according to claim 9, characterized in that, The multimodal task output head includes a VQA answer generation unit, an index expression segmentation unit, and an image description generation unit; the VQA answer generation unit, the index expression segmentation unit, and the image description generation unit are sequentially connected in communication.
Citation Information
Patent Citations
Multi-modal video classification method based on dynamic fusion gate mechanism
CN117058594A
Visual language model prompt fine tuning method based on meta prompt
CN120470121A