A text-guided multi-modal image fusion method and system

By employing a text-guided multimodal image fusion method, utilizing a hierarchical encoder and affine blocks for feature mapping modulation, and combining a hybrid channel fusion module, the degradation problem of infrared-visible images in complex environments is solved, achieving high-quality and flexible image fusion effects that are adaptable to various degradation scenarios.

CN122155968APending Publication Date: 2026-06-05ZHEJIANG NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
ZHEJIANG NORMAL UNIV
Filing Date
2026-03-19
Publication Date
2026-06-05

AI Technical Summary

Technical Problem

Existing image fusion methods cannot effectively solve the degradation problem of infrared-visible images in complex environments, and traditional methods are difficult to meet the requirements of flexibility and interactivity, and cannot make full use of the complex relationship between textual semantic information and image fusion features.

Method used

A text-guided multimodal image fusion method is adopted, which uses a hierarchical encoder and CLIP text encoder to extract features, combines spatial affine blocks and channel affine blocks for feature mapping and modulation, performs fusion through a multi-level semantic interactive decoder, and uses a hybrid channel fusion module to achieve refined information interaction and redundancy reduction.

Benefits of technology

It achieves more flexible and higher-quality image fusion in complex scenes, can adaptively recover various degradation problems, improves the performance and interactivity of image fusion models, and generates fusion results that are more in line with human visual perception.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122155968A_ABST
    Figure CN122155968A_ABST
Patent Text Reader

Abstract

The application discloses a kind of based on text guide's multimodal image fusion method and system, belong to image fusion technique, its method includes the following steps: using hierarchical encoder respectively to infrared image and visible light image are extracted, and different levels of infrared features and visible light features are obtained;Text semantic features of input task text are obtained using CLIP text encoder;Infrared features and visible light features are respectively mapped using spatial affine block, and the mapping result is modulated to text semantic features, to obtain the text guide feature under the same spatial resolution;After the most deep level infrared feature and visible light feature are decomposed and fused, input multistage semantic interaction decoder is restored into high-resolution feature of different levels, each layer high-resolution feature is modulated using channel affine block combined with text semantic features, and the modulation result is mixed and fused with text guide feature.The limitation of image fusion model under degradation condition is improved.
Need to check novelty before this filing date? Find Prior Art