Infrared and visible image fusion method based on text-visual semantic logic alignment
Patent Information
- Application Number
- CN202610856722.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-15
- Publication Date
- 2026-10-16
AI Technical Summary
然而，深度特征提取类方法仅依靠底层视觉特征完成融合，缺乏语义引导能力；任务导向优化类方法的语义信息仅源于图像自身，难以响应外部文本指令；现有文本引导类方法多停留在实体词汇的浅层对齐，缺乏对文本句法逻辑与图像空间结构的深度匹配
[0008]本发明提出的一种文本-视觉语义逻辑对齐的红外与可见光图像融合方法，通过文本-视觉语义逻辑对齐模块与语义逻辑引导特征融合模块的协同设计，实现文本语义与跨模态特征的精准匹配，保障红外热辐射信息与可见光纹理细节的互补融合，最终提升了红外与可见光图像融合性能。解决了现有技术所出现的文本引导红外-可见光图像融合中的“词汇-空间错位”问题，实现了抽象文本语义到精准空间融合决策的高效转化。
Smart Images
Figure CN122928639A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision, and in particular relates to a method for fusing infrared and visible light images based on text-visual semantic logic alignment. Background Technology
[0002] Infrared and visible light image fusion technology integrates the infrared modality's ability to perceive heat sources and the visible light modality's texture details, providing crucial support for applications such as nighttime surveillance, intelligent transportation, and military reconnaissance. Existing fusion methods have evolved from deep feature extraction-based fusion to task-oriented optimization, and have begun to incorporate text guidance. However, deep feature extraction methods rely solely on low-level visual features for fusion, lacking semantic guidance capabilities; task-oriented optimization methods derive their semantic information only from the image itself, making them difficult to respond to external text commands; and existing text-guided methods often remain at a shallow alignment of entity words, lacking deep matching between text syntactic logic and image spatial structure. Therefore, existing methods struggle to achieve effective unification of semantic understanding and spatial localization, easily leading to the loss of key information or mismatch with text commands in the fusion results. Summary of the Invention
[0003] The present invention aims to solve the aforementioned technical problems existing in the prior art by providing a method for fusion of infrared and visible light images based on text-visual semantic logic alignment.
[0004] The technical solution of this invention is: a method for fusing infrared and visible light images based on text-visual semantic logic alignment, which utilizes a trained network to fuse infrared and visible light images. The trained network is constructed through the following steps: Step 1. Create a fused dataset of infrared images, visible light images, and text commands; Step 2. Establish and initialize the network, which consists of a dual-branch deep neural network encoder, a text-visual semantic logic alignment module, a semantic logic-guided feature fusion module, and a decoder. The dual-branch deep neural network encoder is used to extract infrared image features and visible light image visual features respectively. The text-visual semantic logic alignment module is used to perform cross-attention alignment processing on text features, infrared visual features, and visible light visual features to generate infrared semantic logic modulation maps and visible light semantic logic modulation maps. The semantic logic-guided feature fusion module is used to fuse infrared visual features, visible light visual features, and corresponding semantic logic modulation maps through a cross-attention mechanism to generate fused features. The decoder is used to decode the fused features to generate an infrared and visible light fused image. Step 3. Based on the fused dataset, complete network training using a preset loss function.
[0005] The preferred text-visual semantic logic alignment module operates as follows: Step 2.1 Text Semantic Encoding The input text instructions are processed by a text encoder and a self-attention layer to obtain text semantic features, and a text self-attention matrix is constructed. Step 2.2 Cross-modal cross-attention interaction Text semantic features are input into two parallel cross-attention layers, corresponding to the infrared branch and the visible light branch, respectively. The infrared branch calculates the association between text semantics and infrared visual features to obtain infrared attention features; the visible light branch calculates the association between text semantics and visible light visual features to obtain visible light attention features. Step 2.3 Generate the initial heatmap set Based on infrared attention features and visible light attention features, initial infrared thermal map sets and visible light thermal map sets are generated respectively; Step 2.4 Heatmap Optimization and Modulation Output The heatmap group is input into the heatmap optimization module, which performs constraint optimization on the heatmap based on the text self-attention matrix, and then performs weighted aggregation on the optimized heatmap to output the infrared semantic logic modulation map and the visible light semantic logic modulation map.
[0006] The preferred semantic logic-guided feature fusion module operates as follows: Step 2.5 Infrared visual features and visible light visual features are first processed through convolutional layers, normalization layers, and linear layers to generate their respective query features. Q ir , Q vi ; Step 2.6 The infrared visual features and visible light visual features are first processed through convolutional layers and normalization layers respectively, and then concatenated along the channel dimension to obtain fused features. These features are then sequentially fed into a linear layer and the SE channel attention module to generate shared key features. K f ; Step 2.7 The infrared semantic logic modulation map and the visible light semantic logic modulation map are obtained by L2 normalization and spatial replication to obtain semantic value features. V ir , V vi ; Step 2.8 will Q ir , Q vi respectively with K f transposed K f TPerform attention calculations, and then connect them with the corresponding... V ir , V vi Weighted fusion is performed, and the results from the two paths are concatenated and reconstructed before being fed into a double convolutional layer to obtain the final fused feature.
[0007] The preferred preset loss function is as follows: ; in, The weights are used to balance the semantic-logical alignment loss. The loss function is based on SSIM; ; , These are the infrared visual semantic association matrix and the visible light visual semantic association matrix, generated based on the initial heatmap. M The text self-attention matrix, MSE (·) represents the mean squared error loss calculated by averaging all elements of the matrix.
[0008] This invention proposes a text-visual semantic logic alignment method for infrared and visible light image fusion. Through the collaborative design of a text-visual semantic logic alignment module and a semantic logic-guided feature fusion module, it achieves precise matching of text semantics and cross-modal features, ensuring complementary fusion of infrared thermal radiation information and visible light texture details, ultimately improving the performance of infrared and visible light image fusion. This solves the "lexical-spatial misalignment" problem in existing text-guided infrared-visible light image fusion technologies, achieving efficient transformation from abstract text semantics to precise spatial fusion decisions. Attached Figure Description
[0009] Figure 1 This is a schematic diagram of the overall network architecture of an embodiment of the present invention.
[0010] Figure 2 This is a flowchart illustrating the architecture of the text-visual semantic logic alignment module in an embodiment of the present invention.
[0011] Figure 3 This is a flowchart illustrating the architecture of the semantic logic-guided feature fusion module in an embodiment of the present invention.
[0012] Figure 4 This is a comparison image of the fused image from an embodiment of the present invention and the fused image from an existing method. Detailed Implementation
[0013] This invention discloses a method for fusing infrared and visible light images based on text-visual semantic logic alignment. The method utilizes a trained network to fuse infrared and visible light images. The trained network is constructed through the following steps: Step 1. Create a fused dataset of infrared images, visible light images, and text commands; Step 1.1 Obtain the infrared source image set in the remote sensing scene and visible light source image set This ensures that each infrared image and visible light image are a one-to-one registration pair; Step 1.2: Transfer the infrared image and visible light source images Adjust the height and width to be the same; Step 1.3 For each registered image pair , Based on the features of ground objects in the image, the corresponding text instructions T (such as "a person is sitting on a chair next to a tree") are manually labeled to form three sets of samples: "infrared image - visible light image - text instruction". Step 1.4 divides the three sets of samples into training and testing sets in a 5:3 ratio to form a complete text-guided infrared and visible light image fusion dataset.
[0014] Step 2. Establish and initialize the network, the network architecture as follows: Figure 1 As shown: It consists of a deep neural network encoder (UNet encoder) with a dual-branch structure, a text-visual semantic logic alignment module, a semantic logic guided feature fusion module, and a decoder (UNet decoder); The dual-branch deep neural network encoder is used to extract features from infrared images and visual features from visible light images, specifically infrared images. Input one encoder, visible light image Input another encoder to obtain infrared visual feature F ir ∈R B×C×H×W Visible light visual characteristics (deep features of the brightness channel) F vi ∈R B×C×H×W Where B is the batch size, C is the feature dimension, and H and W are the height and width of the feature map; The text-visual semantic logic alignment module is used to perform cross-attention alignment processing on text features, infrared visual features, and visible light visual features to generate an infrared semantic logic modulation map G. ir and visible light semantic logic modulation map G vi ; The flowchart of the text-visual semantic logic alignment module architecture is as follows: Figure 2 As shown, the specific operation is as follows: Step 2.1 Text Semantic Encoding First, the input text instruction T is converted into a token sequence by CLIPTokenizer, and then the input parameters are frozen by CLIP text encoder to extract text semantic features. Ψ text ∈R B×N×C Where B is the batch size, N is the text token length, and C is the feature dimension, the average of the self-attention matrices of the last K attention heads is taken to obtain the text syntactic association matrix M. This process is expressed as: ; in, Let i and j be the self-attention weights of the i-th and j-th text tokens, respectively, under the K-th attention head in the b-th sample. The loop index for traversing K attention heads; M[b,i,j] represents the syntactic association strength between the i-th token and the j-th token in the b-th sample. The larger the value, the closer the semantic logic between the two. It is used to encode the semantic logic relationship within the text and provide the text syntax basis for subsequent visual semantic alignment. Step 2.2 Cross-modal cross-attention interaction Specifically, infrared visual features F ir ∈R B×C×H×W and visible light visual features F vi ∈R B×C×H×W Flattened into a visual token sequence, the visual sequence serves as the query (Q), and textual semantic features are used. Ψ text ∈R B×N×C Using key (K) and value (V), the semantic similarity between the visual token and the text token is calculated. Attention weights are applied to V, and textual semantic information is fused to obtain the aligned infrared attention feature F. ir attn and visible light attention features F vi attn This ensures that each visual location is fused and matched with the corresponding text semantics; Step 2.3 Generate the initial heatmap set Based on infrared attention features F ir attn Visible light attention characteristics F vi attn The initial infrared thermal map set and visible light thermal map set are reconstructed by linear projection into multi-token semantics. Step 2.4 Heatmap Optimization and Modulation Output After normalizing the multi-token semantic infrared heatmap group and the visible light heatmap group along the token dimension, the heatmaps are constrained and optimized based on the text self-attention matrix to ensure that the distribution pattern of the heatmaps is consistent with the semantic association logic within the text. Then, the optimized heatmaps are weighted and aggregated to finally output the infrared semantic logic modulation map G. ir ∈R B×1×H×W With visible light semantic logic modulation map G vi ∈R B×1×H×W ; The semantic logic-guided feature fusion module is used to fuse infrared visual features, visible light visual features, and corresponding semantic logic modulation maps through a cross-attention mechanism to generate fused features. ; The semantic logic-guided feature fusion module architecture process is as follows: Figure 3 As shown, the specific operation is as follows: Step 2.5 Infrared visual features F ir and visible light visual features F vi First, the channel dimensions are adjusted using 1×1 convolutional layers, then normalized using normalization layers, followed by spatial flattening, and finally independent linear mapping using linear layers to generate infrared visual features F. ir and visible light visual features F vi Each corresponding query feature Q ir , Q vi This ensures that each modality query vector is associated only with the core features of its own modality, avoiding cross-modality feature interference; Step 2.6: Infrared and visible light visual features are first processed through 1×1 convolutional layers to adjust channel dimensions, then through normalization layers for normalization, and finally flattened in spatial dimension to obtain preprocessed infrared and visible light visual features. These two preprocessed visual features are then concatenated along the channel dimension, compressed through a linear layer, and input into the SE channel attention module to enhance the cross-modal key channel response, ultimately generating shared key features. K f ; Step 2.7 Infrared Semantic Logic Modulation Map G ir With visible light semantic logic modulation map G vi Semantic value features were obtained after L2 normalization and spatial copying. V ir , V vi ; Step 2.4 will Q ir , Q vi respectively withK f transposed K f T Perform attention calculations, and then connect them with the corresponding... V ir , V vi Weighted fusion, the process is expressed as: ; In the formula, d is the scaling factor. k For the attention head dimension, · represents matrix multiplication, and ⊙ represents element-wise multiplication, realizing semantic value features V. ir With V vi The weighted regulation of attention interaction results is the core improvement that distinguishes this module from traditional cross-attention. Then the results of the two paths will be... , After being concatenated and reshaped, the data is fed into a dual convolutional layer to achieve dimensionality compression and nonlinear enhancement, resulting in integrated features. C The (·) method consists of two 3×3 convolutional layers, each followed by BatchNorm2d normalization and ReLU activation. Finally, the integrated features are element-wise summed with the mean of the original features from both modalities to achieve residual connections, yielding the final fused features. ; This process can be expressed as: ; The decoder is used for fused features. Decoding is performed to generate a fused infrared and visible light image. ; Step 3. Based on the fused dataset, use the following preset total loss function. Complete the training; ; in, = 0.1 is the balancing weight of semantic logic alignment loss, used to avoid semantic constraints from excessively suppressing the visual fusion effect; The loss function is based on SSIM; ; in , These are the infrared visual semantic association matrix and the visible light visual semantic association matrix, generated based on the initial heatmap. M The text self-attention matrix, MSE (·) represents the mean squared error loss averaged over all elements of the matrix; In the text-visual semantic logical alignment module, the visual semantic association matrix S, constrained by MSE loss, is... ir and S vi Text Syntactic Association Matrix M Consistency is ensured to match the semantic logic modulation diagram with the text logic; Specific , It is a visual semantic association matrix constructed based on the initial heatmap using cosine similarity, and the process uses infrared branches. For example, the initial infrared semantic heatmap corresponding to each text token is first flattened into a one-dimensional vector. Then calculate the cosine similarity between any two vectors. This process can be expressed as: ; Where b is the batch index, and i,j are the valid text token indexes. For the i-th token The corresponding infrared semantic heatmap flattening vector, where L is the number of valid tokens, and D = H × W is the spatial dimension of the heatmap. It is the L2 norm. This is a numerically stable term used to prevent the denominator from approaching 0; The parameters for network training are set as follows: base_lr: 1e-4 / learning rate; momentum: 0.9 / momentum; weight_decay: 1e-5 / weight decay; Batch size: 2 / batch size; solver_mode:GPU / This example uses GPU for training; The above steps complete the construction of the trained network.
[0015] Testing phase; Input infrared and visible light images with width W and height H from the test set. , The corresponding text instruction T is used to generate a semantic logic modulation map through the text-visual semantic logic alignment module. Then, the feature interaction and weighted fusion are completed by the semantic logic-guided cross-modal feature fusion module, and finally the fused image is output. .
[0016] The fused image obtained during the testing phase of this invention is compared with fused images obtained by other fusion methods, and the results are as follows: Figure 4 As shown. Figure 4(a) is the source image; (b) to (j) are the images processed by YDTR, SwinFusion, LRRNet, EMMA, MMDRFuse, Text-IF, and DCE, respectively. V0 The fused images obtained by SAGE and the method of this invention are compared. The results show that, compared with other fusion methods, the fusion result obtained by the method of this invention can better preserve the main features of the source image, including brightness features and texture features.
Claims
1. A method for fusing infrared and visible light images based on text-visual semantic logic alignment, which utilizes a trained network to fuse infrared and visible light images, characterized in that: The trained network is constructed through the following steps: Step 1. Create a fused dataset of infrared images, visible light images, and text commands; Step 2. Establish and initialize the network, which consists of a dual-branch deep neural network encoder, a text-visual semantic logic alignment module, a semantic logic-guided feature fusion module, and a decoder. The dual-branch deep neural network encoder is used to extract infrared image features and visible light image visual features respectively. The text-visual semantic logic alignment module is used to perform cross-attention alignment processing on text features, infrared visual features, and visible light visual features to generate infrared semantic logic modulation maps and visible light semantic logic modulation maps. The semantic logic-guided feature fusion module is used to fuse infrared visual features, visible light visual features, and corresponding semantic logic modulation maps through a cross-attention mechanism to generate fused features. The decoder is used to decode the fused features to generate an infrared and visible light fused image. Step 3. Based on the fused dataset, complete network training using a preset loss function.
2. The infrared and visible light image fusion method based on text-visual semantic logic alignment according to claim 1, characterized in that... The specific operations of the text-visual semantic logical alignment module are as follows: Step 2.1 Text Semantic Encoding The input text instructions are processed by a text encoder and a self-attention layer to obtain text semantic features, and a text self-attention matrix is constructed. Step 2.2 Cross-modal cross-attention interaction The text semantic features are input into two parallel cross-attention layers, corresponding to the infrared branch and the visible light branch, respectively. The infrared branch calculates the association between the text semantics and the infrared visual features to obtain the infrared attention features. The visible light branch calculates the association between text semantics and visible light visual features to obtain visible light attention features; Step 2.3 Generate the initial heatmap set Based on infrared attention features and visible light attention features, initial infrared thermal map sets and visible light thermal map sets are generated respectively; Step 2.4 Heatmap Optimization and Modulation Output The heatmap group is input into the heatmap optimization module, which performs constraint optimization on the heatmap based on the text self-attention matrix, and then performs weighted aggregation on the optimized heatmap to output the infrared semantic logic modulation map and the visible light semantic logic modulation map.
3. The infrared and visible light image fusion method based on text-visual semantic logic alignment according to claim 2, characterized in that... The semantic logic-guided feature fusion module operates as follows: Step 2.5 Infrared visual features and visible light visual features are first processed through convolutional layers, normalization layers, and linear layers to generate their respective query features. Q ir , Q vi ; Step 2.6 The infrared visual features and visible light visual features are first processed through convolutional layers and normalization layers respectively, and then concatenated along the channel dimension to obtain fused features. These features are then sequentially fed into a linear layer and the SE channel attention module to generate shared key features. K f ; Step 2.7 The infrared semantic logic modulation map and the visible light semantic logic modulation map are obtained by L2 normalization and spatial replication to obtain semantic value features. V ir , V vi ; Step 2.8 will Q ir , Q vi respectively with K f transposed K f T Perform attention calculations, and then connect them with the corresponding... V ir , V vi Weighted fusion is performed, and the results from the two paths are concatenated and reconstructed before being fed into a double convolutional layer to obtain the final fused feature.
4. The infrared and visible light image fusion method based on text-visual semantic logic alignment according to claim 3, characterized in that... The preset loss function is as follows: ； in, The weights are used to balance the semantic-logical alignment loss. The loss function is based on SSIM; ； S ir 、 S vi These are the infrared visual semantic association matrix and the visible light visual semantic association matrix, generated based on the initial heatmap. M The text self-attention matrix, MSE (·) represents the mean squared error loss calculated by averaging all elements of the matrix.