A visible light and infrared image fusion method based on feature matching under a drone visual angle
By using feature matching methods, the computational efficiency and hardware dependency issues of fusion of visible light and infrared images from the perspective of UAVs were solved, achieving efficient and robust image fusion and improving the real-time performance and fusion quality of UAV inspections.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI YUNTONG ZHIHANG TECHNOLOGY CO LTD
- Filing Date
- 2025-05-30
- Publication Date
- 2026-05-12
AI Technical Summary
Existing methods for fusing visible light and infrared images from the perspective of UAVs suffer from problems such as low computational efficiency, high hardware dependence, and unstable fusion quality, making it difficult to meet the requirements of real-time performance and robustness, especially in UAV inspection.
A feature-matching-based approach is adopted, which integrates image preprocessing, target detection, feature extraction, feature alignment and fusion, and feature reconstruction to achieve efficient and robust fusion of visible light and infrared images.
It improves the computational efficiency of image fusion, reduces hardware costs, enhances the stability and clarity of the fusion results, and meets the real-time requirements of UAV inspection.
Smart Images
Figure CN120823463B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and particularly relates to a method for fusing visible light and infrared images based on feature matching from the perspective of a drone. Background Technology
[0002] Target detection systems based on UAV perspectives have been widely applied in engineering inspection. Most existing UAV-based target detection schemes rely on visible light or infrared images for prediction. Visible light images offer high resolution and contrast, but are difficult to image under low-light conditions such as nighttime or inclement weather. Infrared images, on the other hand, can perform all-weather detection and capture objects that visible light images cannot detect, but they typically have lower resolution and overlapping textures. Therefore, by rationally utilizing and complementing visible light and infrared images through fusion, richer semantic information can be obtained, resulting in a robust and information-rich fused image.
[0003] Current technologies for fusing visible and infrared light typically employ pixel fusion, but this approach still faces numerous technical bottlenecks and challenges in practical applications, specifically in the following aspects:
[0004] Computational efficiency and real-time performance issues: In practical applications such as drone inspection, the original image data is massive. Traditional pixel-level fusion methods (such as multi-scale transformation and deep learning networks) require pixel-by-pixel computation, resulting in a time-consuming fusion process that fails to meet real-time requirements. This is especially true for end-to-end fusion networks based on deep learning, whose complex model structures and training processes further exacerbate the computational burden.
[0005] Hardware dependency and cost issues: Pixel-level fusion requires precise alignment of visible light and infrared images at the pixel level, which places extremely high demands on the synchronization and stability of imaging equipment and increases hardware costs.
[0006] Fusion quality and robustness issues: Pixel information is easily contaminated by noise and other interference, leading to unstable quality of the original pixels, which affects the fusion result after stacking. Traditional multi-scale transformation methods struggle to effectively distinguish noise from real features when dealing with complex scenes; while deep learning networks, although possessing strong feature extraction capabilities, are prone to overfitting or insufficient generalization when data is insufficient or scene variations are significant, affecting the robustness of the fusion result.
[0007] To address the shortcomings of pixel-level fusion methods in terms of computational efficiency, hardware dependence, and fusion quality, this patent proposes a feature-level visible light and infrared image fusion algorithm. This algorithm abandons the traditional pixel-by-pixel processing approach and instead focuses on the feature level, extracting and aligning salient features of the same target in both visible light and infrared images to achieve efficient and robust image fusion. Summary of the Invention
[0008] The purpose of this invention is to solve the problems in the prior art, and to propose a visible light and infrared image fusion method based on feature matching from the perspective of a drone.
[0009] To address the aforementioned problems, the present invention aims to provide a method for fusing visible light and infrared images based on feature matching from the perspective of an unmanned aerial vehicle (UAV), comprising the following parts:
[0010] Image preprocessing module, used to receive visible light images and infrared images As input, preprocessing operations are used to improve image quality and eliminate modal differences, providing high-quality input data for subsequent modules.
[0011] Target detection module for detecting objects from visible light images and infrared images The system detects targets in the images and matches targets in two images through cross-modal association, providing target information for subsequent feature extraction and fusion.
[0012] The feature extraction module is used to extract modal complementary features of targets in visible light and infrared images, including shape, texture and edge information, to provide rich and robust feature representations for subsequent feature fusion.
[0013] The feature-level fusion module aligns and fuses the features extracted from visible light and infrared images to generate information-rich and complementary fused features, providing high-quality feature representations for subsequent tasks. It includes a feature alignment submodule and a feature fusion submodule.
[0014] The feature reconstruction module will optimize the fused features. The reconstructed image is of high quality, preserving complementary information from the visible light and infrared images while ensuring the sharpness and detail of the reconstructed image.
[0015] In the aforementioned visible light and infrared image fusion method based on feature matching from the perspective of a UAV, the image preprocessing module includes the following processing steps:
[0016] S1: Cross-modal registration: The image is geometrically corrected using an affine transformation matrix to eliminate field of view shift, as shown in the following formula:
[0017]
[0018] in For rotation and scaling parameters, , The translation amount is used to optimize parameters through SIFT feature point matching, and then the registered visible light image and infrared image are output.
[0019] S2: Noise suppression. On the one hand, Gaussian filtering is applied to the infrared image to smooth out noise, as shown in the following formula:
[0020]
[0021] in, is the Gaussian kernel function, used to calculate the weights between pixels, and Z is the normalization factor;
[0022] On the other hand, non-local mean denoising is used for visible light images to preserve image details, as shown in the following formula:
[0023]
[0024] in, Z is the similarity weight, calculated based on the similarity between pixel blocks. The normalization factor is used to output the denoised visible light image and infrared image.
[0025] S3: Dynamic Range Compression: Histogram equalization of infrared images to avoid overexposed or underexposed areas. Refer to the following formula:
[0026]
[0027] in, Where N is the grayscale frequency, and N is the total number of pixels;
[0028] Finally, output the infrared image after dynamic range compression.
[0029] In the aforementioned visible light and infrared image fusion method based on feature matching from the perspective of a UAV, the target detection module includes the following processing steps:
[0030] S1: Dual-modal detection network: using a YOLOv5 network with a shared backbone to process visible light images simultaneously. and infrared images Output target bounding box , And confidence level C, classification probability Calculated using the Softmax function:
[0031]
[0032] in, The features extracted for ROIAlign are W, where W is the classification weight and b is the bias term.
[0033] S2: Cross-modal association: Based on the Hungarian algorithm, combining the intersection-over-union (IoU) ratio of the target bounding box and feature similarity, it matches the same target in visible light and infrared images, and the matching score is... The calculation is as follows:
[0034]
[0035] To balance the weights, the contributions of IoU and feature similarity are controlled separately;
[0036] Output the matched target pairs and the matching score. .
[0037] In the aforementioned visible light and infrared image fusion method based on feature matching from the perspective of a UAV, the feature extraction module extracts the following:
[0038] Visible light features, wherein the visible light features are extracted from the visible light image using a ResNet-50 network, including edges and gradients, wherein the first... Layer characteristics The calculation is as follows:
[0039]
[0040] in, This represents the convolution operation. and The first Layer weights and bias terms;
[0041] Then output the visible light characteristics. ;
[0042] Infrared features: The infrared features employ an attention mechanism to enhance the salience of thermal targets in infrared images;
[0043] Among them, attention weight The calculation is as follows:
[0044]
[0045] in, For the Sigmoid function, and For the weights and biases of the attention mechanism, Indicating infrared features and visible light Feature splicing;
[0046] Then use attention weights Used for weighted infrared signatures to highlight the salience of thermal targets:
[0047]
[0048] Finally, the infrared signature is output. .
[0049] In the aforementioned visible light and infrared image fusion method based on feature matching from the perspective of a UAV, the feature alignment submodule is used to ensure the spatial and semantic consistency of features, and its workflow includes:
[0050] Geometric alignment: The geometric alignment is based on the matching pairs output by the target detection module. Spatial registration of infrared feature maps is performed using an affine transformation matrix:
[0051]
[0052] in It is an affine transformation operator; Optimize by backpropagation to minimize visible light features. and transformed infrared features The differences between them are then analyzed, and the aligned infrared features are output. ;
[0053] Semantic alignment: Utilizing cross-modal contrastive learning to construct feature similarity constraints, maximizing the similarity of cross-modal features with the same target:
[0054]
[0055] in Characteristic of visible light and infrared features The cosine similarity between them; τ is a temperature coefficient that controls the smoothness of the similarity distribution; K is the number of negative samples, and its output is the semantically aligned visible light features. and infrared features .
[0056] In the aforementioned visible light and infrared image fusion method based on feature matching from the perspective of a UAV, the feature fusion submodule includes:
[0057] Feature fusion:
[0058]
[0059] Where A is the attention weight, used to dynamically allocate the contributions of the two modalities, and its output is the preliminary fused feature. .
[0060] Feature optimization: The fused features are optimized to eliminate noise and inconsistencies, and improve their robustness and discriminative ability. The optimization process is implemented through a convolutional neural network.
[0061]
[0062] in, and To optimize the network's weights and biases, its output is the optimized fusion feature.
[0063] In the above-mentioned visible light and infrared image fusion method based on feature matching from the perspective of a UAV, the feature reconstruction module includes a decoding network part and an image post-processing part;
[0064] The decoding network portion uses a lightweight convolutional neural network or generative adversarial network as the decoder to fuse features. Decode into an image.
[0065] .
[0066] In the aforementioned visible light and infrared image fusion method based on feature matching from the perspective of a UAV, the image post-processing part includes the following steps:
[0067] S1: Denoising: Remove noise from the reconstructed image.
[0068] S2: Sharpening: Enhances image edges and details using the Laplacian operator or an adaptive sharpening filter;
[0069] S3: Dynamic Range Adjustment: Optimizes image brightness and contrast using histogram equalization or adaptive contrast stretching.
[0070]
[0071] Then output the final reconstructed image. .
[0072] In the aforementioned visible light and infrared image fusion method based on feature matching from the perspective of a UAV,
[0073] This patent addresses the shortcomings of pixel-level fusion methods in terms of computational efficiency, hardware dependence, and fusion quality by proposing a feature-level visible light and infrared image fusion method. This method abandons the traditional pixel-by-pixel processing approach and instead focuses on the feature level. By extracting and aligning salient features of the same target in both visible light and infrared images, it achieves efficient and robust image fusion. Attached Figure Description
[0074] Figure 1 This is a flowchart of a visible light and infrared image fusion method based on feature matching from the perspective of an unmanned aerial vehicle (UAV).
[0075] Figure 2 This is a decoding network structure diagram in a visible light and infrared image fusion method based on feature matching from the perspective of an unmanned aerial vehicle (UAV). Detailed Implementation
[0076] Reference Figure 1-2 A visible light and infrared image fusion method based on feature matching from the perspective of a drone includes the following modules:
[0077] 1) Image preprocessing module
[0078] This module receives visible light images. and infrared images As input, preprocessing operations are used to improve image quality and eliminate modal differences, providing high-quality input data for subsequent modules.
[0079] Specific workflow:
[0080] Step 1: Cross-modal registration: Use an affine transformation matrix to perform geometric correction on the infrared / visible light images to eliminate field of view offset.
[0081]
[0082] in For rotation and scaling parameters, , The translation amount is optimized by SIFT feature point matching.
[0083] Output: Registered visible light and infrared images.
[0084] Step 2: Noise suppression, including:
[0085] ① Apply Gaussian filtering to the infrared image to smooth out noise.
[0086]
[0087] in, is the Gaussian kernel function used to calculate the weights between pixels. Z is the normalization factor.
[0088] ② Visible light images are denoised using non-local means to preserve image details:
[0089]
[0090] in, The similarity weight is calculated based on the similarity between pixel blocks. Z This is the normalization factor.
[0091] Output: Denoising-reduced visible light and infrared images.
[0092] Step 3: Dynamic range compression: Histogram equalization is performed on the infrared image to avoid overexposed or underexposed areas.
[0093]
[0094] in, Where is the grayscale frequency, and N is the total number of pixels.
[0095] Output: Infrared image with dynamic range compression.
[0096] 2) Target Detection Module
[0097] This module is responsible for processing visible light images. and infrared images The system detects targets in the images and matches targets in two images through cross-modal association, providing accurate target information for subsequent feature extraction and fusion.
[0098] Specific workflow:
[0099] Step 1: Dual-modal detection network: Using a YOLOv5 network with a shared backbone, simultaneously processing visible light images. and infrared images Output target bounding box , And confidence level C. Classification probability. Calculated using the Softmax function:
[0100]
[0101] in, is the feature extracted by ROIAlign, where W is the classification weight and b is the bias term.
[0102] Step 2: Cross-modal association: Based on the Hungarian algorithm, combining the intersection-over-union (IoU) ratio of the target bounding boxes and feature similarity, match the same target in visible light and infrared images. Matching score. The calculation is as follows:
[0103]
[0104] To balance the weights, the contributions of IoU and feature similarity are controlled separately.
[0105] Output: Matched target pairs, and matching scores. .
[0106] 3) Feature extraction module
[0107] Modal complementary features of targets in visible light and infrared images are extracted, including key information such as shape, texture, and edges, to provide rich and robust feature representations for subsequent feature fusion.
[0108] Among them, visible light features: high-frequency texture features (such as edges and gradients) in visible light images are extracted using a ResNet-50 network. Layer characteristics The calculation is as follows:
[0109]
[0110] in, This represents the convolution operation. and The first Layer weights and biases.
[0111] Output: Visible light characteristics .
[0112] Infrared features: An attention mechanism is used to enhance the salience of thermal targets in infrared images.
[0113] Attention weight The calculation is as follows:
[0114]
[0115] in, For the Sigmoid function, and These are the weights and biases for the attention mechanism. Indicating infrared features and visible light Feature splicing.
[0116] Using attention weights Used for weighted infrared signatures to highlight the salience of thermal targets:
[0117]
[0118] Output: Infrared signature .
[0119] 4) Feature-level fusion module
[0120] This module is responsible for aligning and fusing features extracted from visible light and infrared images to generate informative and complementary fused features, providing high-quality feature representations for subsequent tasks.
[0121] (1) Feature Alignment Submodule
[0122] Because visible light and infrared images operate on different imaging principles, the features of the same target may exhibit spatial shifts or scale differences in the two modalities. Therefore, geometric and semantic alignment of the extracted features is necessary to ensure spatial and semantic consistency.
[0123] Working principle and process:
[0124] Step 1: Geometric Alignment: Based on the matching pairs output by the target detection module Spatial registration of infrared feature maps is performed using an affine transformation matrix:
[0125]
[0126] in It is an affine transformation operator; Optimize by backpropagation to minimize visible light features. and transformed infrared features The differences between them.
[0127] Output: Aligned infrared features .
[0128] Step 2: Semantic Alignment: Construct feature similarity constraints using Cross-Modal Contrastive Learning (CMCL) to maximize the similarity of cross-modal features with the same target.
[0129]
[0130] in Characteristic of visible light and infrared features The cosine similarity between them; τ is the temperature coefficient, which controls the smoothness of the similarity distribution; K is the number of negative samples.
[0131] Output: Semantically aligned visible light features and infrared features .
[0132] (2) Feature Fusion Submodule
[0133] The aligned visible light features and infrared features are weighted and fused to generate a fused feature representation.
[0134] Working principle and process:
[0135] Step 1: Feature Fusion
[0136]
[0137] Where A is the attention weight, which is used to dynamically allocate the contribution of the two modal features.
[0138] Output: Preliminary fusion features .
[0139] Step 2: Feature Optimization
[0140] The fused features are optimized to eliminate noise and inconsistencies, thereby improving their robustness and discriminative ability. This optimization process is implemented using a convolutional neural network.
[0141]
[0142] in, and To optimize the network's weights and biases
[0143] Output: Optimized fusion features .
[0144] 5) Feature Reconstruction
[0145] The fused features The reconstructed image is of high quality, preserving complementary information from the visible light and infrared images while ensuring the sharpness and detail of the reconstructed image.
[0146] Working principle and process:
[0147] Step 1: Decoding the network:
[0148] Using a lightweight convolutional neural network (CNN) or generative adversarial network (GAN) as the decoder, the fused features are... Decode into an image.
[0149]
[0150] Step 2: Image post-processing, including:
[0151] ① Noise Removal: Use non-local means or bilateral filtering to remove noise from the reconstructed image.
[0152] ② Sharpening: Use the Laplacian operator or adaptive sharpening filter to enhance image edges and details.
[0153] ③ Dynamic Range Adjustment: Optimize image brightness and contrast using histogram equalization or adaptive contrast stretching.
[0154]
[0155] Output: Final reconstructed image It combines complementary information from visible light and infrared images, resulting in high-quality clarity and detail.
[0156] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative and not exhaustive. All modifications within the scope of this invention or its equivalents are included in this invention.
Claims
1. A method for fusing visible light and infrared images based on feature matching from the perspective of an unmanned aerial vehicle (UAV), characterized in that, Includes the following parts: Image preprocessing module, used to receive visible light images and infrared images As input, preprocessing operations are used to improve image quality and eliminate modal differences, providing high-quality input data for subsequent modules. Target detection module for detecting objects from visible light images and infrared images The system detects targets in the images and matches targets in two images through cross-modal association, providing target information for subsequent feature extraction and fusion. The feature extraction module is used to extract modal complementary features of targets in visible light and infrared images, including shape, texture and edge information, to provide rich and robust feature representations for subsequent feature fusion. The feature-level fusion module aligns and fuses features extracted from visible light and infrared images, generating informative and complementary fused features that provide high-quality feature representations for subsequent tasks. Includes a feature alignment submodule and a feature fusion submodule; The feature reconstruction module will optimize the fused features. The reconstructed image is of high quality, preserving complementary information from the visible light and infrared images while ensuring the sharpness and detail of the reconstructed image. The feature extraction module includes: Visible light features, wherein the visible light features are extracted from the visible light image using a ResNet-50 network, including edges and gradients, wherein the first... Features of the layer The calculation is as follows: in, This represents the convolution operation. and The first Layer weights and biases; Then output the visible light characteristics. ; Infrared features: The infrared features employ an attention mechanism to enhance the salience of thermal targets in infrared images; Among them, attention weight The calculation is as follows: in, For the Sigmoid function, and For the weights and biases of the attention mechanism, Indicating infrared features and visible light characteristics splicing; Then use attention weights Used for weighted infrared signatures to highlight the salience of thermal targets: Finally, the infrared signature is output. .
2. The visible light and infrared image fusion method based on feature matching from the perspective of a UAV as described in claim 1, characterized in that: The image preprocessing module includes the following processing steps: S1: Cross-modal registration: The image is geometrically corrected using an affine transformation matrix to eliminate field of view shift, as shown in the following formula: in For rotation and scaling parameters, , The translation amount is used to optimize parameters through SIFT feature point matching, and then the registered visible light image and infrared image are output. S2: Noise suppression. On the one hand, Gaussian filtering is applied to the infrared image to smooth out noise, as shown in the following formula: in, is the Gaussian kernel function, used to calculate the weights between pixels, and Z is the normalization factor; On the other hand, non-local mean denoising is used for visible light images to preserve image details, as shown in the following formula: in, Z' is the similarity weight, calculated based on the similarity between pixel blocks. Z' is the normalization factor. Finally, the denoised visible light image and infrared image are output. S3: Dynamic Range Compression: Histogram equalization of infrared images to avoid overexposed or underexposed areas. Refer to the following formula: in, Where N is the grayscale frequency, and N is the total number of pixels; Finally, output the infrared image after dynamic range compression.
3. The visible light and infrared image fusion method based on feature matching from the perspective of a UAV as described in claim 1, characterized in that, The target detection module includes the following processing steps: S1: Dual-modal detection network: using a YOLOv5 network with a shared backbone to process visible light images simultaneously. and infrared images Output target bounding box , And confidence level C, classification probability Calculated using the Softmax function: in, The features extracted for ROIAlign are W, where W is the classification weight and b is the bias term. S2: Cross-modal association: Based on the Hungarian algorithm, combining the intersection-over-union ratio (IoU) and feature similarity of the target bounding box, it matches the same target in visible light and infrared images, and the matching score is calculated. The calculation is as follows: To balance the weights, the contributions of IoU and feature similarity are controlled separately; Output the matched target pairs and the matching score. .
4. The visible light and infrared image fusion method based on feature matching from the perspective of a UAV as described in claim 3, characterized in that, The feature alignment submodule is used to ensure the spatial and semantic consistency of features, and its workflow includes: Geometric alignment: The geometric alignment is based on the matching pairs output by the target detection module. Spatial registration of infrared feature maps is performed using an affine transformation matrix: in It is an affine transformation operator; Optimize by backpropagation to minimize visible light features. and transformed infrared features The differences between them are then analyzed, and the aligned infrared features are output. ; Semantic alignment: Utilizing cross-modal contrastive learning to construct feature similarity constraints, maximizing the similarity of cross-modal features with the same target: in Characteristic of visible light and infrared features The cosine similarity between them; τ is a temperature coefficient that controls the smoothness of the similarity distribution; K is the number of negative samples, and then the semantically aligned visible light features are output. and infrared features .
5. The visible light and infrared image fusion method based on feature matching from the perspective of a UAV as described in claim 2, characterized in that, The feature fusion submodule includes: Feature fusion: Where A is the attention weight, used to dynamically allocate the contributions of the two modalities, and its output is the preliminary fused feature. ; Feature optimization: The fused features are optimized to eliminate noise and inconsistencies, and improve their robustness and discriminative ability. The optimization process is implemented through a convolutional neural network. in, and To optimize the network's weights and biases, its output is the optimized fusion feature.
6. The visible light and infrared image fusion method based on feature matching from the perspective of a UAV as described in claim 2, characterized in that, The feature reconstruction module includes a decoding network and an image post-processing part; The decoding network portion uses a lightweight convolutional neural network or generative adversarial network as the decoder to fuse features. Decode into an image. 。 7. The visible light and infrared image fusion method based on feature matching from the perspective of a UAV as described in claim 6, characterized in that, The image post-processing section Includes the following steps: S1： Denoising: Removing noise from the reconstructed image; S2: Sharpening: Enhances image edges and details using the Laplacian operator or an adaptive sharpening filter; S3: Dynamic Range Adjustment: Optimizes image brightness and contrast using histogram equalization or adaptive contrast stretching. Then output the final reconstructed image. .
8. The visible light and infrared image fusion method based on feature matching from the perspective of a UAV according to claim 7, characterized in that, The denoising method employs either nonlocal mean or bilateral filtering.