Low-illumination infrared visible light video fusion method based on LUT and Transformer

Abstract

The invention discloses a low-illumination infrared visible light video fusion method based on an LUT and a Transform, and belongs to the field of infrared visible light fusion. Comprising dynamic object shielding extraction; static and dynamic parts are separated; dynamic target enhancement; performing time logic consistency processing; coding and decoding processing; style guidance and final fusion; and a target synthesis step. Through respective reconstruction of the dynamic part and the static part and time sequence consistency processing, the time sequence synchronization problem is effectively solved, and the reconstruction quality is improved. Particularly in multi-source video fusion, the optical flow module is used for accurately shielding the dynamic part of each video sequence, so that the influence of a dynamic background on static scene reconstruction is effectively reduced. And in combination with fine tuning of the large language model, the adaptability and fusion effect of the technology are further enhanced, and especially in a complex dynamic scene, the precision and stability of overall reconstruction are improved.

Description

Technical Field

[0001] The present invention belongs to the field of infrared and visible light fusion, and specifically relates to a low-illumination infrared and visible light video fusion method based on LUT and Transformer. Background Art

[0002] With the widespread application of infrared and visible light fusion technology in multiple high-end fields such as autonomous driving, smart security, smart medical care, and virtual reality, this technology has great market application potential. Due to its uniqueness and the high commercial value of the problems it solves, this technical solution has strong potential for technical licensing and industrial promotion in related fields. This technology can effectively meet the needs of dynamic and static scene fusion in complex environments such as low light and strong reflections, provide high-precision reconstruction effects, and greatly enhance the market competitiveness of related products. However, general image fusion models often only consider the fusion between different modalities at the same time, ignoring the enhancement of single modality and temporal consistency, resulting in color flickering problems in the fused static background or dynamic targets.

[0003] To achieve efficient dynamic scene reconstruction in infrared and visible light video fusion, existing solutions first use optical flow algorithms to temporally align and mask each video frame after video sequence acquisition. Subsequently, masks are used to separate the dynamic and static portions, and the static portions are used for frame-by-frame reconstruction to ensure high-quality reconstruction of the static scene. When processing the dynamic portion, the dynamic portion is first enhanced using a lookup table (LUT) before being fed into a multi-scale Transformer-U-Net model for dynamic feature extraction and fusion. A channel-wise attention mechanism fine-tunes the feature maps at each layer to ensure spatiotemporal consistency between dynamic objects and the static scene. Finally, the static and dynamic reconstruction results are combined and fused to produce a final video scene that exhibits temporal consistency and dynamic effects. Furthermore, existing technologies also introduce a Clip model to guide and optimize the representation of video features, further enhancing fusion performance. By understanding multimodal data, Clip adjusts the weights of various features in the video content, helping the model better adapt to different scenes and object dynamics, thereby achieving more refined fusion.

[0004] However, existing technical solutions still have several significant shortcomings in the fusion of infrared and visible light videos. Current fusion methods have significant differences in performance when processing different scenes, and are unstable under complex image conditions such as low light. This significantly affects the fusion effect. When processing noisy images, existing technologies typically rely on independently processing the dynamic and static components at each time point, failing to fully consider the consistency between perspectives. This approach leads to the following two problems:

[0005] Inconsistent image details: Because there is no information interaction in the temporal dimension, only the two modalities at the same moment are fused, ignoring the temporal logic information, which affects image quality, produces flickering enhancement effects, and causes visual distortion.

[0006] Long reconstruction time: Video fusion is more time-intensive than single-frame image fusion, and the internal data processing is more complex. Reconstruction delays are more noticeable in high-resolution video. However, processing dynamic objects separately can effectively reduce unnecessary calculations, shorten overall reconstruction time, and improve processing efficiency.

[0007] It is worth noting that current mainstream image fusion methods, such as U2Fusion (Liu et al., 2020), EMFusion / EMMA (Ma et al., 2022), and MetaFusion (Xu et al., 2023), have achieved promising results in static image tasks. However, none of these methods consider the temporal consistency of video sequences. In practical applications, U2Fusion employs an unsupervised image fusion framework that ignores temporal continuity, often resulting in color jumps. While EMMA incorporates a saliency-guided mechanism into image fusion, it lacks temporal modeling capabilities for dynamic scenes, resulting in content jumps and detail drift in the fused image. MetaFusion, while achieving modality-adaptive fusion through meta-learning, is also limited to the static image level and cannot handle dynamic content and style drift in videos. These shortcomings directly impact the performance of infrared and visible light video fusion in real-world scenarios.

[0008] Aiming at the above problems, the present invention proposes a new video fusion framework that combines dynamic / static separation, LUT enhancement, Transformer fusion and CLIP style guidance. It can significantly improve the video fusion quality in low-light and complex dynamic environments, and effectively solve the problems of timing inconsistency and difficulty in dynamic target modeling. Summary of the Invention

[0009] The present invention divides the fusion problem into two steps: static reconstruction and dynamic object enhancement. First, optical flow is used to extract the dynamic object mask from each frame and use it as a cue for SAM image segmentation. Once the complete mask is obtained, the dynamic objects are extracted from the image sequences of both modalities based on the mask, and the portion of the image corresponding to the first moment of the mask is removed. The resulting image with holes is reconstructed based on the dynamic mask of the subsequent frame. Holes that are not within the dynamic mask are filled. This allows for the generation of complete infrared and visible light static scenes. After the static scene is fixed, flicker in the static scene is eliminated. The extracted dynamic objects in the visible light modality are then enhanced using a LUT table, and the visible light image is subjected to temporal logic color consistency processing. One of the enhanced image sequences is selected as a style image to guide further color consistency. The dynamic and static visible light and infrared image sequences are then fed into an encoder based on the Transformer module to encode the infrared and visible light images. The decoder uses the same basic module as the decoder and uses the CLIP model as a style guide for the fusion effect. Finally, the dynamic target is attached to the static scene using the mask. This significantly improves the accuracy and speed of infrared and visible light video fusion. Compared to traditional technologies, this solution effectively solves the temporal alignment issues between different source videos and significantly improves the quality of the fused image.

[0010] Compared to traditional video fusion techniques, this solution effectively resolves timing synchronization issues and improves reconstruction quality by reconstructing dynamic and static components separately and performing temporal consistency processing. In particular, in multi-source video fusion, the optical flow module is used to precisely mask the dynamic components of each video sequence, effectively reducing the impact of dynamic background on static scene reconstruction. Combined with fine-tuning of a large language model, the technology's adaptability and fusion performance are further enhanced, especially in complex dynamic scenes, improving the accuracy and stability of the overall reconstruction.

[0011] The technical solution of the present invention:

[0012] The low-light infrared and visible light video fusion method based on LUT and Transformer is as follows: Figure 1 shown.

[0013] Step 1: Dynamic object mask extraction.

[0014] Step 2: Separate the static and dynamic parts.

[0015] Step 3: Dynamic target enhancement.

[0016] Step 4: Time logic consistency processing.

[0017] Step 5: Encoding and decoding processing.

[0018] Step 6: Style guidance and final fusion.

[0019] Step 7: Target synthesis.

[0020] Beneficial Effects of the Invention: The core of this proposal is a high-fidelity image reconstruction and fusion technology for infrared and visible light video fusion. Its key points and innovations to be protected include:

[0021] The key point of this application proposal is to solve multiple technical challenges in the fusion of infrared and visible light videos through innovative image processing methods. First, the optical flow algorithm extracts the mask of the dynamic object and separates it from the static scene, ensuring that the static part is not affected by the dynamic object. This method reduces the interference of the static scene by accurately identifying the dynamic object area, laying the foundation for subsequent image reconstruction. Secondly, when processing dynamic targets, the visual effect of dynamic objects is improved through LUT enhancement technology, ensuring that the image details and colors are more vivid and natural, thereby enhancing the expressiveness of the video.

[0022] Furthermore, this technology solution's unique temporal consistency processing method ensures consistent color and visual effects across different time frames in a video sequence, resolving the flickering issue common in low-light and complex scenes. This processing step not only improves image stability but also enhances the visual experience of the video.

[0023] By meticulously fusing dynamic objects with static backgrounds, using a Transformer-based encoder model, the fusion process further optimizes the spatiotemporal characteristics of the image. Using the CLIP model for style guidance ensures a consistent visual style, resulting in a more natural and commercially valuable fusion effect. Ultimately, the precise alignment of dynamic objects with static scenes ensures a seamless connection between video elements, enhancing the stability and accuracy of the fusion effect.

[0024] These technical solutions not only offer significant functional innovations, but also carry high infringement risks during implementation, particularly in key steps such as dynamic object mask extraction, temporal consistency processing, and image fusion. Through these innovative steps, this patent possesses strong technical protection value, effectively preventing circumvention by competitors.

[0025] This technical solution innovatively combines optical flow masking, dynamic / static segmentation, temporal consistency processing, multi-scale Transformer feature fusion, and large language model fine-tuning to address key issues in infrared and visible light video fusion, such as temporal consistency of dynamic objects and low-contrast image processing, thereby improving the accuracy and stability of the fusion effect. These innovations are highly protective and have high evidentiary value in infringement cases.

[0026] The technical advantage of this proposal lies in its innovative solution to several key technical challenges in infrared and visible light video fusion. First, an optical flow algorithm is used to precisely extract dynamic object masks, enabling accurate separation of dynamic targets from static backgrounds. This reduces the interference of dynamic objects in static scenes and improves the accuracy of subsequent image processing. Furthermore, LUT enhancement technology is employed to optimize dynamic targets, further enhancing their visual quality. The resulting fusion results in a more vivid and natural appearance, while preserving detail and realism.

[0027] Furthermore, the proposed temporal consistency processing ensures consistent color and visual quality across different time frames within a video, resolving issues such as color flickering and instability commonly seen in low-light or complex dynamic scenes. This processing significantly improves the visual quality of the video and ensures smooth image transitions.

[0028] In terms of fusion, this technical solution uses advanced encoding and decoding technologies. Through a Transformer-based encoder model, it accurately extracts the spatiotemporal features of infrared and visible light images and seamlessly fuses them. This method improves image processing efficiency while also enhancing image accuracy, ensuring high-quality fusion effects, especially in complex video scenes.

[0029] Finally, by introducing the CLIP model for style guidance, this proposal further optimizes the fusion of image features, ensuring stylistic consistency across images of different modalities. This fusion not only improves the quality of the final image but also enhances the consistency and coordination of visual styles, ensuring the naturalness and coherence of the video.

[0030] Overall, this proposal provides an efficient and accurate infrared and visible light video fusion method through a series of innovative technical steps, which significantly improves the image quality, stability and visual effects, and has strong application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the overall flow chart;

[0032] Figure 2 Schematic diagram of the fusion module. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described in detail below based on the accompanying drawings and embodiments:

[0034] Step 1: Dynamic object mask extraction

[0035] The optical flow algorithm is used to process each frame in the video sequence. The optical flow algorithm calculates the motion vector of each pixel based on the two frames before and after the image to obtain the motion vector field. , the motion vector can be obtained by the following formula:

[0036] in, For location The motion vector of the pixel at is the horizontal coordinate of the pixel in the image, is the horizontal displacement of the pixel, is the vertical coordinate of the pixel in the image, is the vertical displacement of the pixel.

[0037] These motion vectors can help detect which areas in the image have undergone dynamic changes. , extract the significant motion area, that is, the location of the dynamic object, and generate a mask map of the dynamic object The mask image is used to mark which pixels in the image belong to the dynamic object area and can be defined as:

[0038]

[0039] Step 2: Separate the static and dynamic parts

[0040] In order to achieve stability in subsequent image fusion and reconstruction, it is necessary to effectively separate the infrared and visible static background parts from the dynamic target parts in the original image. This process relies on the dynamic object mask Identify the image area, where:

[0041] Indicates that the frame contains only static background areas. Represents the original image.

[0042] Step 3: Dynamic Target Enhancement

[0043] Perform LUT (lookup table) enhancement on dynamic targets in the visible light modality to improve the visual effect of dynamic targets. The LUT enhancement operation adjusts the color curve to make the dynamic targets more vivid and vivid visually. At the same time, LUT enhancement can retain the details of the image, making the enhanced dynamic targets look more natural and realistic. The purpose of this step is to make the dynamic objects more prominent in the final fusion effect and enhance the visual perception quality of the image. Perform LUT (lookup table) enhancement on dynamic targets in the visible light modality, and the enhanced color values Convert via lookup table:

[0044]

[0045] Among them, the LUT function maps the original pixel value to the enhanced color value.

[0046] Step 4: Time logic consistency processing

[0047] Temporal consistency processing is performed on the enhanced visible light image to ensure that the color and visual effects remain consistent between different time frames in the video sequence, avoiding visual discomfort or image distortion caused by temporal discontinuity. In the specific implementation process, this paper introduces a temporal logic consistency modeling method based on multi-scale style guidance to achieve continuity and stability of color style between frames. The method is as follows:

[0048] Step 4.1: First, select a frame from the enhanced visible light image sequence as a style reference frame. A representative keyframe is selected and recorded as the reference style image. The reference style image is then passed through a five-stage downsampling module to extract multi-scale color style features. A global guided feature vector is generated at the deepest layer for controlling style propagation.

[0049] In step 4.2, each enhanced visible light image and infrared image to be aligned is downsampled using multiple Transformer modules to extract spatial features. Except for the last level of spatial features, the remaining infrared and visible light spatial features are expressed through a convolutional neural network (CNN) encoding structure. The last level of infrared and visible light spatial features are sequentially processed through the cross-attention, convolution, channel attention, and Transformer modules.

[0050] Step 5: Feature guidance and dynamic target fusion

[0051] During the feature fusion phase, the last layer of convolutional features first enhances the interaction between infrared and visible light features through the cross-attention mechanism described in step 4.2. The resulting convolutional features are then weighted using a channel-wise attention mechanism to reconstruct the weight distribution between channels and achieve adaptive adjustment of the importance of feature channels. In addition to the features of the last layer, the decoder fuses the upsampled features from the next decoder level with the features extracted by the convolutional network at the current level to generate the feature representation of the current decoder, thereby gradually completing the integrated reconstruction of multi-scale, semantic, and structural information.

[0052] 5.1. The text semantic guidance features obtained by the CLIP model and the feature map obtained by downsampling the visible light image four times are used for text guidance and image style guidance respectively to optimize the fusion effect of image features. The CLIP model dynamically adjusts the weights of different features by analyzing the multimodal data of images and text, where the text is manually entered by the user according to specific task requirements. Image style guidance improves the color consistency of the image, further enhancing the overall visual effect and ensuring that the final fusion result meets the expected effect. The process expression of feature guidance is as follows:

[0053]

[0054] in, Represents the adjustment weight of each feature map element, MLP represents the multi-layer perceptron, Represents the global guided feature obtained in step 4, that is, the feature extracted based on the image color style. Represents the text semantics guided features generated by the CLIP model.

[0055] 5.2 Using the style feature Guiding infrared target images With visible light target image The specific fusion process is as follows:

[0056]

[0057] in is the final dynamic target fusion result; represents the fused features processed and upsampled by the lower decoder; It represents the encoder output feature corresponding to the current level, which is extracted jointly by the encoder and the convolution module; Indicates feature fusion.

[0058] Step 6: Target synthesis:

[0059] The goal is to finally fuse the processed dynamic target area with the static background area to generate a complete image frame with consistent time and space and unified style. First, the static infrared and visible background images extracted in step 2 are To perform the fusion:

[0060]

[0061] in, For the final fused image, is the fused image of the static part.

[0062] This solution uses an optical flow algorithm to generate a mask, accurately separating dynamic and static components and ensuring temporal consistency. Compared to existing technologies, this method more effectively addresses the timing synchronization issue between multi-view videos, avoiding the detail inconsistencies that can occur when processing dynamic scenes. To this end, an interactive comparison tool can be developed to collect multiple video sequences for temporal consistency testing. The superiority of this technology can be verified by calculating the temporal differences and image quality consistency between video frames.

[0063] To separate and reconstruct dynamic and static scenes, this solution uses a multi-scale Transformer module to extract and fuse features from the dynamic portion, ensuring precise alignment between dynamic objects and the static background. This process improves image clarity and stability in complex environments. To verify this, the processing effects of the dynamic and static portions can be extracted and compared with the results of unified processing using current technology, demonstrating significant differences in image quality. This can be achieved by comparing video fusion effects under varying lighting and background conditions, focusing on verifying the advantages of the multi-scale Transformer module.

[0064] In addition, the solution introduces a large language model (such as CLIP) for fine-tuning, which can optimize the expression of fused features and enhance the adaptability of images in complex environments. Existing technologies do not fully utilize the advantages of large language models, resulting in limited performance under conditions such as high exposure and low contrast. This solution can maintain good image quality in these complex scenes. To verify this innovation, an automated testing platform can be developed to test and record the fusion effects of different models under the same conditions using standardized low-contrast or high-exposure video sequences, with a focus on detecting the clarity, stability, and consistency of images in high-exposure or low-light environments.

[0065] Finally, this technical solution achieves high-fidelity video reconstruction and fusion through innovative methods such as optical flow mask generation and large language model fine-tuning. This is particularly effective in handling dynamic scenes and complex lighting issues when fusion is performed on multiple sources. To verify this, a dataset of noisy 2D image inputs can be established to test and verify the quality of image fusion under different noise models. Quantitative analysis (such as PSNR and SSIM) demonstrates the advantages of this technical solution in terms of image quality.

[0066] In summary, by combining temporal consistency processing, dynamic and static scene separation and reconstruction, and CLIP optimization, this solution provides an innovative solution for video image fusion. Its technical approach is clear and its innovations are prominent. Through interactive comparison tools and automated testing platforms, we can effectively verify the superiority of this technical solution and ensure the implementation and evidence collection of its patent protection.

[0067] For separating dynamic and static scenes, in addition to using optical flow algorithms, we can also try combining semantic segmentation techniques based on deep learning, leveraging deep neural networks to automatically identify and segment dynamic and static scenes in videos. This approach not only improves separation accuracy but also reduces reliance on traditional algorithms. Through deep network training, the system can more intelligently identify dynamic objects and backgrounds, thereby improving image processing performance, especially in complex scenes.

[0068] Furthermore, in terms of multi-scale Transformer feature fusion, the currently used channel attention mechanism is already effective in optimizing feature maps, but other mechanisms could also be explored, such as the self-attention mechanism, which can better capture information relevance within the image and provide stronger feature alignment capabilities, especially when processing complex scenes. The self-attention mechanism can reduce dependence on external input, thereby improving the model's adaptability, especially in low-contrast or complex lighting environments.

[0069] When fine-tuning large language models like CLIP, competitors may consider using other pre-trained language models, such as OpenAI's GPT series or BERT series. These models offer more granular understanding and optimization across different domains. Choosing the right pre-trained model for each application scenario can further enhance the expressive power of video content, thereby optimizing image fusion. This flexibility in model selection will enable the system to achieve greater adaptability and performance under diverse conditions.

[0070] In terms of optimization algorithms, competitors may try different optimization strategies or improve existing loss functions to further enhance model performance, particularly in terms of training efficiency and inference capabilities on large-scale datasets. These improvements are expected to further enhance the quality of image fusion and the processing speed of the system, particularly when dealing with complex video data, providing more accurate and stable processing results.

Claims

1. A low-light infrared and visible light video fusion method based on LUT and Transformer, characterized by: The following steps are involved: Step 1: Dynamic object mask extraction; Step 2: Separate the static and dynamic parts; Step 3: Dynamic target enhancement; Step 4: Time logic consistency processing; Step 5: Encoding and decoding processing; Step 6: Style guidance and final fusion; Step 7: Target synthesis.

2. The low-light infrared-visible light video fusion method based on LUT and Transformer according to claim 1 is characterized in that: Step 1 is as follows: The optical flow algorithm is used to process each frame in the video sequence. The optical flow algorithm calculates the motion vector of each pixel based on the two frames before and after the image to obtain the motion vector field. , the motion vector can be obtained by the following formula: in, For location The motion vector of the pixel at is the horizontal coordinate of the pixel in the image, is the horizontal displacement of the pixel, is the vertical coordinate of the pixel in the image, is the vertical displacement of the pixel; By analyzing the motion vector field of consecutive frame images , extract the significant motion area, that is, the location of the dynamic object, and generate a mask map of the dynamic object ; The mask image is used to mark which pixels in the image belong to the dynamic object area and can be defined as: 。 3. The low-light infrared-visible light video fusion method based on LUT and Transformer according to claim 1 is characterized in that: Step 2 is as follows: Effectively separate the infrared and visible static background parts from the dynamic target parts in the original image; this process relies on the dynamic object mask Identify the image area, where: , Indicates that the frame image only contains the static background area; Represents the original image.

4. The low-light infrared-visible light video fusion method based on LUT and Transformer according to claim 1 is characterized in that: Step 3 is as follows: Perform LUT enhancement on dynamic targets in visible light mode, and the enhanced color value Convert via lookup table: ， Among them, the LUT function maps the original pixel value to the enhanced color value.

5. The low-light infrared-visible light video fusion method based on LUT and Transformer according to claim 1, characterized in that: Step 4 is as follows: Step 4.1: First, a frame is selected from the enhanced visible light image sequence as a style reference frame and recorded as the reference style image. Then, the reference style image is continuously down-sampled to extract multi-scale color style features, and a global guided feature vector is generated at the deepest layer for controlling style propagation. Step 4.2: Use multiple Transformer modules to downsample each frame of the enhanced visible light image and each frame of the infrared image to be aligned to extract spatial features. Except for the last level of spatial features, the remaining infrared and visible light spatial features of each level are expressed through the convolutional neural network (CNN) encoding structure; the last level of infrared and visible light spatial features are sequentially operated through cross attention, convolution, channel attention, and Transformer modules.

6. The low-light infrared-visible light video fusion method based on LUT and Transformer according to claim 1, characterized in that: Step 5 is as follows: In the feature fusion stage, the last layer of convolutional features first enhances the interaction between infrared and visible light features through the cross-attention mechanism described in step 4.2, and the features obtained after convolution are weighted through the channel attention mechanism to reconstruct the weight distribution between channels and achieve adaptive adjustment of the importance of feature channels. In addition to the features of the last layer, the decoder fuses the upsampled features from the next level decoder with the features extracted by the convolutional network of the current level to generate the feature representation of the current decoder, thereby gradually completing the integrated reconstruction of multi-scale, semantic and structural information. 5.

1. The text semantic guidance features obtained by the CLIP model and the feature map obtained by downsampling the visible light image four times are respectively used for text guidance and image style guidance to optimize the fusion effect of image features. The CLIP model dynamically adjusts the weights of different features by analyzing the multimodal data of images and text. The text is manually entered by the user according to specific task requirements. The image style guidance improves the color consistency of the image, further enhancing the overall visual effect and ensuring that the final fusion result meets the expected effect. The process expression of feature guidance is as follows: ， in, Represents the adjustment weight of each feature map element, MLP represents the multi-layer perceptron, Represents the global guided feature obtained in step 4, that is, the feature extracted based on the image color style. Represents the text semantics guided features generated by the CLIP model; 5.2 Using the style feature Guiding infrared target images With visible light target image The specific fusion process is as follows: ， in is the final dynamic target fusion result; represents the fused features processed and upsampled by the lower decoder; It represents the encoder output feature corresponding to the current level, which is extracted jointly by the encoder and the convolution module; Indicates feature fusion.

7. The low-light infrared-visible light video fusion method based on LUT and Transformer according to claim 1, characterized in that: Step 6 is as follows: It aims to finally fuse the processed dynamic target area with the static background area to generate a complete image frame with consistent time and space and unified style; first, the static infrared and visible background images extracted in step 2 are To perform the fusion: ， in, For the final fused image, is the fused image of the static part.

Images