Method for converting end-to-end lightweight visible light image into infrared image
Through an end-to-end lightweight method, combined with enhanced feature mapping, color perception and dynamic fusion modules, the fabrication and artifact problems in visible light images to infrared image conversion are solved, and high-quality infrared image generation is achieved, which significantly improves image contrast and clarity.
Patent Information
- Application Number
- CN202410959643.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-01
AI Technical Summary
The prior art has fabrication and irregular visual artifacts in the conversion of visible light images to infrared images, especially in areas without objects, which are difficult to effectively solve the problems of low contrast and blurred edges.
Using an end-to-end lightweight approach, the texture and color features of visible light images are extracted and fused by enhanced feature mapping modules, color-aware adapters, dynamic fusion modules, color-aware adapters and enhanced attention modules. , and transpose it to the infrared pixel domain, retaining and highlighting key texture details, using a dual loss function strategy to enhance image contrast.
It significantly improves the clarity and information content of infrared images, reduces artifact phenomena, improves the quality and contrast of image conversion, and has low calculation overhead.
Smart Images

Figure CN118887484A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing and relates to an image conversion method, and specifically to a lightweight visible light image to infrared image conversion method based on Transformer. Background Art
[0002] During visible light imaging, in some cases the acquired images have low contrast or certain objects have reduced reflectivity in the visible light spectrum. Infrared images (IR) can alleviate these problems, providing enhanced contrast and detectability of objects regardless of their visible light reflectivity. For example, visible light images acquired in low-light conditions are prone to problems such as low contrast, blurred edges, and lost details. Fields with high demand for edge information - including pedestrian detection, autonomous driving, robotic navigation, and precision agriculture - may see substantial improvements from the superior edge clarity provided by infrared imaging. Such enhancements are particularly important for safety-critical applications such as autonomous driving. However, the cost of infrared imaging equipment remains high compared to widely used visible light cameras.
[0003] Related technologies have deficiencies in conversion, for example, GANs have shown superior performance than convolutional neural networks (CNNs) in image generation tasks. Applications of image translation using GANs, such as segmentation, semantic to visual conversion, super-resolution, image synthesis, and style transfer, have shown significant success in papers. However, when GANs are used for VIS-IR (visible to infrared) image conversion, the results may not be ideal. Although GAN models can effectively extract contrast features from infrared images, they are often prone to misjudging features related to photometry, resulting in problems such as fabricated situations and irregular visual artifacts in areas where no objects exist.
[0004] Based on this, there is an urgent need for an end-to-end lightweight visible light image to infrared image conversion method to solve the above technical problems. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose an end-to-end lightweight visible light image to infrared image conversion method to solve the technical problems existing in the prior art such as fabrication and irregular visual artifacts in areas where no objects exist.
[0006] To achieve the above object, the present invention comprises the following steps:
[0007] Step S1: Obtain visible light image data to be converted;
[0008] Step S2: inputting the visible light image to be converted collected in the above S1 into the enhanced feature mapping module and the color perception adapter, so as to extract the texture and color features in the visible light image;
[0009] Step S3: The texture and color features extracted in S2 are fused through a dynamic fusion module;
[0010] Step S4: Transpose the fused features in S3 to the infrared pixel domain through a color perception adapter;
[0011] Step S5: The result of S4 is passed through an enhanced perceptual attention module to retain and highlight key texture details, ultimately forming an infrared image with better clarity and information content;
[0012] Step S6: Using the Transformer architecture to enhance the integration of information extrapolated by previous modules is beneficial for completing and improving the infrared image reconstruction process.
[0013] Furthermore, the step S2 specifically includes:
[0014] Step S21: extract infrared color features from the visible light image through a color perception adapter. The working process of the color perception adapter is to divide the input features into four different parts, namely, three RGB channels of visible light and the infrared component contained in the visible light image, and then extract features from the RGB channels through convolution and fuse them with the infrared features;
[0015] Step S22: Enhance the details and granularity by using an enhanced feature mapping module. The enhanced feature mapping module obtains multi-scale feature representation by downsampling at various scales.
[0016] Furthermore, the dynamic fusion aggregation module emphasized in step S3 is characterized in that the module is responsible for fusing the features of visible light and projecting them into the latent space. We construct an attention matrix, which is provided by the maximum and average values of the matrix set. The subsequent feature mapping is performed through this attention-driven mechanism. The process is performed through the following formula:
[0017] f=[MaxPoold0(x);AvgPoold0(x)]
[0018] y=x×σ(f(x))
[0019] Where f represents the attention matrix and σ represents the logic function
[0020] Furthermore, the color perception adapter emphasized in step S4 divides the input features into four different parts, namely the three RGB channels of visible light and the infrared component contained in the visible light image, and then extracts features from the RGB channels through convolution and fuses them with the infrared features; the features extracted in the previous process are transposed to the infrared pixel domain.
[0021] Furthermore, the enhanced perceptual attention module emphasized in step S5 is characterized by using a dual attention system consisting of channel and pixel attention strategies to refine and enhance the feature set of the latent space. This dual mechanism ensures that each pixel is adjusted to retain and highlight key texture details, ultimately forming an infrared image with better clarity and information content, specifically including:
[0022] S51: Compensate for information loss caused by low light conditions or occlusion by enhancing feature representation using the following formula:
[0023] Y=x+Conv(X1⊙X2)×W1
[0024] Where Conv represents the convolution operation, X1 and X2 are two identical feature maps obtained by splitting the channels of X, and W1 represents a set of custom learnable weights.
[0025] S52: The channel attention mechanism (CA) can capture global information. It first compresses the spatial information into the channel dimension, and then applies a multi-layer perceptron to calculate the channel attention for weighted feature maps. The formula is as follows:
[0026] CA(X)=X×σ(W2 max(0,W1pool(X)))
[0027] Where σ is a nonlinear activation function, Sigmoid, W1, and W2 are fully connected layers
[0028] S53: The pixel attention mechanism (PA) can adaptively adjust the weight of each pixel, so that the model can better focus on important pixels, thereby capturing subtle changes and details in the image and extracting more informative features. The formula used is as follows:
[0029] PA(X)=X×σ(W4 max(0,W3(X)))
[0030] Where σ is a nonlinear activation function, Sigmoid, W3, and W4 are fully connected layers
[0031] Furthermore, in order to better preserve structural information and enhance image contrast, our model adopts a dual loss function strategy, which combines the advantages of the Smooth Ll loss function with the structural similarity index measurement LSSIM, using the following formula:
[0032]
[0033] Where μ x and μ y represents the average of x and y, and represents the variance of x and y, σ xy is the covariance of x and y, C1=k1L) 2 、C2=(k2L) 2 are two constants, where L represents the dynamic range of the image data and L represents the total loss function. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1 It is a flow chart for realizing the present invention;
[0036] Figure 2 The overall structural diagram of the model constructed by the present invention;
[0037] Figure 3 This is a schematic diagram of the structure of the color perception adapter used in the present invention;
[0038] Figure 4 This is a schematic diagram of the structure of the enhanced feature mapping module used in the present invention;
[0039] Figure 5 This is a schematic diagram of the structure of the dynamic fusion aggregation module used in the present invention;
[0040] Figure 6 This is a schematic diagram of the structure of the enhanced perception attention module adopted in the present invention. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0042] like Figure 1As shown, the present invention provides an end-to-end lightweight visible light image to infrared image conversion method, comprising the following steps:
[0043] S1: Obtain visible light image data to be converted, such as obtaining visible light images from mobile devices such as mobile phones and cameras, and then input the images into the model;
[0044] S2: Extracting texture and color features from visible light images: The visible light image to be converted collected in S1 is input into an enhanced feature mapping module and a color perception adapter, wherein the color perception adapter extracts infrared color features from the visible light image, and the enhanced feature mapping module enhances details and granularity;
[0045] S3: Fusion of visible light features and projection into latent space: The texture and color features extracted in S2 are fused through a dynamic fusion module, which is carefully designed to ensure seamless transition of visual information and maintain the integrity of the original features;
[0046] S4: Transpose features to infrared pixel domain: The fused features in S3 are transposed to the infrared pixel domain through the color-aware adapter. Through this process, the module learns to adapt to these features, enhancing the model's ability to interpret and reconstruct the infrared aspects of the input;
[0047] S5: Preserve and highlight key texture details: The results of S4 are passed through the enhanced perceptual attention module, which focuses on compensating for information loss in low-light environments or areas with obstructed vision, retaining and highlighting key texture details, and finally forming an infrared image with better clarity and information content;
[0048] S6: Generate infrared images: Using the Transformer architecture to enhance the integration of information extrapolated from previous modules is beneficial for completing and improving the infrared image reconstruction process.
[0049] In some implementations, the specific process of step S2 is as follows:
[0050] S21: See Figure 3 , through the color perception adapter, the features are extracted from the visible light image. The infrared color features exist in the original image but cannot be perceived by the human eye. The working process of the color perception adapter is to divide the input features into four different parts, namely the three RGB channels of visible light and the infrared component contained in the visible light image, and then the RGB channels are extracted through convolution and fused with the infrared features;
[0051] S22: See Figure 4By enhancing the details and granularity through the enhanced feature map module, such rich features are inherently more discriminative and informative, thus providing substantial benefits for subsequent tasks such as object detection and image classification. The enhanced feature map module obtains multi-scale feature representation by adopting downsampling at various scales.
[0052] In some implementations, the specific process of the dynamic fusion aggregation module described in step S3 above is:
[0053] See also Figure 5 ,The dynamic fusion aggregation module is responsible for fusing the features of visible light and projecting them into the latent space. We construct an attention matrix, which is provided by the maximum and average values of the matrix set. The subsequent feature mapping is carried out through this attention-driven mechanism, which ensures a seamless transition of visual information, maintains the integrity of the original features, and prepares for accurate infrared representation. The process is carried out through the following formula:
[0054] f=[MaxPoold0(x);AvgPoold0(x)]
[0055] y=x×σ(f(x))
[0056] Where f represents the attention matrix and σ represents the logic function
[0057] In some embodiments, the specific process of the color perception adapter described in step S4 is as follows:
[0058] The input features are divided into four different parts, namely the three RGB channels of visible light and the infrared component contained in the visible light image. The RGB channels are then convolved to extract features and then fused with the infrared features. The features extracted in the previous process are transposed to the infrared pixel domain. Through this process, the module learns to adapt to these features, enhancing the model's ability to interpret and reconstruct the infrared aspects of the input.
[0059] In some embodiments, the enhanced perception attention module described in step S5 above focuses on compensating for information loss in low-light environments or areas with obstructed vision. This module uses a dual attention system consisting of channel and pixel attention strategies to improve and enhance the feature set of the latent space. This dual mechanism ensures that each pixel is adjusted to retain and highlight key texture details, ultimately forming an infrared image with better clarity and information content. The specific process is as follows:
[0060] S51: Select features that transform infrared images from the latent space to generate higher quality infrared images. This module aims to enhance feature representation to compensate for information loss caused by low light conditions or occlusions, using the following formula:
[0061] Y=x+Conv(X1⊙X2)×W1
[0062] Where Conv represents the convolution operation, X1 and X2 are two identical feature maps obtained by splitting the channels of X, and W1 represents a set of custom learnable weights.
[0063] S52: The channel attention mechanism (CA) can capture global information and has high computational efficiency. It first compresses the spatial information into the channel dimension, and then applies a multi-layer perceptron to calculate the channel attention for weighted feature maps. The formula used is as follows:
[0064] CA(X)=X×σ(W2 max(0,W1pool(X)))
[0065] Where σ is a nonlinear activation function, Sigmoid, W1, and W2 are fully connected layers
[0066] S53: The pixel attention mechanism (PA) can adaptively adjust the weight of each pixel, so that the model can better focus on important pixels, thereby capturing subtle changes and details in the image and extracting more informative features. The formula used is as follows:
[0067] PA(X)=X×σ(W4 max(0,W3(X)))
[0068] Where σ is a nonlinear activation function, Sigmoid, W3, and W4 are fully connected layers
[0069] The features in CA are processed by pooling, reducing the feature size from (C, H, W) to (C, 1, 1). The size of the final convolution output feature is (C, 1, 1). The features in PA are processed by convolution, converting the (C, H, W) features to (1, H, W).
[0070] In some embodiments, in order to better preserve structural information and enhance image contrast, our model adopts a dual loss function strategy, which combines the advantages of the Smooth Ll loss function with the structural similarity index measurement LSSIM, which aims to evaluate the quality of an image by considering the brightness, contrast, and structural similarity of the image, using the following formula:
[0071]
[0072] L(x,y)=Lsmooth(x)+LSSIM(x,y)
[0073] Where μ x and μ y represents the average of x and y, and represents the variance of x and y, σxy is the covariance of x and y, C1=k1L) 2 、C2=(k2L) 2 are two constants, where L represents the dynamic range of the image data and L represents the total loss function.
[0074] The following is a further description of the technical effects of the present invention in conjunction with the simulation results:
[0075] 1. Simulation conditions and contents:
[0076] Our model uses the PyTorch neural network framework to process input images in the visible and infrared spectrum, each with a resolution of 256×256 pixels. To fine-tune the network’s hyperparameters, we start with a 2×10 -4 Starting with an initial learning rate of , and using the AdamW optimizer
[0077] In order to dynamically adjust the learning rate throughout the training process, the CosineAnnealingLR scheduler is used.
[0078] The training was performed for more than 400 rounds to ensure thorough learning and convergence of the model. We applied a batch size of 1, which may increase the computation time, but it leads to more accurate gradient calculations, thus improving learning efficiency and model stability.
[0079] The training sets from five independent datasets, including LLVIP, RoadScene, M3FD, FLIR, and MCubeS, are combined into a comprehensive training program. This integration aims to expand the diversity of the data and improve the generalization of the model. Peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) are used as evaluation metrics. PSNR determines image quality by calculating the mean square error (MSE) between the original image and its distorted image (e.g., compressed version)
[0080] To evaluate the effectiveness of our method, we conducted comparative experiments using the five datasets mentioned above. Our method has been benchmarked against several infrared image generation algorithms: CycleGAN, Pix2Pix, UNIT, MUNIT, ThermalGAN, BCI, InfraGAN, and ClawGAN.
[0081] An extensive and objective evaluation of the experimental results is presented in Table I. The table collects and quantifies the performance metrics of each algorithm, thus facilitating a more comprehensive and quantitative analysis of its effectiveness.
[0082] Table I
[0083]
[0084] Compared with the existing technology, the peak signal-to-noise ratio and structural similarity of the present invention on different test sample sets are significantly improved, and only 0.04M parameters and 2.41G MA are used, maintaining a low computational overhead.
[0085] Based on the analysis of the results in the above simulation experiments, this paper proposes a Transformer-based model that can effectively convert visible light images into infrared images. The dynamic fusion aggregation module and enhanced perception attention module proposed in this paper work together to effectively capture and retain key texture and color features, which are then accurately converted to the infrared domain. The effectiveness of our model is demonstrated through extensive benchmark experiments, which significantly outperform existing methods in producing qualitatively and quantitatively superior infrared images.
[0086] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
Claims
1. An end-to-end lightweight visible light image to infrared image conversion method, characterized in that The following steps are involved: S1: Obtain visible light image data to be converted; S2: Inputting the visible light image to be converted collected in the above S1 into the enhanced feature mapping module and the color perception adapter, so as to extract texture and color features in the visible light image; S3: The texture and color features extracted in S2 are fused through the dynamic fusion module; S4: Transpose the fused features in S3 to the infrared pixel domain through the color perception adapter; S5: The result of S4 is passed through the enhanced perceptual attention module to retain and highlight key texture details, ultimately forming an infrared image with better clarity and information content; S6: Using the Transformer architecture to enhance the integration of information extrapolated from previous modules is beneficial for completing and improving the infrared image reconstruction process.
2. The end-to-end lightweight visible light image to infrared image conversion method according to claim 1, characterized in that: The step S2 specifically includes: S21: Extract infrared color features from visible light images through a color-aware adapter. The color-aware adapter works by dividing the input features into four different parts, namely, the three RGB channels of visible light and the infrared component contained in the visible light image, and then fusing the RGB channels with the infrared features after extracting features through convolution. S22: Enhance the details and granularity by enhancing the feature mapping module; the enhanced feature mapping module obtains multi-scale feature representation by downsampling at various scales.
3. According to the end-to-end lightweight visible light image to infrared image conversion method of claim 1, the dynamic fusion aggregation module emphasized in step S3 comprises: This module is responsible for fusing the features of visible light and projecting them into the latent space. We construct an attention matrix that is informed by the maximum and average values of the matrix set. Subsequent feature mapping is performed through this attention-driven mechanism. The process is performed through the following formula: f=[MaxPoold0(x);AvgPoold0(x)] y=x×σ(f(x)) In the above formula, f represents the attention matrix and σ represents the logistic function.
4. According to the end-to-end lightweight visible light image to infrared image conversion method of claim 1, the color perception adapter emphasized in step S4 comprises: The color perception adapter divides the input features into four different parts, namely the three RGB channels of visible light and the infrared component contained in the visible light image, and then extracts the features of the RGB channels through convolution and fuses them with the infrared features; Transpose the features extracted in the previous process to the infrared pixel domain.
5. According to the end-to-end lightweight visible light image to infrared image conversion method of claim 1, the enhanced perception attention module emphasized in step S5 is characterized in that Utilize a dual attention system consisting of channel and pixel attention strategies to refine and enhance the feature set in the latent space; This dual mechanism ensures that each pixel is adjusted to preserve and accentuate critical texture details, resulting in an infrared image with superior clarity and information content, including: S51: Compensate for information loss caused by low light conditions or occlusion by enhancing feature representation using the following formula: Y=x+Conv(X1⊙X2)×W1 In the above formula, Conv represents the convolution operation, while X1 and X2 are two identical feature maps obtained by splitting the channels of X, and W1 represents a set of custom learnable weights; S52: The channel attention mechanism (CA) can capture global information. It first compresses the spatial information into the channel dimension, and then applies a multi-layer perceptron to calculate the channel attention for weighted feature maps. The formula is as follows: CA(X)=X×σ(W2 max(0,W1pool(X))) In the above formula, σ is a nonlinear activation function, Sigmoid, W1, and W2 are fully connected layers; S53: The pixel attention mechanism (PA) can adaptively adjust the weight of each pixel, so that the model can better focus on important pixels, thereby capturing subtle changes and details in the image and extracting more informative features. The formula used is as follows: PA(X)=X×σ(W4 max(0,W3(X))) In the above formula, σ is a nonlinear activation function, Sigmoid, W3, and W4 are fully connected layers.
6. According to the end-to-end lightweight visible light image to infrared image conversion method of claim 1, in order to better preserve structural information and enhance image contrast, our model adopts a dual loss function strategy, which combines the advantages of the Smooth Ll loss function with the structural similarity index measurement LSSIM, and adopts the following formula: L(x,y)=L smooth (x)+L SSIM (x,y) In the above formula, μ x and μ y represents the average of x and y, and represents the variance of x and y, σ xy is the covariance of x and y, C1=(k1L) 2 、C2=(k2L) 2 are two constants, where L represents the dynamic range of the image data and L represents the total loss function.
