An infrared-to-visible image conversion method based on object reflection prior guidance
By using a priori guidance method based on object reflection, infrared images are decomposed into illumination and reflection components. High-quality visible light images are generated using image segmentation and estimation networks. This solves the problems of data acquisition difficulties and illumination adaptability in infrared-to-visible light conversion, and improves the quality and applicability of image conversion.
Patent Information
- Application Number
- CN202411430269.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2044-10-14
AI Technical Summary
Existing infrared-to-visible image conversion methods suffer from difficulties in acquiring paired training data, color deviation and loss of detail in the conversion results, and insufficient processing capabilities in dynamic lighting scenarios.
An object reflection prior-guided method is adopted. The infrared image is decomposed into illumination and reflection components through an image decomposition network. The object reflection prior is obtained by an image segmentation network. These components are then processed by illumination estimation and reflection guidance networks respectively. Finally, a high-quality visible light image is generated through an image reconstruction network.
It improves color consistency and detail reproduction in image conversion, enhances the model's adaptability under different lighting conditions, and generates visible light images that are closer to the real effect. It is suitable for fields such as medical diagnosis, agricultural monitoring, traffic management, driver assistance and monitoring systems.
Smart Images
Figure CN119251270B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to an infrared-to-visible image conversion method based on object reflection prior guidance and belongs to the technical field of computer image processing. BACKGROUND
[0002] With the rapid development of deep learning technology in the field of image processing, infrared-to-visible (NIR2VIS) conversion has become a popular research direction, aiming to convert infrared (NIR) images into more rich and intuitive visible light (VIS) images. This technology has a wide range of applications in medical diagnosis, agricultural monitoring, traffic management, auxiliary driving, monitoring systems and other fields.
[0003] Although infrared imaging technology is widely used due to its unique ability to penetrate the atmosphere and its characteristics of not being detected by the human eye, infrared images often lack color and detail information that is more intuitive to human vision, which limits its application potential in intuitive visual presentation and advanced visual tasks.
[0004] Early infrared-to-visible conversion methods rely heavily on traditional image processing techniques such as histogram matching and rule-based color mapping. These methods often rely on precise prior knowledge and a large amount of manual adjustment. With the introduction of convolutional neural networks (CNN) and generative adversarial networks (GAN), the research on infrared-to-visible conversion gradually shifted to data-driven methods, which learn the mapping relationship between a large number of infrared and visible light image pairs to achieve conversion. However, one of the main challenges faced by this type of method is how to obtain a large number of accurately aligned infrared-to-visible image pairs, which is often difficult to achieve in practical applications.
[0005] To solve the problem of paired training data acquisition, some researchers began to explore unsupervised learning methods. Such as CycleGAN, which allows the model to learn infrared-to-visible image conversion without paired data. Although this type of method alleviates the limitations of training data to some extent, the conversion results often have color deviation and detail loss problems.
[0006] Although these technologies have improved the performance of infrared-to-visible conversion, they still face many challenges, including the physical reality of the conversion process, the generalization ability of the model, and the handling of dynamic lighting scenes. Therefore, accurately modeling the mapping relationship between infrared and visible light and generating high-quality visible light images remains a problem to be solved. SUMMARY
[0007] The object of the present application is to overcome the deficiencies and shortcomings of the existing infrared-to-visible image conversion method, and creatively propose an infrared-to-visible image conversion method and system based on object reflection prior guidance.
[0008] The present application can fully utilize the image segmentation result as object reflection prior, decompose the image into illumination component and object-specific reflection component, and then recover the illumination and reflection component of the visible light, to realize high-quality infrared-to-visible image conversion under various illumination conditions.
[0009] The present application adopts the following technical scheme.
[0010] An infrared-to-visible image conversion method based on object reflection prior guidance, comprising the following steps:
[0011] Step 1: Construct an object reflection prior guided infrared-to-visible image conversion network.
[0012] The object reflection prior guided infrared-to-visible image conversion network includes an image decomposition network, an image segmentation network, an illumination estimation network, a reflection guidance network and an image reconstruction network.
[0013] Firstly, the target input of the image decomposition network based on the residual network and self-attention is the original single-channel infrared image, and the target output is the single-channel environmental infrared illumination component and the single-channel object infrared reflection component.
[0014] This decomposition takes into account that the illumination is the light energy irradiated to the object in the scene, which is independent of the object in the scene. The reflection component is specific to the object, reflecting the reflectivity of the object in different spectral bands, and is independent of the environment to show different reflection characteristics in the infrared and visible light spectral range.
[0015] Specifically, the image decomposition network includes a residual network module and a self-attention network module. By introducing a residual network module in the network, the network can learn more complex and abstract feature representations. The self-attention network module calculates the correlation between different positions in the image, so that the network can capture long-distance dependencies, which helps to more accurately decompose the illumination and reflection characteristics in the image.
[0016] At the same time, the image segmentation network uses scene segmentation to process the input infrared image to obtain the image segmentation result, which is used as object reflection prior to guide the recovery of object visible light reflection.
[0017] Secondly, the illumination estimation network based on encoder and decoder architecture maps the infrared illumination component obtained from the image decomposition network to the visible light image's illumination component; the reflection guiding network based on self-attention uses the object reflection prior obtained from the image segmentation network to map the infrared reflection component to the visible light image's reflection component.
[0018] Finally, the image reconstruction network reconstructs a high-quality visible light image by performing a point multiplication operation on the processed illumination component and reflection component. This process not only emphasizes the importance of object reflection characteristics in infrared-to-visible light conversion, but also enhances the model's adaptability and accuracy under different lighting conditions.
[0019] Step 2: Use the loss function to train the object reflection prior guided infrared-to-visible light image conversion network.
[0020] Step 3: Use the trained object reflection prior guided infrared-to-visible light image conversion network to convert infrared images to visible light images.
[0021] Advantages
[0022] Compared with the prior art, the present application has the following advantages:
[0023] 1. The present application provides a new way to understand and simulate the physical differences between infrared and visible light imaging by decomposing the image into illumination source components and object-specific reflection components, and processing these components separately. This method can effectively improve the model's adaptability to scene lighting changes, making the converted visible light image maintain color and brightness consistency under different environmental lighting conditions, thereby enhancing the naturalness and visual quality of the image.
[0024] 2. By using advanced segmentation models as prior knowledge, the present application can more accurately identify and process various objects and scenes in the image, especially at object boundaries and details. This object-aware approach not only improves the detail restoration of the converted image, but also exhibits better stability and accuracy when dealing with complex scenes. In addition, this method provides more abundant and accurate information for further image analysis and processing, which helps to improve the execution effect of subsequent visual tasks such as object recognition and scene understanding.
[0025] 3. Through the object reflection prior guided method, the present application can effectively bridge the gap between infrared images and visible light images while preserving object details and texture information. This method not only improves the color accuracy and visual quality of the converted image, but also maintains good conversion effect under different lighting conditions, greatly improving the naturalness and realism of the converted image.
[0026] The present application not only improves the quality and efficiency of infrared-to-visible image conversion, but also provides new perspectives and tools for related research and application. These advantages and benefits make the present application have wide application prospects and important significance in medical diagnosis, agricultural monitoring, traffic management, auxiliary driving and monitoring systems and other fields. BRIEF DESCRIPTION OF DRAWINGS
[0027] Figure 1 is a flowchart of the method of the present application;
[0028] Figure 2 is a schematic diagram of the object reflection prior guided infrared-to-visible image conversion network structure of the method of the present application;
[0029] Figure 3 is a schematic diagram of the structure of the system of the present application. DETAILED DESCRIPTION
[0030] In order to better illustrate the purposes and advantages of the present application, the following further describes the content of the application in conjunction with the drawings.
[0031] As shown in Figure 1 , a method for converting infrared-to-visible image based on object reflection prior guidance, comprising the following steps:
[0032] Step 1: Constructing an object reflection prior guided infrared-to-visible image conversion network. As shown in Figure 2 .
[0033] The object reflection prior guided infrared-to-visible image conversion network includes an image decomposition network, an image segmentation network, an illumination estimation network, a reflection guidance network and an image reconstruction network.
[0034] First, the image decomposition network based on the residual network and the self-attention module, the target input is the original single-channel infrared image , the target output is the single-channel environmental infrared illumination component and the single-channel object infrared reflection component :
[0035] wherein, represents the image decomposition network.
[0036] This decomposition takes into account that the illumination is the light energy in the scene that is irradiated onto the object, and is independent of the object in the scene; while the reflection component is specific to the object, reflecting the reflectivity of the object in different spectral bands, and is independent of the environment to show different reflection characteristics in the infrared and visible spectral range.
[0037] The image decomposition network is composed of a residual network module and a self-attention network module. By introducing the residual network module in the network, the network can learn more complex and abstract feature representations. The self-attention network module calculates the correlation between different positions in the image, enabling the network to capture long-distance dependencies and help more accurately decompose the illumination and reflection characteristics in the image.
[0038] At the same time, in order to accurately extract the reflection characteristics of different objects in the image, the present application uses an image segmentation network to identify various objects in the image. Specifically, the image segmentation network uses scene segmentation to process the input infrared image to obtain image segmentation results , which are used as object reflection priors to guide the subsequent recovery of object visible light reflections.
[0039]
[0040] wherein, represents the image segmentation network.
[0041] These object reflection priors not only help the conversion network understand the spatial distribution of objects in the image, but also provide a basis for subsequent reflection characteristic analysis.
[0042] Secondly, the illumination estimation network based on the encoder and decoder architecture maps the infrared illumination component obtained by the image decomposition network to the illumination component of the visible light image :
[0043]
[0044] wherein, represents the illumination estimation network.
[0045] The reflection guiding network based on self-attention uses the object reflection priors obtained by the image segmentation network to map the infrared reflection component to the reflection component of the visible light image . After the object segmentation model extracts the object information in the image, the reflection guiding network will combine the object reflection priors to apply appropriate illumination and reflection processing strategies to each identified object region.
[0046]
[0047] wherein, represents the reflection guiding network based on self-attention.
[0048] This object-based processing method not only more accurately simulates the appearance of objects under visible light, but also effectively handles complex lighting effects such as shadows and highlights in the image. Through this step, the final generated visible light image can be closer to the real visible light imaging effect in terms of color saturation, contrast, and detail clarity.
[0049] Finally, the image reconstruction network reconstructs a high-quality visible light image by performing a point multiplication operation on the processed illumination component and the reflection component :
[0050]
[0051] wherein, represents the image reconstruction network.
[0052] This process not only emphasizes the importance of object reflection characteristics in infrared-to-visible light conversion, but also enhances the adaptability and accuracy of the model under different lighting conditions.
[0053] Step 2: Train the object reflection prior guided infrared-to-visible light image conversion network using the loss function.
[0054] Specifically, the loss function of the infrared-to-visible light image conversion task is usually represented as a pixel-level loss, which measures the difference between the model's prediction and the actual label. In this invention, a common infrared-to-visible light image conversion loss function is the Mean Absolute Error function, also known as , also known as the least absolute deviation:
[0055]
[0056] wherein is the total number of pixels in the image, is the prediction result, is the true value. The infrared-to-visible light image conversion network is trained and optimized end-to-end through the above loss function.
[0057] Step 3: Use the trained object reflection prior guided infrared-to-visible light image conversion network to convert infrared images to visible light images.
[0058] To achieve the purpose of the invention, the present invention further proposes an object reflection prior guided infrared-to-visible light image conversion method, as shown in Figure 3 , which includes an object reflection prior guided infrared-to-visible light image conversion network building module, an image conversion network training module, and an infrared-to-visible light image conversion inference module.
[0059] The object reflection prior guided infrared-to-visible light image conversion network building module is used to construct a deep learning network that can utilize object reflection characteristics, which can convert infrared images to visible light images.
[0060] The image conversion network training module is configured to train the constructed conversion network using a pre-prepared infrared-to-visible image pair dataset to optimize the conversion accuracy and effect thereof;
[0061] The infrared-to-visible image conversion inference module is configured to perform inference on new infrared images using the optimized conversion network after training is completed, thereby realizing conversion from infrared images to visible light images.
[0062] The connection relationship between the above modules is as follows:
[0063] The output end of the object reflection prior guided infrared-to-visible image conversion network building module is connected to the input end of the image conversion network training module;
[0064] The output end of the image conversion network training module is connected to the input end of the infrared-to-visible image conversion inference module.
[0065] Through this series of connections, the entire conversion process is ensured to be smooth and efficient, and ultimately the precise conversion from infrared images to high-quality visible light images is realized.
Claims
1. An infrared-to-visible image conversion method based on object reflection prior guided, characterized in that, The method comprises the following steps: Step 1: constructing an object reflection prior guided infrared-to-visible image conversion network, including an image decomposition network, an image segmentation network, an illumination estimation network, a reflection guiding network and an image reconstruction network; Wherein, the target input of the image decomposition network based on the residual network and the self-attention is the original single-channel infrared image, and the target output is a single-channel environmental infrared illumination component and a single-channel object infrared reflection component; the image decomposition network comprises a residual network module and a self-attention network module; The image segmentation network processes the input infrared image by scene segmentation to obtain an image segmentation result, which is used as an object reflection prior to guide the subsequent recovery of the object visible light reflection; The illumination estimation network based on the encoder and decoder architecture maps the infrared illumination component obtained by the image decomposition network into the illumination component of the visible light image; The reflection guiding network based on the self-attention uses the object reflection prior obtained by the image segmentation network to map the infrared reflection component into the reflection component of the visible light image; Finally, the image reconstruction network reconstructs a high-quality visible light image by point multiplication of the processed illumination component and reflection component; Step 2: training the object reflection prior guided infrared-to-visible image conversion network using a loss function; Step 3: converting the infrared image into a visible light image using the trained object reflection prior guided infrared-to-visible image conversion network.
2. The method of claim 1, wherein the method is based on object reflection prior guided infrared-to-visible image conversion. In step 2, the infrared-to-visible image conversion loss function is the mean absolute error function, referred to as L1Loss, also known as the least absolute deviation: Wherein, N is the total number of pixels in the image, P is the prediction result, and G is the true value; the infrared-to-visible image conversion network is trained and optimized end-to-end through the above loss function.
3. The method of claim 1, wherein the method further comprises: In step 1, there is R N , L N = DecomposeNet(I N ), where DecomposeNet represents an image decomposition network, I N is an original single-channel infrared image, L N is a single-channel ambient infrared illumination component, and R N is a single-channel object infrared reflection component.
4. The infrared-to-visible image conversion method based on prior guidance of object reflection as described in claim 1, characterized in that, The obtained image segmentation result P obj is: P obj = Segmentation(I N ) wherein Segmentation denotes an image segmentation network, I N is the original single-channel infrared image.
5. The infrared-to-visible image conversion method based on prior guidance of object reflection as described in claim 1, characterized in that, The luminance component L of the visible light image V is: L V = LuminanceNet(L N ) where LuminanceNet denotes the illumination estimation network, I N is the original single-channel infrared image.
6. The method of claim 1, wherein the method further comprises: A reflection component R of the visible light image V is: R V = ObjectGuidedReflectionNet(R N , P obj ) wherein ObjectGuidedReflectionNet denotes a self-attention based reflection guided network, P obj R is the obtained image segmentation result N is the single-channel object infrared reflection component.
7. The infrared-to-visible image conversion method based on prior guidance of object reflection as described in claim 1, characterized in that, Visible light image I V is: I V = ReconstructNet(R v , L v ) where ReconstructNet denotes the image reconstruction network, R V is the reflectance component of the visible light image, L V is the illumination component of the visible light image.
8. An object reflection prior guided infrared-to-visible image conversion system for implementing an object reflection prior guided infrared-to-visible image conversion method according to any one of claims 1 to 7, characterized in that, The method comprises an object reflection prior guided infrared-to-visible image conversion network construction module, an image conversion network training module and an infrared-to-visible image conversion inference module; The object reflection prior guided infrared-to-visible image conversion network construction module is used to construct a deep learning network capable of utilizing the object reflection characteristics, which can convert the infrared image into a visible light image; The image conversion network training module is used to train the constructed conversion network using a pre-prepared infrared-to-visible image pair dataset to optimize the conversion accuracy and effect; The infrared-to-visible image conversion inference module is used to infer new infrared images using the optimized conversion network after training to realize the conversion from infrared images to visible light images; The connection relationship between the above modules is as follows: The output end of the object reflection prior guided infrared-to-visible image conversion network construction module is connected to the input end of the image conversion network training module; The output end of the image conversion network training module is connected to the input end of the infrared-to-visible image conversion inference module.