Infrared-to-visible light image conversion method based on object reflection prior guidance

Abstract

The invention provides an infrared-to-visible light image conversion method based on object reflection prior guidance, and belongs to the technical field of computer image processing. The method comprises the following steps: firstly, constructing an infrared-to-visible light image conversion network guided by object reflection prior, and then training the infrared-to-visible light image conversion network guided by object reflection prior by using a loss function; and finally, using the trained object to reflect the priori guided infrared-to-visible light image conversion network, and converting an infrared image into a visible light image. According to the method, the adaptability of the model to scene illumination change can be effectively improved, and the naturalness and the impression quality of the image are enhanced; various objects and scenes in the image can be identified and processed more accurately, and the execution effect of a subsequent visual task can be improved; according to the method, the difference between the infrared image and the visible light image can be effectively bridged while object details and texture information are reserved, the color accuracy and the visual quality of the converted image are improved, and the naturalness and the fidelity of the converted image are improved.

Description

Technical Field

[0001] The invention relates to an infrared to visible light image conversion method based on object reflection priori guidance, belonging to the technical field of computer image processing. Background Art

[0002] With the rapid development of deep learning technology in the field of image processing, infrared to visible light (NIR2VIS) conversion has become a hot research direction, aiming to convert infrared (NIR) images into more rich and intuitive visible light (VIS) images. This technology has a wide range of applications in medical diagnosis, agricultural monitoring, traffic management, assisted driving, monitoring systems and other fields.

[0003] Although infrared imaging technology is widely used due to its unique ability to penetrate the atmosphere and its undetectable properties to the human eye, infrared images usually lack color and detail information that is more intuitive to human vision, which limits its application potential in intuitive visual presentation and advanced visual tasks.

[0004] Early infrared-to-visible conversion methods mostly relied on traditional image processing techniques, such as histogram matching and rule-based color mapping. These methods often rely on precise prior knowledge and a lot of manual adjustments. With the introduction of convolutional neural networks (CNNs) and generative adversarial networks (GANs), the research on infrared-to-visible conversion has gradually turned to data-driven methods, which achieve conversion by learning the mapping relationship between a large number of infrared and visible light image pairs. However, one of the main challenges faced by such methods is how to obtain a large number of accurately aligned infrared-to-visible light image pairs, which is often difficult to achieve in practical applications.

[0005] In order to solve the problem of difficulty in obtaining paired training data, some researchers have begun to explore unsupervised learning methods, such as cycle consistency GAN (CycleGAN), which allows the model to learn the conversion of infrared to visible light images without paired data. Although such methods have alleviated the limitation of training data to a certain extent, the conversion results often have problems such as color deviation and loss of details.

[0006] Although these technologies have improved the performance of infrared to visible light conversion, they still face many challenges, including the physical reality of the conversion process, the generalization ability of the model, and the processing of dynamic lighting scenes. Therefore, accurately modeling the mapping relationship between infrared and visible light and generating high-quality visible light images is still an urgent problem to be solved. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings and defects of the existing infrared to visible light image conversion methods, and creatively propose an infrared to visible light image conversion method and system based on object reflection prior guidance.

[0008] The present invention can make full use of the image segmentation results as the object reflectance prior, decompose the image into an illumination component that is independent of the object and an object-specific reflectance component, and then restore the illumination and reflectance components of visible light respectively, so as to achieve high-quality infrared to visible light image conversion under various lighting conditions. At the same time, this method also proposes an infrared to visible light image conversion system based on object reflectance prior guidance.

[0009] The present invention is implemented by adopting the following technical solutions.

[0010] A method for converting infrared to visible light images based on object reflection prior guidance includes the following steps: Step 1: Build an infrared-to-visible image conversion network guided by object reflectance prior.

[0011] The image network for infrared to visible image conversion guided by object reflectance prior includes image decomposition network, image segmentation network, illumination estimation network, reflectance guidance network and image reconstruction network.

[0012] First, the target input of the image decomposition network based on residual network and self-attention is the original single-channel infrared image, and the target output is a single-channel ambient infrared illumination component and a single-channel object infrared reflection component.

[0013] This decomposition takes into account that illumination is the light energy that hits objects in the scene, and is independent of the objects in the scene. The reflection component is object-specific, reflecting the reflectivity of the object in different spectral bands, and is independent of the environment, showing different reflection characteristics in the infrared and visible light spectrum ranges.

[0014] Specifically, the image decomposition network includes a residual network module and a self-attention network module. By introducing the residual network module into the network, the network can learn more complex and abstract feature representations. The self-attention network module enables the network to capture long-range dependencies by calculating the correlation between different positions within the image, which helps to more accurately decompose the illumination and reflection characteristics in the image.

[0015] At the same time, the image segmentation network uses scene segmentation to process the input infrared image to obtain the image segmentation result, which is used as the object reflection prior to subsequently guide the restoration of the object's visible light reflection.

[0016] Secondly, the illumination estimation network based on the encoder and decoder architecture maps the infrared illumination component obtained by the image decomposition network to the illumination component of the visible light image; the reflection guidance network based on self-attention utilizes the object reflection prior obtained by the image segmentation network to map the infrared reflection component to the reflection component of the visible light image.

[0017] Finally, the image reconstruction network reconstructs a high-quality visible light image by performing a dot product operation on the processed illumination component and the reflection component. This process not only emphasizes the importance of the object's reflective properties in the conversion of infrared to visible light, but also enhances the adaptability and accuracy of the model under different lighting conditions.

[0018] Step 2: Using the loss function, train an infrared-to-visible image conversion network guided by the object reflectance prior.

[0019] Step 3: Use the trained object reflectance prior-guided infrared-to-visible image conversion network to convert the infrared image to a visible light image.

[0020] Beneficial Effects

[0021] Compared with the prior art, the present invention has the following advantages: 1. The present invention provides a new way to understand and simulate the physical differences between infrared and visible light imaging by decomposing the image into an illumination source component that is independent of the object and an object-specific reflection component, and processing these components separately. This method can effectively improve the adaptability of the model to scene illumination changes, so that the converted visible light image can maintain the consistency of color and brightness under different ambient illumination conditions, thereby enhancing the naturalness and visual quality of the image.

[0022] 2. By using advanced segmentation models as prior knowledge, the present invention can more accurately identify and process various objects and scenes in images, especially at object boundaries and details. This object perception-based method not only improves the detail restoration of the converted image, but also shows better stability and accuracy when processing complex scenes. In addition, this method provides richer and more accurate information for further image analysis and processing, which helps to improve the performance of subsequent visual tasks, such as object recognition, scene understanding, etc.

[0023] 3. Through the method guided by the object reflection prior, the present invention can effectively bridge the gap between infrared images and visible light images while retaining the object details and texture information. This method can not only improve the color accuracy and visual quality of the converted image, but also maintain a good conversion effect under different lighting conditions, greatly improving the naturalness and realism of the converted image.

[0024] The present invention not only improves the quality and efficiency of infrared to visible light image conversion, but also provides new perspectives and tools for research and application in related fields. These advantages and beneficial effects make the present invention have broad application prospects and important significance in many fields such as medical diagnosis, agricultural monitoring, traffic management, assisted driving and monitoring systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of the method of the present invention; Figure 2 Schematic diagram of the network structure of infrared to visible light image conversion guided by object reflection prior in the method of the present invention; Figure 3 It is a schematic diagram of the structure of the system of the present invention. DETAILED DESCRIPTION

[0026] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings.

[0027] like Figure 1 As shown, a method for converting infrared to visible light images based on object reflection prior guidance includes the following steps: Step 1: Construct an infrared to visible light image conversion network guided by object reflectance prior. Figure 2 shown.

[0028] The object reflectance prior-guided infrared-to-visible image conversion network includes an image decomposition network, an image segmentation network, an illumination estimation network, a reflectance guidance network, and an image reconstruction network.

[0029] First, the image decomposition network based on the residual network and the self-attention module, whose target input is the original single-channel infrared image , the target output is a single-channel ambient infrared illumination component And the infrared reflection component of the object in a single channel :

[0030] in, Represents an image decomposition network.

[0031] This decomposition takes into account that illumination, as the light energy that shines on objects in the scene, is independent of the objects in the scene; whereas the reflection component is object-specific, reflecting the magnitude of the object's reflectivity in different spectral bands, and is independent of the environment, exhibiting different reflection characteristics in the infrared and visible light spectrum ranges.

[0032] The image decomposition network consists of a residual network module and a self-attention network module. By introducing the residual network module into the network, the network can learn more complex and abstract feature representations. The self-attention network module calculates the correlation between different positions within the image, allowing the network to capture long-distance dependencies, which helps to more accurately decompose the illumination and reflection characteristics in the image.

[0033] At the same time, in order to accurately extract the reflection characteristics of different objects in the image, the present invention uses an image segmentation network to identify various objects in the image. Specifically, the image segmentation network uses scene segmentation to process the input infrared image to obtain the image segmentation result. , and serves as the object reflectance prior to subsequently guide the recovery of the object's visible light reflectance.

[0034]

[0035] in, Represents an image segmentation network.

[0036] These object reflectance priors not only help the transformer network understand the spatial distribution of objects in the image, but also provide a basis for subsequent reflectance property analysis.

[0037] Secondly, the illumination estimation network based on the encoder and decoder architecture maps the infrared illumination components obtained by the image decomposition network to the illumination components of the visible light image. :

[0038] in, Represents the lighting estimation network.

[0039] Based on the self-attention reflection guidance network, the infrared reflection component is mapped to the reflection component of the visible light image using the object reflection prior obtained by the image segmentation network. After the object segmentation model extracts the object information in the image, the reflection guidance network will combine the object reflection prior and apply the corresponding lighting and reflection processing strategy to each identified object area.

[0040]

[0041] in, Represents a self-attention based reflection guided network.

[0042] This object-based processing method can not only more accurately simulate the appearance of objects under visible light, but also effectively handle complex lighting effects such as shadows and highlights in the image. Through this step, the final generated visible light image can be closer to the real visible light imaging effect in terms of color saturation, contrast and detail clarity.

[0043] Finally, the image reconstruction network reconstructs a high-quality visible light image by performing a dot multiplication operation on the processed illumination component and the reflection component. :

[0044] in, represents the image reconstruction network.

[0045] This process not only emphasizes the importance of the object's reflective properties in the conversion of infrared to visible light, but also enhances the adaptability and accuracy of the model under different lighting conditions.

[0046] Step 2: Using the loss function, train an infrared-to-visible image conversion network guided by the object reflectance prior.

[0047] Specifically, the loss function of the infrared to visible light image conversion task is usually expressed as a pixel-level loss, which is used to measure the difference between the model's prediction and the actual label. In the present invention, a common infrared to visible light image conversion loss function is the mean absolute error function (Mean Absolute Error), also known as , also called the minimum absolute deviation:

[0048] in is the total number of pixels in the image, is the prediction result, is the true value. The infrared to visible light image conversion network completes end-to-end training and optimization through the above loss function.

[0049] Step 3: Use the trained object reflectance prior-guided infrared-to-visible image conversion network to convert the infrared image to a visible light image.

[0050] To achieve the purpose of the present invention, the present invention further proposes an infrared to visible light image conversion method based on object reflection prior guidance, such as Figure 3 As shown, it includes an infrared to visible light image conversion network building module guided by object reflection prior, an image conversion network training module, and an infrared to visible light image conversion reasoning module.

[0051] The module for building a network for infrared-to-visible image conversion guided by object reflectance priors is used to build a deep learning network that can use the reflectance characteristics of objects to convert infrared images into visible light images. The image conversion network training module is used to train the constructed conversion network using a pre-prepared infrared to visible light image pair dataset to optimize its conversion accuracy and effect; The infrared to visible light image conversion inference module is used to use the optimized conversion network to infer new infrared images after training is completed, thereby realizing the conversion from infrared images to visible light images.

[0052] The connection relationship between the above modules is as follows: The output end of the infrared to visible light image conversion network building module guided by the object reflection prior is connected to the input end of the image conversion network training module; The output end of the image conversion network training module is connected to the input end of the infrared to visible light image conversion reasoning module.

[0053] Through this series of connections, the smoothness and efficiency of the entire conversion process are ensured, ultimately achieving accurate conversion from infrared images to high-quality visible light images.

Claims

1. A method for converting infrared to visible light images based on object reflection prior guidance, characterized in that: The following steps are involved: Step 1: Construct an infrared-to-visible light image conversion network guided by object reflectance prior, including an image decomposition network, an image segmentation network, an illumination estimation network, a reflection guidance network, and an image reconstruction network; wherein the target input of the image decomposition network based on the residual network and self-attention is the original single-channel infrared image, and the target output is a single-channel ambient infrared illumination component and a single-channel object infrared reflection component; the image decomposition network includes a residual network module and a self-attention network module; The image segmentation network uses scene segmentation to process the input infrared image to obtain the image segmentation result, and uses it as the object reflection prior to subsequently guide the restoration of the object's visible light reflection; the illumination estimation network based on the encoder and decoder architecture maps the infrared illumination component obtained by the image decomposition network to the illumination component of the visible light image; the self-attention-based reflection guidance network uses the object reflection prior obtained by the image segmentation network to map the infrared reflection component to the reflection component of the visible light image; finally, the image reconstruction network reconstructs a high-quality visible light image by performing a dot product operation on the processed illumination component and the reflection component; Step 2: Use the loss function to train the infrared to visible light image conversion network guided by the object reflection prior; Step 3: Use the trained object reflection prior-guided infrared to visible light image conversion network to convert the infrared image to a visible light image.

2. The infrared to visible light image conversion method based on object reflection prior guidance as claimed in claim 1, characterized in that: In step 2, the loss function used for infrared to visible image conversion is the mean absolute error function, called Loss, also called minimum absolute deviation: in is the total number of pixels in the image, is the prediction result, is the true value; the infrared to visible light image conversion network completes end-to-end training and optimization through the above loss function.

3. The infrared to visible light image conversion method based on object reflection prior guidance as claimed in claim 1, characterized in that: In step 1, there is ,in, represents the image decomposition network, is the original single-channel infrared image, is the single-channel ambient infrared illumination component, It is the infrared reflection component of a single channel object.

4. The infrared to visible light image conversion method based on object reflection prior guidance as claimed in claim 1, characterized in that: The image segmentation results obtained for: in, represents the image segmentation network, is the original single-channel infrared image.

5. The infrared to visible light image conversion method based on object reflection prior guidance as claimed in claim 1, characterized in that: Illumination Component of Visible Light Images for: in, represents the illumination estimation network, is the original single-channel infrared image.

6. The infrared to visible light image conversion method based on object reflection prior guidance as claimed in claim 1, characterized in that: Reflection component of visible light image for: in, represents the self-attention based reflection guided network, is the image segmentation result obtained, It is the infrared reflection component of a single channel object.

7. The infrared to visible light image conversion method based on object reflection prior guidance as claimed in claim 1, characterized in that: Visible light images for: in, represents the image reconstruction network, is the reflection component of the visible light image, is the illumination component of the visible light image.

8. An infrared to visible light image conversion system based on object reflection prior guidance, characterized in that: It includes an infrared to visible light image conversion network building module guided by object reflection prior, an image conversion network training module, and an infrared to visible light image conversion reasoning module; the infrared to visible light image conversion network building module guided by object reflection prior is used to build a deep learning network that can utilize the reflection characteristics of objects, and the network can convert infrared images into visible light images; The image conversion network training module is used to use a pre-prepared infrared to visible light image pair data set to train the constructed conversion network to optimize its conversion accuracy and effect; the infrared to visible light image conversion reasoning module is used to use the optimized conversion network to infer the new infrared image after the training is completed to achieve the conversion from infrared image to visible light image; the connection relationship between the above modules is as follows: the output end of the infrared to visible light image conversion network construction module guided by the object reflection prior is connected to the input end of the image conversion network training module; the output end of the image conversion network training module is connected to the input end of the infrared to visible light image conversion reasoning module.

Images