A Near-Infrared-Driven Method for Mobile Image Enhancement

Abstract

The present invention provides a method for enhancing mobile phone imaging images driven by near-infrared, including: Step 1, an additional near-infrared spectral sensor is mounted on the mobile terminal to select the near-infrared spectral band; an RGB image is captured by the visible light camera of the mobile terminal, and a near-infrared image is obtained through the near-infrared spectral sensor; Step 2, a visible light and near-infrared joint decoupling module is established to map the near-infrared image and the RGB image to the same feature space, and then the light and dark components are output through a decoder; Step 3, a light and dark component embedding framework is established to embed the light and dark components into the image enhancement network in a plug-and-play manner to obtain the final enhanced result. The present invention can make full use of the additional near-infrared band information to provide a prior with clear physical information guidance for mobile phone imaging, and to a certain extent, can alleviate problems such as the dependence on data and poor interpretability of the end-to-end deep learning method.

Description

Technical Field

[0001] The present invention belongs to the field of spectral applications and image enhancement, and particularly relates to a method for enhancing mobile phone imaging images driven by near-infrared light. Background Art

[0002] In the current mainstream CMOS filters of RGB cameras in smart phones, the cut-off wavelength of the near-infrared band is usually set between 700 - 850 nm. By effectively blocking most of the near-infrared light, problems such as color cast caused by infrared light interference can be prevented. In recent years, equipping smart phones with additional spectral sensors has provided more possibilities for improving mobile phone imaging and exploring potential downstream applications, such as skin condition analysis, cosmetics detection, food safety, etc. With the development of micro-spectrometer technology, spectral sensors can provide scene spectral information with higher spatial resolution and spectral resolution, and by removing the infrared cut-off filter, additional near-infrared band information can be obtained. However, how to utilize the additional near-infrared band information as a prior for mobile phone imaging and give full play to its potential in image enhancement on smart phones is an urgent problem to be solved. Summary of the Invention

[0003] Object of the Invention: The technical problem to be solved by the present invention is to provide a method for enhancing mobile phone imaging images driven by near-infrared light in view of the deficiencies of the prior art, aiming to make full use of the additional near-infrared band information to further improve the performance of mobile phone imaging.

[0004] The method of the present invention includes the following steps:

[0005] Step 1, equip the mobile device with an additional near-infrared spectral sensor, and select the near-infrared spectral band from where represents the starting band of the spectral sensing curve of the near-infrared spectral sensor, and represents the ending band of the spectral sensing curve of the near-infrared spectral sensor; capture an RGB (red, green, blue) image through the visible light camera of the mobile device , and obtain a near-infrared image through the near-infrared spectral sensor ; the prominent feature of this step is to approximate the shadow component of the real-world outdoor environment through near-infrared band information, thereby overcoming the problem of difficulty in intrinsic decomposition of real-world outdoor scenes.

[0006] Step 2, establish a visible light and near-infrared (RGB-NIR) joint decoupling module, map the near-infrared image and the RGB image to the same feature space, and then output the light and dark components through a decoder ; This step is characterized by making full use of the RGB camera and the near-infrared spectral sensor for information fusion, and performing feature interpolation enhancement operations in the potential feature mapping space, thereby achieving the acquisition of more fine-grained outdoor scene shadow component information;

[0007] Step 3, establish a light and dark component embedding framework, and embed the light and dark components into the image enhancement network to obtain the final enhanced result. The characteristics of this step can be that additional light and dark component priors can be introduced in existing image enhancement methods, which have the advantages of being lightweight, plug-and-play, and strong generalization, thereby realizing the application of additional near-infrared band spectral information in existing image enhancement methods.

[0008] Step 1 includes:

[0009] Step 1-1, using the outdoor near-infrared band selection method, select the near-infrared spectral band from as the prior information for image enhancement;

[0010] Step 1-2, in the hardware layout of the sensors, it is necessary to minimize the parallax between multiple sensors as much as possible. The optical axis distance between the visible light camera and the near-infrared spectral sensor is less than a threshold (usually taken as 10 mm); capture the RGB image through the mobile visible light camera , , denotes the real number space of of, and H and W respectively denote the height and width of the RGB image

[0011] Obtain the near-infrared image through the near-infrared spectral sensor , , and h and w respectively denote the height and width of the near-infrared image . Use the collected near-infrared image as an approximation of the shadow component of the real-world outdoor environment.

[0012] In step 1-1, the outdoor near-infrared band selection method includes: using outdoor sunlight as the light source, jointly shooting a standard color card with a visible light and near-infrared scanning hyperspectral camera, and through spectral curve change analysis of the reflection spectral bands of different colors of the standard color card from the visible light to the near-infrared band, record the spectral curve of the th color as , calculate the spectral curve change rate of the th color :

[0013] = ,

[0014] wherein represents the change amount of the band in the spectral dimension , indicating the rate of change of the spectrum with wavelength;

[0015] The value of the band that satisfies the following formula is used as the value of the starting band :

[0016] ,

[0017] wherein represents the threshold of the change rate of different color spectral curves, and N represents the total number of colors;

[0018] Combined with the detection cut-off frequency of the silicon-based sensor in the near-infrared band, thereby establishing the starting band and the ending band of the spectral sensing curve of the near-infrared spectral sensor. Generally is generally not greater than 850 nm, and generally not less than 1000 nm.

[0019] Step 2 includes:

[0020] The visible light and near-infrared joint decoupling module is used to perform the following steps: Map the RGB image and the near-infrared image to the same feature space through multi-spectral feature mapping, and respectively obtain the depth feature map of the RGB image and the depth feature map of the near-infrared image ; In the feature space, obtain the feature map through feature interpolation enhancement operation, and based on the feature map , output the light and dark components through the decoder; , wherein is a single-channel image, reflecting the brightness change or light and shadow effect in the image caused by the lighting conditions (such as the intensity, direction and position of the light source); Through the above design, the acquisition of more fine-grained shadow component information of the outdoor scene is realized.

[0021] In step 2, the RGB encoder and the near-infrared encoder map the RGB image and the near-infrared image to the same feature space (the encoder can usually select resnet, and the technological innovation point of this part lies in the feature interpolation enhancement to realize the multi-sensor information fusion, while the RGB encoder With the NIR encoder (which serves for information fusion), the RGB image and the depth feature map of the near-infrared image are obtained respectively and the depth feature map of the near-infrared image :

[0022] ,

[0023] .

[0024] In step 2, for the i-th point in the depth feature map , select the 9 points adjacent to the i-th point in the depth feature map , calculate the fusion weight according to the feature similarity metric, and then obtain the feature map through the following formula

[0025] .

[0026] Step 3 includes: the light and dark component embedding framework includes a light and dark component adaptive module , input image space transformation, and output image space transformation;

[0027] First, based on the light and dark component adaptive module predict the transformation coefficient ; then perform the input image space transformation. For the input image that needs to be enhanced , transform the input image to the reflectance space through the transformation coefficient (generally, a typical image enhancement task network can be used, such as the general image enhancement model Restormer, the high dynamic range image enhancement model HDRNet, the low light enhancement model Retinexformer, etc.) for processing; finally, perform the output image space transformation, and transform back to the original image space through the transformation coefficient to obtain the final enhancement result

[0028] In step 3, the light and dark component adaptive module includes two layers of convolution and deconvolution. The light and dark component adaptive module is used to adaptively adjust the light and dark components to fit the image enhancement task

[0029] In step 3, for the light and dark components Perform adaptive normalization based on statistical mean and calculate the light and dark components The variance of the th pixel is :

[0030] = ,

[0031] where represents the statistical mean of the light and dark components ;

[0032] Then, adjust the light and dark components The th pixel is adjusted, and the corrected light and dark components are obtained through the following formula :

[0033] ,

[0034] where e is the natural constant; is the hyperparameter learned from the pixel histogram of the light and dark components ; ; represents the th pixel in the corrected light and dark components ;

[0035] For the corrected light and dark components , the transformation coefficient is predicted through the light and dark component adaptive module :

[0036] ,

[0037] The transformation coefficient .

[0038] The computational overhead brought by the lightweight designed light and dark component adaptive module to the original image enhancement network can be ignored

[0039] In step 3, the input of the image enhancement network is transformed into , that is, transformed into the reflectance space for image enhancement learning; the advantage of transforming into the reflectance space is that it can avoid the influence brought by factors such as local high dynamic range;

[0040] The predicted output of the image enhancement network is , and through the transformation of the transformation coefficient , the final enhancement result is obtained

[0041] Overall, through the utilization of near-infrared prior and the above-mentioned technological innovations, the present invention includes multi-sensor collaboration to obtain light and dark distributions prior, RGB-NIR joint decoupling module, and light and dark component embedding framework. The present invention can be applied to tasks such as tone mapping, image de-shadowing, local brightness adjustment, and low-light enhancement in mobile phone imaging for high dynamic range scenes.

[0042] The present invention has the following beneficial effects: (1) The present invention can make full use of additional near-infrared band information to provide a prior with clear physical information guidance for mobile phone imaging, which can, to a certain extent, alleviate problems such as the dependence on data and poor interpretability of end-to-end deep learning methods.

[0043] (2) By utilizing the complementarity between high-spatial-resolution RGB images and low-spatial-resolution spectral images captured by spectral sensors, the limited spectral imaging performance on the smartphone side caused by physical space limitations can be overcome.

[0044] (3) The present invention can embed the near-infrared prior in a plug-and-play manner without changing the main structure of the existing image enhancement network, and can be applied to various mobile phone imaging tasks, having good multi-task generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The following further specifically describes the present invention in conjunction with the drawings and specific embodiments, and the above and / or other advantages of the present invention will become clearer.

[0046] Figure 1 is the flowchart of the method of the present invention.

[0047] Figure 2 is the structural schematic diagram of the visible light and near-infrared (RGB-NIR) joint decoupling module.

[0048] Figure 3 is the schematic diagram of the light and dark component embedding framework. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] As Figure 1 shown, the embodiment of the present invention provides a near-infrared-driven mobile phone imaging image enhancement method, including the following steps:

[0050] Step 1, install an additional near-infrared spectral sensor on the mobile device (such as a smartphone), and select an appropriate near-infrared spectral band through an outdoor near-infrared band selection method; capture an RGB (red, green, blue, visible light) image through the visible light camera on the mobile device , and obtain a near-infrared image through the near-infrared spectral sensor . Approximation of the shadow component of the real-world outdoor environment is achieved through near-infrared band information.

[0051] Step 2, as Figure 2 shown, establish a visible light and near-infrared (RGB-NIR) joint decoupling module, map the near-infrared image and the RGB image to the same feature space, perform feature interpolation enhancement operations in the latent feature mapping space, and finally predict the light and dark component distribution of the shooting scene through the decoder , achieving the acquisition of more fine-grained outdoor scene shadow component information.

[0052] Step 3, establish a light and dark component embedding framework, and embed the light and dark components into the image enhancement network in a plug-and-play manner to obtain the final enhancement result. The feature of this step can be introducing additional light and dark component priors in existing image enhancement methods, which has the advantages of being lightweight, plug-and-play, and strong generalization, thus realizing the application of additional near-infrared band spectral information in existing image enhancement methods.

[0053] Step 1 includes: Common spectral imaging methods include prism spectroscopy, grating spectroscopy, narrowband filtering, etc. Due to the physical space limitations of mobile devices, high-resolution spectral imaging is usually restricted. Currently, spectral filtering is usually used on silicon-based detectors to achieve spectral detection on mobile devices. At the same time, the cut-off frequency of the silicon-based sensor in the near-infrared band is about 1100nm, and the signal-to-noise ratio in the near-infrared band imaging is low. It is necessary to select a suitable near-infrared band as the prior information for image enhancement. Considering that the difference between the reflection spectral curves of different colors gradually becomes smaller in the near-infrared band, the near-infrared band has a smaller difference in spectral curves of different colors compared to the visible light band, and the change of the spectral curve also tends to be consistent. Based on the above principle, the present invention proposes that the near-infrared band can be used as an approximation of the light and dark components in the scene. An important issue is how to select a suitable near-infrared band interval. Here, an outdoor near-infrared band selection method is proposed. Specifically, taking outdoor sunlight as the light source, a standard color card is photographed jointly with a visible light and near-infrared scanning hyperspectral camera. By analyzing the spectral curve changes of different colors of the standard color card from the visible light to the near-infrared band, the spectral curve of the th color is denoted as , and the spectral curve change rate of the mth color is calculated:

[0054] = ,

[0055] where represents in the spectral dimension A tiny change amount represents the rate of change of the spectrum with respect to wavelength, that is, the derivative of the spectral intensity with respect to wavelength; the rate of change of the spectral curve of each color is recorded here .

[0056] The wavelength band that satisfies the following formula is used as the value of the starting wavelength band :

[0057] ,

[0058] where represents the threshold of the rate of change of the spectral curves of different colors, N represents the total number of colors. Generally speaking can be set to 15%. The above formula means that when the average rate of change of the spectral curves of different colors is less than or equal to 15%, the value is used as the value of the starting wavelength band , and this starting wavelength band is often located in the near-infrared wavelength band range. Generally ;

[0059] Steps 1-4, taking into comprehensive consideration the detection cut-off frequency of the silicon-based sensor in the near-infrared wavelength band and the near-infrared starting wavelength band , determine the starting wavelength band and the ending wavelength band of the spectral perception curve of the near-infrared spectral sensor; generally is generally not greater than 850 nm, is generally not less than 1000 nm. The near-infrared wavelength band image with clear physical information can be used as a good estimation approximation of the light and dark components. In the hardware layout of the sensor, it is necessary to minimize the parallax between multiple sensors as much as possible. The optical axis distance between two sensors generally needs to be less than 10 mm; the mobile main camera RGB camera is used to collect RGB images , and the near-infrared spectral sensor is used to collect near-infrared images ;

[0060] Step 2 includes: after obtaining the RGB image captured by the main camera visible light camera and the near-infrared wavelength band image , in order to solve the spectral imaging limitation caused by the physical space limitation of the mobile device, here the complementary characteristics between the RGB image and the near-infrared single-channel image are utilized to obtain more fine-grained outdoor scene shadow component information.

[0061] Specifically, a visible light - near infrared RGB - NIR joint decoupling module is designed to predict the distribution of light and dark components of the entire scene. The RGB - NIR joint decoupling module consists of two sub - steps: feature mapping and feature interpolation decoding. In the first step of feature mapping, usually has a relatively small spatial resolution, while has a relatively large spatial resolution. The RGB - NIR joint decoupling module first maps and through the RGB encoder and into the same feature space to obtain the feature maps and , as follows:

[0062] ,

[0063] ,

[0064] Fuse the feature maps and within the feature space. Specifically, the depth feature maps and are processed through feature interpolation enhancement operations to obtain the enhanced feature maps ; specifically, for the i - th point in the depth feature map , select 9 points adjacent to the i - th point in the depth feature map , calculate the fusion weights according to the feature similarity metric, which can usually be given by the cosine similarity metric; and achieve fine - grained fusion of multi - spectral information through the following expression:

[0065] ,

[0066] Based on the fused feature map , predict the distribution of light and dark components of the entire scene through the light and dark component decoder . Among them, is a single - channel image, reflecting the brightness changes or light and shadow effects in the image caused by lighting conditions (such as the intensity, direction, and position of the light source), and H and W are the length and width of the light and dark components in the spatial dimension. represents the real - number space;

[0067] Step 3 includes: Using the light and dark component embedding framework proposed by the present invention, which consists of three parts: a light and dark component adaptive module, an input image space transformation, and an output image space transformation. The prior information S is embedded into the existing image enhancement method in a plug-and-play manner (generally, a typical image enhancement task network can be used, such as the general image enhancement model Restormer, the high dynamic range image enhancement model HDRNet, the low light enhancement model Retinexformer, etc.). Specifically, the input image to be enhanced is marked as and the existing image enhancement method is marked as . In the light and dark component embedding framework, first, based on the light and dark component adaptive module the transformation coefficients are predicted; then, for the input image to be enhanced, the input image is transformed into the reflectance space through , and then processed in the reflectance space by the existing image enhancement method . Finally, it is transformed back to the original image space through to obtain the enhanced target image;

[0068] Specifically, in order to constrain the results predicted by the light and dark component adaptive module , it is necessary to avoid the adverse effects caused by overly large brightness values. Therefore, first, the light and dark components are adaptively normalized based on the statistical mean, and the variance of the th pixel in the light and dark components is calculated, where represents the statistical mean of the light and dark components :

[0069] = ;

[0070] Then, the th pixel in the light and dark components is adjusted. For pixel values that deviate significantly from the mean, a smaller adjustment weight is given, where is a hyperparameter learned from the pixel histogram of the light and dark components . The corrected light and dark components are obtained through the following formula, where represents the th pixel in the corrected light and dark components

[0071] ,

[0072] For the calibrated light and dark components , the light and dark component adaptive module predicts the transform coefficients :

[0073] ,

[0074] The light and dark component adaptive module consists of two layers of convolution and deconvolution. The function of this module is to perform adaptive adjustment to adapt to the image enhancement task; at the same time, the computationally lightweight light and dark component adaptive module brings negligible computational overhead to the original image enhancement network.

[0075] Then, based on the transform coefficients , the image enhancement learning process is transformed into the reflectance space; specifically, the input of the image enhancement network is changed to , that is, it is transformed into the reflectance space for image enhancement learning; the advantage of transforming into the reflectance space is that it can avoid the influence brought by factors such as local high dynamic range; as Figure 3 shown, the abscissa represents the correlation coefficient and the ordinate represents the number of samples. Compared with the original image space, the statistical similarity between the input image and the target enhanced image in the reflectance space is higher, which shows that by transforming the original image space into the reflectance space, the difficulty of color mapping and image enhancement learning can be reduced.

[0076] The predicted output of the image enhancement network is , multiply it by the transform coefficient to obtain the final enhanced result . It should be noted that by transforming into the reflectance space, the image enhancement network can focus on the learning of components independent of light intensity.

[0077] Through the above design, the effective utilization of the additional near-infrared band can be achieved in mobile imaging. It can be applied to potential tasks such as tone mapping, image de-shadowing, local brightness adjustment, and low-light enhancement in high dynamic range scenes.

[0078] The present invention provides a method for enhancing near-infrared-driven mobile imaging images. There are many methods and ways to specifically implement this technical solution. The above description is only a preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention. Each component not clearly defined in this embodiment can be implemented by existing technologies.

Claims

1. A near-infrared driven mobile phone image enhancement method, characterized in that: The following steps are involved: Step 1: an additional near-infrared spectral sensor is installed on the mobile terminal, and a near-infrared spectral band from λ1 to λ2 is selected, where λ1 represents the starting band of the spectral perception curve of the near-infrared spectral sensor, and λ2 represents the ending band of the spectral perception curve of the near-infrared spectral sensor; an RGB image I is captured by a visible light camera on the mobile terminal. rgb , obtain near infrared image I through near infrared spectral sensor nir ; Step 2: Establish a visible light and near infrared joint decoupling module to obtain the near infrared image I nir With RGB image I rgb Mapped to the same feature space, and then output the light and dark components S through the decoder; Step 2 includes: The visible light and near infrared joint decoupling module is used to perform the following steps: rgb With near infrared image I nir Mapped to the same feature space, we get RGB images I rgb The deep feature map With near infrared image I nir The depth feature map of In the feature space, a feature map is obtained by feature interpolation enhancement operation Based on feature map Output the light and dark components S through the decoder; Among them, S is a single-channel image; H and W represent RGB images I and rgb height and width; Step 3: Establish a light and dark component embedding framework to embed the light and dark component S into the image enhancement network Φ in a plug-and-play manner to obtain the final enhancement result.

2. The method according to claim 1, characterized in that Step 1 includes: Step 1-1, using the outdoor near-infrared band selection method, select the near-infrared spectral band from λ1 to λ2 as the prior information for image enhancement; Step 1-2: the optical axis distance between the visible light camera and the near infrared spectrum sensor is less than a threshold; the RGB image I is captured by the visible light camera on the mobile terminal. rgb , represents the real number space of H×W×3; Acquire near infrared image by near infrared spectral sensor I nir , h and w represent the near infrared image I nir height and width.

3. The method according to claim 2, characterized in that In step 1-1, the outdoor near-infrared band selection method includes: using outdoor sunlight as a light source, combining visible light and near-infrared scanning hyperspectral cameras to shoot a standard color card, analyzing the change of spectral curves of the reflective spectral bands of different colors of the standard color card from visible light to near-infrared bands, and recording the spectral curve of the mth color as C m (λ), calculate the spectral curve change rate C′ of the mth color m (λ): Where dλ represents the change of band λ in the spectral dimension, C′ m (λ) represents the rate at which the spectrum changes with wavelength; The value of the band λ that satisfies the following formula is used as the value λ1 of the starting band: where ε C′ represents the threshold value of the change rate of the spectrum curve of different colors, and N represents the total number of colors; Combined with the detection cutoff frequency of silicon-based sensors in the near-infrared band, the starting band λ1 and the ending band λ2 of the spectral perception curve of the near-infrared spectral sensor are established.

4. The method according to claim 3, characterized in that In step 2, through the RGB encoder E rgb With near infrared encoder E nir The RGB image I rgb With near infrared image I nir Mapped to the same feature space, we get RGB images I rgb The depth feature map of With near infrared image I nir The depth feature map of 5. The method according to claim 4, characterized in that In step 2, for the depth feature map The i-th point in Select the deep feature map and the i-th point 9 neighboring points Calculate fusion weights based on feature similarity metrics Then the feature map is obtained by the following formula 6. The method according to claim 5, characterized in that Step 3 includes: the light and dark component embedding framework includes a light and dark component adaptive module Y, an input image space transformation, and an output image space transformation; First, the transform coefficients are predicted based on the adaptive module Y of the light and dark components Then the input image space transformation is performed. For the input image I that needs to be enhanced input , the input image I input By transforming the coefficients Converted to reflectivity space, processed by image enhancement network Φ in reflectivity space; finally, the output image space transformation is implemented, through the transformation coefficient Transform back to the original image space to get the final enhanced result.

7. The method according to claim 6, characterized in that In step 3, the light and dark component adaptive module Y includes two layers of convolution and deconvolution, and the light and dark component adaptive module Y is used to adaptively adjust the light and dark component S to adapt to the image enhancement task.

8. The method according to claim 7, characterized in that In step 3, the light and dark components S are adaptively normalized based on the statistical mean, and the i-th pixel S in the light and dark components S is calculated. i Variance where μ S Represents the statistical mean of the light and dark components S; Then for the i-th pixel S in the light and dark component S i Adjust and get the corrected brightness and darkness components through the following formula Where e is a natural constant; w s is the hyperparameter learned from the pixel histogram of the light and dark components S, Represents the corrected brightness and darkness components The i-th pixel in ; For the corrected light and dark components Predict the transform coefficients through the light and dark component adaptation module Y Transformation coefficient 9. The method according to claim 8, characterized in that In step 3, the input of the image enhancement network Φ is transformed into That is, transform it into reflectivity space for image enhancement learning; The predicted output of the image enhancement network Φ is By transforming the coefficients Convert to get the final enhanced result

Images