Near-infrared driven mobile phone image enhancement method

Abstract

The invention provides a near-infrared driven mobile phone image enhancement method, which comprises the following steps of: 1, carrying an additional near-infrared spectrum sensor on a mobile terminal, and selecting a near-infrared spectrum wave band; an RGB image is shot through a mobile terminal visible light camera, and a near-infrared image is obtained through a near-infrared spectrum sensor; step 2, establishing a visible light and near-infrared combined decoupling module, mapping a near-infrared image and an RGB image to the same feature space, and then outputting light and shade components through a decoder; and step 3, establishing a bright and dark component embedding framework, and embedding the bright and dark components into the image enhancement network Yoliang in a plug-and-play mode to obtain a final enhancement result. According to the method, extra near-infrared band information can be fully utilized, priori with clear physical information guidance is provided for mobile phone images, and the problems that an end-to-end deep learning method depends on data and is poor in interpretability and the like can be relieved to a certain extent.

Description

Technical Field

[0001] The invention belongs to the field of spectrum application and image enhancement, and in particular relates to a near-infrared driven mobile phone image enhancement method. Background Art

[0002] In the CMOS filter of the current mainstream smartphone RGB camera, the cutoff wavelength of the near-infrared band is usually set between 700 and 850nm. By effectively blocking most of the near-infrared light, problems such as color cast caused by infrared light interference can be prevented. In recent years, the installation of additional spectral sensors on smartphones has provided more possibilities for the improvement of mobile phone images and the exploration of potential downstream applications, such as skin condition analysis, cosmetics testing, food safety, etc. With the development of micro-spectrometer technology, spectral sensors can provide scene spectral information with higher spatial resolution and spectral resolution, and obtain additional near-infrared band information by removing the infrared cutoff filter. However, how to use the additional near-infrared band information as a priori for mobile phone images and give full play to its potential for image enhancement on the smartphone side is an urgent problem to be solved. Summary of the invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a near-infrared driven mobile phone image enhancement method in view of the deficiencies in the prior art, and to make full use of additional near-infrared band information to further improve the performance of mobile phone images.

[0004] The method of the present invention comprises the following steps: Step 1: Install an additional near-infrared spectral sensor on the mobile terminal and select The near-infrared spectral band, Represents the starting band of the spectral perception curve of the near-infrared spectral sensor, The end band of the spectral perception curve representing the near-infrared spectral sensor; RGB (red, green, blue) images captured by a mobile visible light camera , using near infrared spectral sensors to obtain near infrared images ; The outstanding feature of this step is to achieve the approximation of the shadow component of the real-world outdoor environment through the near-infrared band information, thereby overcoming the difficulty of intrinsic decomposition of the real-world outdoor scene; Step 2: Establish a visible light and near-infrared (RGB-NIR) joint decoupling module to decouple the near-infrared image With RGB image Mapped to the same feature space, and then output the light and dark components through the decoder ; This step is characterized by making full use of the RGB camera and the near-infrared spectrum sensor to fuse information and perform feature interpolation enhancement operations in the potential feature mapping space, thereby achieving more fine-grained acquisition of outdoor scene shadow component information; Step 3: Establish a light and dark component embedding framework to embed the light and dark components in a plug-and-play manner. Embedded into image enhancement network The final enhancement result is obtained. The characteristic of this step is that it can introduce additional light and dark component priors into the existing image enhancement method, which has the advantages of lightweight, plug-and-play, and strong generalization, thereby realizing the application of additional near-infrared band spectral information in the existing image enhancement method.

[0005] Step 1 includes: Step 1-1, using the outdoor near-infrared band selection method, select The near-infrared spectral band is used as the prior information for image enhancement; Step 1-2: In the hardware arrangement of sensors, the parallax between multiple sensors needs to be minimized as much as possible. The optical axis distance between the visible light camera and the near-infrared spectrum sensor is less than the threshold (generally 10 mm). The RGB image is captured by the mobile visible light camera. , , express The real number space, H and W represent RGB images respectively. height and width; Acquiring near-infrared images through near-infrared spectral sensors , , h and w represent near infrared images respectively Using the collected near infrared images , as an approximation of the shadow component of a real-world outdoor environment.

[0006] In step 1-1, the outdoor near-infrared band selection method includes: using outdoor sunlight as a light source, combining visible light and near-infrared scanning hyperspectral cameras to shoot a standard color card, and analyzing the change of spectral curves of the reflective spectrum bands of different colors of the standard color card from visible light to near-infrared bands. The spectrum curve of each color is recorded as , calculate the The rate of change of the spectrum curve of each color : = , in Represents the band in the spectral dimension The amount of change, Indicates the rate at which the spectrum changes with wavelength; The band that satisfies the following formula The value of the starting band : , in represents the threshold value of the change rate of the spectrum curve of different colors, and N represents the total number of colors; Combined with the detection cutoff frequency of silicon-based sensors in the near-infrared band, the starting band of the spectral perception curve of the near-infrared spectral sensor is established With stop band Typically, Generally no more than 850nm, Generally not less than 1000nm.

[0007] Step 2 includes: The visible light and near infrared joint decoupling module is used to perform the following steps: With near infrared imaging Mapped to the same feature space, we get RGB images respectively The depth feature map of With near infrared imaging The depth feature map of ; In the feature space, the feature map is obtained by feature interpolation enhancement operation , based on the feature map , the decoder outputs the light and dark components ; ,in, It is a single-channel image that reflects the brightness changes or light and shadow effects in the image caused by lighting conditions (such as the intensity, direction, and position of the light source). Through the above design, more fine-grained shadow component information of outdoor scenes can be obtained.

[0008] In step 2, through the RGB encoder With near infrared encoder Convert RGB image With near infrared imaging Mapped to the same feature space (the encoder can usually select resnet, the technical innovation of this part is to enhance the feature interpolation to achieve multi-sensor information fusion, and the RGB encoder With NIR encoder is for information fusion), and RGB images are obtained respectively The deep feature map With near infrared imaging The depth feature map of : , .

[0009] In step 2, for the depth feature map The i-th point in , select the deep feature map and the i-th point 9 neighboring points , calculate the fusion weight based on the feature similarity measure , and then the feature map is obtained by the following formula : .

[0010] Step 3 includes: the light and dark component embedding framework includes a light and dark component adaptive module , input image space transformation, output image space transformation; First, based on the light and dark component adaptive module Prediction Transform Coefficients ; Then perform input image space transformation, for the input image that needs to be enhanced , the input image By transforming the coefficients Transformed to reflectance space, passed through the image enhancement network in reflectance space (Typical image enhancement task networks can generally be used for processing, such as the general image enhancement model Restormer, the high dynamic range image enhancement model HDRNet, the dark light enhancement model Retinexformer, etc.) Finally, the output image space transformation is implemented, and the transformation coefficients are used Transform back to the original image space to get the final enhanced result.

[0011] In step 3, the light and dark component adaptive module It includes two layers of convolution and deconvolution, the brightness and darkness component adaptive module For light and dark components Adaptively adjust to image enhancement tasks.

[0012] In step 3, the light and dark components Adaptive normalization based on statistical mean to calculate light and dark components Middle Pixels Variance : = , in Represents light and dark components The statistical mean of Then for the light and dark components Middle Pixels Adjust and get the corrected brightness and darkness components through the following formula : , Where e is a natural constant; According to the light and dark components The hyperparameters learned from the pixel histogram, ; Represents the corrected brightness and darkness components The pixels; For the corrected light and dark components , through the light and dark component adaptive module Prediction Transform Coefficients : , Transformation coefficient .

[0013] The lightweight design of the light and dark component adaptive module brings negligible computational overhead to the original image enhancement network.

[0014] In step 3, the image enhancement network The input is transformed into , that is, transforming to the reflectivity space for image enhancement learning; the advantage of transforming to the reflectivity space is that it can avoid the influence of factors such as local high dynamic range; Image Enhancement Network The predicted output is , by transforming the coefficients Convert to get the final enhanced result .

[0015] In summary, the present invention utilizes near-infrared priors and the above-mentioned technical innovations, including the coordinated acquisition of light and dark distribution by multiple sensors. Prior, RGB-NIR joint decoupling module and light and dark component embedding framework. The present invention can be applied to mobile phone imaging tasks such as tone mapping of high dynamic range scenes, image shadow removal, local brightness adjustment, dark light enhancement, etc.

[0016] The present invention has the following beneficial effects: (1) The present invention can make full use of additional near-infrared band information to provide a priori with clear physical information guidance for mobile phone images, which can alleviate the problems of end-to-end deep learning methods' dependence on data and poor interpretability to a certain extent.

[0017] (2) By utilizing the complementarity between high spatial resolution RGB images and low spatial resolution spectral images captured by spectral sensors, the limited spectral imaging performance of smartphones due to physical space constraints can be overcome.

[0018] (3) The present invention can embed near-infrared priors in a plug-and-play manner without changing the main structure of the existing image enhancement network, and can be applied to a variety of mobile phone imaging tasks, with good multi-task generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0020] Figure 1 It is a flow chart of the method of the present invention.

[0021] Figure 2 It is a schematic diagram of the structure of the visible light and near-infrared (RGB-NIR) joint decoupling module.

[0022] Figure 3 It is a schematic diagram of the light and dark component embedding framework. DETAILED DESCRIPTION

[0023] like Figure 1 As shown, the embodiment of the present invention provides a near-infrared driven mobile phone image enhancement method, comprising the following steps: Step 1: Equip an additional near-infrared spectral sensor on a mobile terminal (such as a smartphone), select a suitable near-infrared spectral band through the outdoor near-infrared band selection method; and use a visible light camera on the mobile terminal to capture RGB (red, green, blue, visible light) images. , and near-infrared spectral sensors to obtain near-infrared images The shadow component of the real-world outdoor environment is approximated by near-infrared band information.

[0024] Step 2, such as Figure 2 As shown in the figure, a visible light and near infrared (RGB-NIR) joint decoupling module is established to decouple the near infrared image With RGB image Mapped to the same feature space, and perform feature interpolation enhancement in the potential feature mapping space, and finally predict the distribution of light and dark components of the shooting scene through the decoder , achieving more fine-grained acquisition of shadow component information of outdoor scenes.

[0025] Step 3: Establish a light and dark component embedding framework to embed the light and dark components in a plug-and-play manner. Embedded into image enhancement network The final enhancement result is obtained. The characteristic of this step is that it can introduce additional light and dark component priors into the existing image enhancement method, which has the advantages of lightweight, plug-and-play, and strong generalization, thereby realizing the application of additional near-infrared band spectral information in the existing image enhancement method.

[0026] Step 1 includes: Common spectral imaging methods include prism spectroscopy, grating spectroscopy, narrowband filtering, etc. Due to the physical space limitations of the mobile terminal, high-resolution spectral imaging is usually restricted. At present, spectral filtering is usually used on silicon-based detectors to achieve spectral detection on the mobile terminal. At the same time, the cutoff frequency of the near-infrared band of silicon-based sensors is about 1100nm, and imaging in the near-infrared band has a low signal-to-noise ratio, and it is necessary to select a suitable near-infrared band as prior information for image enhancement. At the same time, considering that the difference between the reflection spectral curves of different colors in the near-infrared band gradually becomes smaller, the near-infrared band has a smaller difference than the spectral curves of different colors in the visible light band, and the changes in the spectral curves also tend to be consistent. Through the above principles, the present invention proposes that the near-infrared band can be used as an approximation of the light and dark components in the scene. An important issue is how to select a suitable near-infrared band interval. Here, a method for selecting an outdoor near-infrared band is proposed. Specifically, outdoor sunlight is used as the light source, and a visible light and near-infrared scanning hyperspectral camera is used to shoot the standard color card. The spectral curve change analysis of the reflective spectrum bands of different colors of the standard color card from visible light to near-infrared bands is performed to obtain the first The spectrum curve of each color is recorded as , calculate the rate of change of the spectrum curve of the mth color : = , in Represents the spectral dimension Small changes, Indicates the rate at which the spectrum changes with wavelength, that is, the rate of change of the spectral intensity with respect to wavelength; here the rate of change of the spectral curve of each color is recorded .

[0027] The band that satisfies the following formula The value of the starting band : , in Represents the threshold value of the change rate of the spectrum curve of different colors, N represents the total number of colors, generally speaking, It can be set to 15%. The above formula indicates that when the average change rate of the spectrum curve of different colors is less than or equal to 15%, The value of the starting band , the starting band is often located in the near-infrared band. Usually, ; Steps 1-4, taking into account the detection cutoff frequency of silicon-based sensors in the near-infrared band and the near-infrared starting band , establish the starting band of the spectral perception curve of the near-infrared spectral sensor With stop band ; Usually, Generally no more than 850nm, Generally not less than 1000nm. Near-infrared band images with clear physical information can be used as a good estimate of the light and dark components. In the hardware arrangement of the sensor, it is necessary to minimize the parallax between multiple sensors as much as possible, and the optical axis distance between two sensors generally needs to be less than 10 mm; the main RGB camera on the mobile terminal is used to collect RGB images The near infrared spectral sensor is used to collect near infrared images. ; Step 2 includes: obtaining the RGB image captured by the main visible light camera Near infrared image Then, in order to solve the spectral imaging limitations caused by the physical space limitations of the mobile terminal, the RGB image With near infrared single channel image The complementary characteristics between them are utilized to obtain more fine-grained shadow component information of outdoor scenes.

[0028] Specifically, a visible light-near infrared RGB-NIR joint decoupling module is designed to predict the distribution of light and dark components of the entire scene. The RGB-NIR joint decoupling module consists of two sub-steps: feature mapping and feature interpolation decoding. In the first step of feature mapping, usually Has a smaller spatial resolution, With a larger spatial resolution, the RGB-NIR joint decoupling module first passes through the RGB encoder With NIR encoder Will and Mapped to the same feature space, we get the feature map and , as shown below: , ,

[0029] The feature map and Fusion is performed in the feature space. Specifically, the deep feature map is enhanced by feature interpolation. and Processing to obtain the enhanced feature map ; Specifically, for the deep feature map The i-th point in , select the deep feature map The 9 points adjacent to the i-th point , calculate the fusion weight based on the feature similarity measure , which can usually be given by the cosine similarity metric; and the fine-grained fusion of multispectral information is achieved through the following expression: , Based on the fused feature map , the light and dark component distribution of the entire scene is predicted by the light and dark component decoder .in, It is a single-channel image that reflects the brightness changes or light and shadow effects in the image caused by lighting conditions (such as the intensity, direction, and position of the light source). H and W are the light and dark components, respectively. Length and width in spatial dimensions. represents the real number space; Step 3 includes: using the light and dark component embedding framework proposed in the present invention, the light and dark component embedding framework includes three parts: light and dark component adaptive module, input image space transformation, and output image space transformation, to embed the prior information S into the existing image enhancement method in a plug-and-play manner (generally, a typical image enhancement task network can be used, such as the general image enhancement model Restormer, the high dynamic range image enhancement model HDRNet, the dark light enhancement model Retinexformer, etc.). Specifically, the input image to be enhanced is marked as , the existing image enhancement methods are labeled In the light and dark component embedding framework, firstly, based on the light and dark component adaptive module Prediction Transform Coefficients ; Then, for the input image that needs to be enhanced , the input image pass Transformed into reflectance space, and then in reflectance space by using existing image enhancement methods Processing, and finally through Transform back to the original image space to obtain the enhanced target image; Specifically, in order to realize the adaptive module of light and dark components To constrain the predicted results, it is necessary to avoid adverse effects caused by excessive brightness. Therefore, first Adaptive normalization based on statistical mean to calculate light and dark components Middle Pixels Variance ,in Represents light and dark components The statistical mean of : = , Then for the light and dark components Middle Pixels Adjustment is made. For pixel values ​​that deviate greatly from the mean, a smaller adjustment weight is given ,in According to the light and dark components The hyperparameters learned from the pixel histogram are used to obtain the corrected light and dark components through the following formula ,in Represents the corrected light and dark components The Pixels: , For the corrected light and dark components , through the light and dark component adaptive module Prediction Transform Coefficients : , The light and dark component adaptive module consists of two layers of convolution and deconvolution. Adaptive adjustments are made to suit image enhancement tasks; at the same time, the lightweight light and dark component adaptive module brings negligible computational overhead to the original image enhancement network.

[0030] Then, based on the transformation coefficients , transform the image enhancement learning process into the reflectivity space; specifically, the image enhancement network The input is transformed into , that is, transforming to the reflectivity space for image enhancement learning; the advantage of transforming to the reflectivity space is that it can avoid the influence of factors such as local high dynamic range; such as Figure 3 As shown, the horizontal axis represents the correlation coefficient and the vertical axis represents the number of samples. Compared with the original image space, the statistical similarity between the input image and the target enhanced image in the reflectance space is higher, which shows that by converting the original image space to the reflectance space, the difficulty of color mapping and image enhancement learning can be reduced.

[0031] The image enhancement network predicts the output as , multiply it by the transformation coefficient , and obtain the final enhanced result It is worth noting that through Transforming to reflectance space allows the image enhancement network to focus on learning components that are independent of illumination intensity.

[0032] Through the above design, the additional near-infrared band can be effectively utilized in mobile phone images, which can be applied to potential tasks such as tone mapping of high dynamic range scenes, image shadow removal, local brightness adjustment, and dark light enhancement.

[0033] The present invention provides a near-infrared driven mobile phone image enhancement method. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A near-infrared driven mobile phone image enhancement method, characterized in that: The following steps are involved: Step 1: Install an additional near-infrared spectral sensor on the mobile terminal and select The near-infrared spectral band, Represents the starting band of the spectral perception curve of the near-infrared spectral sensor, The end band of the spectral perception curve representing the near-infrared spectral sensor; RGB image captured by a mobile visible light camera , using near infrared spectral sensors to obtain near infrared images ; Step 2: Establish a visible light and near-infrared joint decoupling module to decouple near-infrared images With RGB image Mapped to the same feature space, and then output the light and dark components through the decoder ; Step 3: Establish a light and dark component embedding framework to embed the light and dark components in a plug-and-play manner. Embedded into image enhancement network The final enhanced result is obtained.

2. The method according to claim 1, characterized in that Step 1 includes: Step 1-1, using the outdoor near-infrared band selection method, select The near-infrared spectral band is used as the prior information for image enhancement; Step 1-2: The optical axis distance between the visible light camera and the near infrared spectrum sensor is less than the threshold; the RGB image is captured by the mobile visible light camera , , express The real number space, H and W represent RGB images respectively. height and width; Acquiring near-infrared images through near-infrared spectral sensors , , h and w represent near infrared images respectively height and width.

3. The method according to claim 2, characterized in that In step 1-1, the outdoor near-infrared band selection method includes: using outdoor sunlight as a light source, combining visible light and near-infrared scanning hyperspectral cameras to shoot a standard color card, and analyzing the change of spectral curves of the reflective spectrum bands of different colors of the standard color card from visible light to near-infrared bands. The spectrum curve of each color is recorded as , calculate the The rate of change of the spectrum curve of each color : = ， in Represents the band in the spectral dimension The amount of change, Indicates the rate at which the spectrum changes with wavelength; The band that satisfies the following formula The value of the starting band : ， in represents the threshold value of the change rate of the spectrum curve of different colors, and N represents the total number of colors; Combined with the detection cutoff frequency of silicon-based sensors in the near-infrared band, the starting band of the spectral perception curve of the near-infrared spectral sensor is established With stop band .

4. The method according to claim 3, characterized in that: Step 2 includes: The visible light and near infrared joint decoupling module is used to perform the following steps: With near infrared imaging Mapped to the same feature space, we get RGB images respectively The deep feature map With near infrared imaging The deep feature map ; In the feature space, the feature map is obtained by feature interpolation enhancement operation , based on the feature map , the decoder outputs the light and dark components ; ,in, is a single channel image.

5. The method according to claim 4, characterized in that In step 2, through the RGB encoder With near infrared encoder Convert RGB image With near infrared imaging Mapped to the same feature space, we get RGB images respectively The deep feature map With near infrared imaging The deep feature map : ， 。 6. The method according to claim 5, characterized in that In step 2, for the depth feature map The i-th point in , select the deep feature map and the i-th point 9 neighboring points , calculate the fusion weight based on the feature similarity measure , and then the feature map is obtained by the following formula : 。 7. The method according to claim 6, characterized in that Step 3 includes: the light and dark component embedding framework includes a light and dark component adaptive module , input image space transformation, output image space transformation; First, based on the light and dark component adaptive module Prediction Transform Coefficients ; Then perform input image space transformation, for the input image that needs to be enhanced , the input image By transforming the coefficients Transformed to reflectance space, passed through the image enhancement network in reflectance space Processing; finally, the output image space transformation is implemented, through the transformation coefficient Transform back to the original image space to get the final enhanced result.

8. The method according to claim 7, characterized in that In step 3, the light and dark component adaptive module It includes two layers of convolution and deconvolution, the brightness and darkness component adaptive module For light and dark components Adaptively adjust to image enhancement tasks.

9. The method according to claim 8, characterized in that In step 3, the light and dark components Adaptive normalization based on statistical mean to calculate light and dark components Middle Pixels Variance : = ， in Represents light and dark components The statistical mean of Then for the light and dark components Middle Pixels Adjust and get the corrected brightness and darkness components through the following formula : ， Where e is a natural constant; According to the light and dark components The hyperparameters learned from the pixel histogram, ; Represents the corrected brightness and darkness components The pixels; For the corrected light and dark components , through the light and dark component adaptive module Prediction Transform Coefficients : ， Transformation coefficient .

10. The method according to claim 9, characterized in that In step 3, the image enhancement network The input is transformed into , that is, transform it into reflectivity space for image enhancement learning; Image Enhancement Network The predicted output is , by transforming the coefficients Convert to get the final enhanced result .

Images