Night scene imaging method and system based on color and black-and-white dual cameras

Abstract

The present application relates to a kind of night scene imaging method and system based on color and black and white dual camera.The method: first, user presses shutter, system automatically sets the shooting parameter of color and black and white dual camera, and obtains low ISO color image and high ISO black and white image simultaneously through shutter;Second, in order to obtain high-quality imaging result, the final high-quality imaging is obtained by using black and white image to guide color image enhancement in low light degradation correction model (LDRM), wherein additive correction network is used to preliminarily enhance color image, and then multiplicative correction network is used to guide color image to finely enhance by using black and white image;Finally, the result predicted by using training model is compared with original unenhanced color image, and the completion of target in task is judged.The present application is best in accuracy performance compared with other low-light enhancement methods, and has excellent performance in subjective effect, and has obvious advantages in actual low-light imaging application scene.

Description

Technical Field

[0001] This invention relates to the field of underlying computer vision denoising technology, and in particular to a night scene imaging method and system based on dual color and monochrome cameras. Background Technology

[0002] Currently, with the increasing demand for night scene imaging in mobile devices, night scene imaging technology has been widely applied. However, night scene conditions inevitably lead to degradation in lighting conditions, color, noise, and blur, which has seriously affected the quality of night scene imaging. Obtaining high-quality night scene imaging in low-light environments has become a strong demand. One important means to solve the above problems is to explore methods to achieve high-quality night scene imaging to meet the complex real-world night scene imaging needs. Therefore, this invention designs a night scene imaging method and system based on dual color and monochrome cameras, ensuring rapid image capture and high-quality imaging under night scene conditions, to meet user needs in night scene imaging tasks. Summary of the Invention

[0003] The purpose of this invention is to provide a night scene imaging method and system based on dual color and monochrome cameras, which improves the night scene imaging quality compared with other methods.

[0004] To achieve the above objectives, the technical solution of the present invention is: a night scene imaging method based on color and monochrome dual cameras, comprising the following steps: Step S1: The user presses the shutter button, sets the shooting parameters of the color and monochrome dual cameras, and uses a short exposure time to obtain a low ISO color image and a high ISO monochrome image to obtain a color and monochrome image pair. The short exposure time is <0.1s. Step S2: Input the color and black and white image pairs, and use the black and white image to guide the color image enhancement in the Low Light Degradation Correction Model (LDRM) to obtain the final high-quality image. Step S3: Create a synthetic dataset, capture real dataset images, compare the restored results with the original images, and verify the reliability of the system.

[0005] In one embodiment of the present invention, step S1 is specifically implemented as follows: Step S11: The user presses the shutter button, and the monochrome camera automatically focuses and calculates the ISO value of the exposure x. Step S121: Set the color camera parameters, where ISO is set to 0.2x and exposure time is set to 0.05s; Step S122: Set the parameters of the black and white camera, where ISO is set to x and exposure time is set to 0.05s; Step S13: The color camera and the black and white camera simultaneously capture images to obtain color and black and white image pairs.

[0006] In one embodiment of the present invention, step S2 is specifically implemented as follows: Step S21: Call the hardware resources of the device, including CPU, GPU, memory and hard disk, load the low light degradation correction model LDRM, and send the color and black and white image pairs into the low light degradation correction model LDRM. Step S22: Perform preliminary enhancement on the low-light color image using the additive correction network (ARN), and convert the enhanced image from RGB to YUV channels, and depackage it into Y channel image and UV channel image; Step S231: Process the UV channel image by restoring the UV channel through the UV channel multiplicative correction network MRN_UV to obtain the enhanced UV channel features. Step S232: Process the Y channel image. In the Y channel multiplicative correction network MRN_Y, use the black and white image to guide the Y channel to restore it, and obtain the enhanced Y channel features. Step S24: Combine the enhanced UV channel features and Y channel features, further fuse and enhance the features through the detail enhancement module DEM, and then convert to the RGB channel to obtain the multiplicative corrected image; Step S25: Multiply the preliminary enhanced image obtained in step S22 with the multiplicative corrected image to obtain the final output image, which is then stored on the device.

[0007] In one embodiment of the present invention, step S3 is specifically implemented as follows: Step S31: Create a synthetic dataset by adding noise to simulate the difference between color and black-and-white dual cameras. Use the synthetic dataset to train and test model performance. Step S32: Capture real datasets by taking short-exposure pairs of color and black-and-white images and long-exposure high-quality color images using a fixed tripod. Use the real datasets to train and test model performance. Step S33: Use L1 loss and perceptual loss as the training loss of the model, evaluate the model training process in real time, and save the training model and data in real time. Step S34: Introduce referenced performance evaluation metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learning Perceptual Image Patch Similarity (LPIPS) to evaluate task completion.

[0008] In one embodiment of the present invention, in steps S1 to S3, it is recommended that the basic configuration of the accessories be no less than that in Table 1:

[0009] After the user presses the shutter, the color and monochrome dual cameras calculate the exposure based on the ambient brightness and automatically set parameters to capture a pair of color and monochrome images, which are then processed on the local device; the final image result will be saved on the device.

[0010] The present invention also provides a night scene imaging system based on a color and a monochrome dual-camera system, comprising: The data acquisition module allows the user to press the shutter button, set the shooting parameters of the color and monochrome dual cameras, and acquire a low-ISO color image and a high-ISO monochrome image using a short exposure time, resulting in a color and monochrome image pair. The short exposure time is <0.1s. The imaging module takes color and black-and-white image pairs as input and uses the black-and-white image to guide the color image enhancement in the Low Light Degradation Correction Model (LDRM) to obtain the final high-quality image. The testing module generates synthetic datasets and captures real datasets, comparing the restored results with the original images to verify the system's reliability.

[0011] Compared with the prior art, the present invention has the following beneficial effects: The method of the present invention provides a night scene imaging method and system based on color and monochrome dual cameras, which effectively improves the night scene imaging quality on mobile devices and has a very wide range of application prospects. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the overall workflow of the method and system in the embodiments of the present invention.

[0013] Figure 2 This is a method and system framework diagram in an embodiment of the present invention.

[0014] Figure 3 These are before-and-after comparison images of night scene imaging in the synthetic dataset of this invention.

[0015] Figure 4 These are before-and-after comparison images of night scene imaging based on a real dataset from an embodiment of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0017] This invention provides a night scene imaging method based on dual color and monochrome cameras, comprising the following steps: Step S1: When the user presses the shutter button, the system automatically sets the shooting parameters of the color and monochrome dual cameras, and uses a short exposure time (<0.1s) to obtain a low-ISO color image and a high-ISO monochrome image. Step S2: Input the color and black-and-white images into the algorithm, and use the black-and-white image to guide the color image enhancement in the Low Light Degradation Correction Model (LDRM) to obtain the final high-quality image; Step S3: Create a synthetic dataset, capture real dataset images, compare the restored results with the original images, and verify the reliability of the system.

[0018] The present invention also provides a night scene imaging system based on a color and a monochrome dual-camera system, comprising: The data acquisition module allows the user to press the shutter button, set the shooting parameters of the color and monochrome dual cameras, and acquire a low-ISO color image and a high-ISO monochrome image using a short exposure time, resulting in a color and monochrome image pair. The short exposure time is <0.1s. The imaging module takes color and black-and-white image pairs as input and uses the black-and-white image to guide the color image enhancement in the Low Light Degradation Correction Model (LDRM) to obtain the final high-quality image. The testing module generates synthetic datasets and captures real datasets, comparing the restored results with the original images to verify the system's reliability.

[0019] The following are specific implementation examples of the present invention.

[0020] This implementation provides a night scene imaging method based on dual color and monochrome cameras. The overall workflow is as follows: Figure 1 As shown, the system includes the following steps: Step S1: The user presses the shutter button, and the system automatically sets the shooting parameters of the color and monochrome dual cameras, using a short exposure time (<0.1s) to acquire a low-ISO color image and a high-ISO monochrome image; Step S2: The color and monochrome images are input into the algorithm, and the monochrome image guides the color image enhancement in the Low Light Degradation Correction Model (LDRM) to obtain the final high-quality image; Step S3: A synthetic dataset is created, a real dataset is captured, and the restoration result is compared with the original image to verify the reliability of the system.

[0021] In this embodiment, step S1, where the user presses the shutter button, automatically sets the shooting parameters for both the color and monochrome dual cameras, and uses a short exposure time (<0.1s) to acquire a low-ISO color image and a high-ISO monochrome image, specifically includes the following steps: Step S11: The user presses the shutter button, and the system automatically focuses on the monochrome camera and calculates the ISO value x for a reasonable exposure. Step S121: The system automatically sets the color camera parameters, with ISO set to 0.2x and exposure time set to 0.05s; Step S122: The system automatically sets the parameters of the black and white camera, where ISO is set to x and exposure time is set to 0.05s; Step S13: The color camera and the black and white camera simultaneously capture images to obtain color and black and white image pairs.

[0022] LDRM framework model such as Figure 2 In this embodiment, the low-light degradation correction model (LDRM) is used to guide color image enhancement using a black-and-white image to obtain a high-quality imaging result. In this embodiment, step S2 involves inputting both color and black-and-white image pairs into the algorithm, and using the black-and-white image to guide color image enhancement in the low-light degradation correction model (LDRM) to obtain the final high-quality imaging. This specifically includes the following steps: Step S21: The system automatically calls the hardware resources of the device, such as CPU, GPU, memory and hard disk, to load the Low Light Degradation Correction Model (LDRM) and send the color and black and white image pairs into the model; Step S22: Perform preliminary enhancement on the low-light color image using an additive correction network (ARN), and convert the enhanced image from RGB to YUV channels, and depackage it into Y channel image and UV channel image; Step S231: Process the UV channel image by restoring the UV channel through the UV channel multiplicative correction network (MRN_UV) to obtain the enhanced UV channel features; Step S232: Process the Y channel image. In the Y channel multiplicative correction network (MRN_Y), use the black and white image to guide the Y channel restoration to obtain the enhanced Y channel features. Step S24: Combine the enhanced UV channel features and Y channel features, further fuse and enhance the features through the detail enhancement module (DEM), and then convert to the RGB channel to obtain the multiplicative corrected image; Step S25: Multiply the preliminary enhanced image obtained in step S21 with the multiplicative corrected image to obtain the final output image, which is then stored on the device.

[0023] Furthermore, a synthetic dataset is created, and real-world datasets are captured. The restored images are then compared with the original images to verify the reliability of the model. This specifically includes the following steps: Step S31: Create a synthetic dataset by adding noise to simulate the difference between color and black-and-white dual cameras. Use the synthetic dataset to train and test model performance. Step S32: Capture real datasets by taking short-exposure pairs of color and black-and-white images and long-exposure high-quality color images using a fixed tripod. Use the real datasets to train and test model performance. Step S33: Use L1 loss and perceptual loss as the training loss of the model, evaluate the model training process in real time, and save the training model and data in real time. Step S34: Introduce referenced performance evaluation metrics such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learned perceptual image patch similarity (LPIPS) to evaluate the task completion.

[0024] In steps S1 to S3, it is recommended that the basic configuration of the accessories be no less than that in Table 1:

[0025] After the user presses the shutter button, the color and monochrome dual cameras calculate the appropriate exposure based on the ambient brightness and automatically set parameters to capture a pair of color and monochrome images, which are then processed locally on the device. The final image result is saved on the device.

[0026] The present invention also provides a night scene imaging system based on a color and a monochrome dual-camera system, comprising: The data acquisition module allows the user to press the shutter button, set the shooting parameters of the color and monochrome dual cameras, and acquire a low-ISO color image and a high-ISO monochrome image using a short exposure time, resulting in a color and monochrome image pair. The short exposure time is <0.1s. The imaging module takes color and black-and-white image pairs as input and uses the black-and-white image to guide the color image enhancement in the Low Light Degradation Correction Model (LDRM) to obtain the final high-quality image. The testing module generates synthetic datasets and captures real datasets, comparing the restored results with the original images to verify the system's reliability.

[0027] Figure 3 These are before-and-after comparison images of night scene imaging in the synthetic dataset of this invention. Figure 4 These are before-and-after comparison images of night scene imaging based on a real dataset from an embodiment of the present invention.

[0028] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A night scene imaging method based on dual color and monochrome cameras, characterized in that, Includes the following steps: Step S1: The user presses the shutter button, sets the shooting parameters of the color and monochrome dual cameras, and uses a short exposure time to obtain a low ISO color image and a high ISO monochrome image to obtain a pair of color and monochrome images. The short exposure time is <0.1s. Step S2: Input the color and black-and-white image pairs, and use the black-and-white image to guide the color image enhancement in the Low Light Degradation Correction Model (LDRM) to obtain the final high-quality image; the specific implementation is as follows: Step S21: Call the hardware resources of the device, including CPU, GPU, memory and hard disk, load the low light degradation correction model LDRM, and send the color and black and white image pairs into the low light degradation correction model LDRM. Step S22: Perform preliminary enhancement on the low-light color image using the additive correction network (ARN), and convert the enhanced image from RGB to YUV channels, and depackage it into Y channel image and UV channel image; Step S231: Process the UV channel image by restoring the UV channel through the UV channel multiplicative correction network MRN_UV to obtain the enhanced UV channel features. Step S232: Process the Y channel image. In the Y channel multiplicative correction network MRN_Y, use the black and white image to guide the Y channel to restore it, and obtain the enhanced Y channel features. Step S24: Combine the enhanced UV channel features and Y channel features, further fuse and enhance the features through the detail enhancement module DEM, and then convert to the RGB channel to obtain the multiplicative corrected image; Step S25: Multiply the preliminary enhanced image obtained in step S22 with the multiplicative corrected image to obtain the final output image, which is then stored on the device. Step S3: Create a synthetic dataset, capture real-world dataset images, and compare the restored results with the original images to verify the system's reliability; the specific implementation is as follows: Step S31: Create a synthetic dataset by adding noise to simulate the difference between color and black-and-white dual cameras. Use the synthetic dataset to train and test model performance. Step S32: Capture real datasets by taking short-exposure pairs of color and black-and-white images and long-exposure high-quality color images using a fixed tripod. Use the real datasets to train and test model performance. Step S33: Use L1 loss and perceptual loss as the training loss of the model, evaluate the model training process in real time, and save the training model and data in real time. Step S34: Introduce referenced performance evaluation metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learning Perceptual Image Patch Similarity (LPIPS) to evaluate task completion.

2. The night scene imaging method based on a color and monochrome dual-camera system according to claim 1, characterized in that, Step S1 is implemented as follows: Step S11: The user presses the shutter button, and the monochrome camera automatically focuses and calculates the ISO value of the exposure x. Step S121: Set the color camera parameters, where ISO is set to 0.2x and exposure time is set to 0.05s; Step S122: Set the parameters of the black and white camera, where ISO is set to x and exposure time is set to 0.05s; Step S13: The color camera and the black and white camera simultaneously capture images to obtain color and black and white image pairs.

3. The night scene imaging method based on a color and monochrome dual-camera system according to claim 1, characterized in that, In steps S1 to S3, the basic configuration of the accessories is no less than: color camera: IMX600, monochrome camera: IMX600, CPU: Snapdragon 870, GPU: Adreno 650, memory: 8G, hard drive: 64G; after the user presses the shutter, the color and monochrome dual cameras calculate the exposure according to the ambient brightness and automatically set the parameters to capture color and monochrome image pairs, and process them on the local device; the final imaging result will be saved in the device.

4. A night scene imaging system based on color and monochrome dual cameras, characterized in that, include: The data acquisition module allows the user to press the shutter button, set the shooting parameters of the color and monochrome dual cameras, and acquire a low-ISO color image and a high-ISO monochrome image using a short exposure time, resulting in a color and monochrome image pair. The short exposure time is <0.1s. The imaging module takes color and monochrome image pairs as input and uses the monochrome image to guide the color image enhancement in the Low Light Degradation Correction Model (LDRM) to obtain the final high-quality image. The specific implementation is as follows: (1) Call the hardware resources of the device, including CPU, GPU, memory and hard disk, load the low light degradation correction model LDRM, and send the color and black and white image pairs into the low light degradation correction model LDRM. (2) The low-light color image is initially enhanced by the additive correction network (ARN), and the enhanced image is converted from RGB to YUV channel and then depackaged into Y channel image and UV channel image; (3.1) The UV channel image is processed by restoring the UV channel through the UV channel multiplicative correction network MRN_UV to obtain the enhanced UV channel features; (3.2) Process the Y channel image and use the black and white image to guide the restoration of the Y channel in the Y channel multiplicative correction network MRN_Y to obtain the enhanced Y channel features; (4) The enhanced UV channel features and Y channel features are combined, and the features are further fused and enhanced by the detail enhancement module DEM. Then, they are converted to the RGB channel to obtain the multiplicative corrected image. (5) Multiply the preliminary enhanced image obtained in step (2) with the multiplicative corrected image to obtain the final output image, which is then stored on the device; The testing module generates synthetic datasets and captures real-world datasets, comparing the restored images with the original images to verify the system's reliability. The specific implementation is as follows: 1) Create a synthetic dataset by adding noise to simulate the difference between color and monochrome dual cameras, and use the synthetic dataset to train and test model performance; 2) Capture real datasets by taking short-exposure pairs of color and black-and-white images and long-exposure high-quality color images using a fixed tripod. Use real datasets to train and test model performance. 3) Use L1 loss and perceptual loss as the training loss of the model to evaluate the model training process in real time and save the training model and data in real time. 4) Introduce reference performance evaluation metrics such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learning-aware image patch similarity (LPIPS) to evaluate task completion.

Images