Night image enhancement model based on Retinex framework

Abstract

The invention relates to a framework (ORF) based on first-level Retinex. A disturbance term is introduced into the reflectivity to correct an original Retinex model so as to simulate corrosion. Secondly, an illumination guide transformer (IGT) is provided to simulate a remote dependency relationship. And finally, the ORF is inserted into the IGT to serve as a damage restorer.

Description

1. Technical Field

[0001] Convolutional Neural Networks 2 Background technology

[0002] A single-stage Retinex framework based on Retinex theory.

[0003] The low-light image I∈RH×W×3 can be decomposed into a reflectance image R∈RH×W×3 and a light map L∈RH×W, such as I=R⊙

[0004] L(1), where ⊙ represents the multiplication of elements. This Retinex model assumes that I is not corrupted, which is inconsistent with the real underexposed scene. We analyze that the corruption mainly arises from two factors. First, the high ISO and long exposure imaging settings of dark scenes inevitably introduce noise and artifacts. Second, the lighting process may amplify noise and artifacts, and also lead to underexposure / overexposure and color distortion. In order to model the corruption, we reformulate equation (1) by introducing perturbation terms of R and L respectively, as (2) and represents the perturbation. Similar to [15, 18, 49], we regard R as a well-exposed image. To lighten I, we cleverly multiply both sides of Eq. (2) by a lighting map Lˉ, so that Lˉ⊙L=1 is equal to IˉLˉ=R+Rˉ(L′⊙Lˉ)+(R′⊙(L+L′))⊙Lˉ, (3) where R′⊙(L+L′) represents the noise and artifacts hidden in the dark scene and amplified by Lˉ. R⊙(L′⊙Lˉ) represents underexposure / overexposure and color distortion caused by the lighting process. We simplify Eq. (3) to Ilu=I⊙Lˉ=R+C, (4) where Ilu∈RH×W×3 represents the activation image and C∈RH×W×3 represents the overall corruption term. Subsequently, we denote ORF as (Ilu, Flu) = E(I, Lp), Ien = R(Ilu, Flu), (5), where E denotes the illumination estimator and R denotes the damage restorer. E takes I and its illumination prior map Lp∈RH×W as input. Lp=meanc(I), where meanc represents the operation of computing the mean value of each pixel along the channel dimension. E outputs the lit image Ilu and the lit features Flu∈RH×W×C.

[0005] Then, Ilu and Flu are input into R to restore the damage and produce an enhanced image Ien∈RH×W×3. E first fuses the concatenation of i and Lp using conv1×1 (convolution with kernel size 1). We note that well-exposed regions can provide semantic contextual information for under-exposed regions. Therefore, a depthwise separable conv5×5 model is adopted to model the interaction between regions under different lighting conditions, thereby producing the lighting feature Flu. Then, E aggregates Flu using conv1×1 to produce a lighting map Lˉ∈RH×W×3. We set Lˉ to a three-channel RGB tensor instead of a single-channel tensor as in [15,18] to improve its representation ability in simulating nonlinearity across RGB channels for color enhancement. Lˉ is then used to lighten I in Eq. (3). Discussion. Unlike previous Retinex-based deep learning methods [30, 49, 54, 65, 66], our ORF estimates Lˉ instead of the illumination map L, because if ORF estimates L, the lit image is obtained by element-wise division (i / L). Computers are vulnerable to such operations. The values ​​of the tensor can be very small (sometimes even equal to 0), and this division can easily lead to data overflow problems. In addition, small errors randomly generated by computers will be amplified by this operation, resulting in inaccurate estimates. Therefore, modeling Lˉ is more robust. Previous Retinex-based deep learning methods mainly focus on suppressing damage such as noise on the reflection image, i.e., R} in Eq. (2), while ignoring the estimation error on the illumination map, i.e., L} in Eq. (2), which can easily lead to underexposure / overexposure and color distortion during the lighting process. In contrast, our ORF considers all these damages and uses R to restore them. 3. Summary of the invention

[0006] Night image enhancement model 4. Specific implementation methods

[0007] 1) Operating environment

[0008] 1.1 Hardware

[0009] a. Processor model: 1. Memory capacity: >50M;

[0010] b. Required external storage, media, recording format, equipment model and number, online / offline: None;

[0011] c. Model and number of data transmission equipment and conversion equipment: 1.

[0012] 1.2 Support Software

[0013] Anaconda, pytorch.

[0014] 2) Usage process

[0015] 2.1 Installation and Initialization

[0016] 1. Install conda environment

[0017] conda create -n retinexformer python=3.7

[0018] conda activate Retinexformer

[0019] 2. Install Dependencies

[0020] conda install pytorch=1.11 torchvision cudatoolkit=11.3-c pytorch

[0021] pip install matplotlib scikit-learn scikit-image opencv-python yacsjoblib natsort h5py tqdm tensorboard

[0022] pip install einops gdown addict future lmdb numpypyyaml ​​requestsscipy yapf lpips

[0023] 3. Install BasicSR

[0024] python setup.py develop--no_cuda_ext

[0025] 2.2 Input Data

[0026] Download the following datasets:

[0027] LOL-v1 Baidu Cloud Disk, Google Drivecyh2

[0028] LOL-v2 Baidu disk, Google Drivecyh2

[0029] SID Baidu Drive, Google Drivegplv

[0030] SMID Baidu Cloud Disk, Google Drivebtux

[0031] SDSD-Indoor Baidu Cloud Disk, Google1

[0032] SDSD-outdoor Baidu disk, google2

[0033] MIT-Adobe FiveK Baidu disk), googledrive, official cyh2

[0034] NTIRE 2024 Baidu disk, Google Drive link for training input, training GT and mini-val sets. cyh2 Note:

[0035] (1) Please use bandizip to decompress the files of SMID, SDSD-INDOOR and SDSD-outdoor.

[0036] zip.z01

[0037] (2) Please process the original images of the MIT Adobe FiveK dataset in sRGB output mode, or directly download and use our processed sRGB image pair cyh2 in Baidu Netdisk (code: ) and Google Drive.

[0038] (3) Please download from Google Drive or Baidu Drive and put it in a folder

[0039] text_list.txtggbhdata / SMID / SMID_Long_np.

[0040] Baidu Netdisk (code:) or Google Drivecyh2 (the above data is from

[0041] github.com / caiyuanhao1998 / Retinexformer / blob / master / README.md ) website provided.

[0042] 2.3 Testing

[0043] Download the models from the above URL and put them into the folder cyh2pretrained_weights

[0044] #activate the environment

[0045] conda activate Retinexformer

[0046] #LOL-v1

[0047] python3 Enhancement / test_from_dataset.py--opt Options / RetinexFormer_LOL_v1.yml--weights pretrained_weights / LOL_v1.pth--datasetLOL_v1

[0048] #LOL-v2-real

[0049] python3 Enhancement / test_from_dataset.py--opt Options / RetinexFormer_LOL_v2_real.yml--weights pretrained_weights / LOL_v2_real.pth--dataset LOL_v2_real

[0050] #LOL-v2-synthetic

[0051] python3 Enhancement / test_from_dataset.py--opt Options / RetinexFormer_LOL_v2_synthetic.yml--weights pretrained_weights / LOL_v2_synthetic.pth--dataset LOL_v2_synthetic

[0052] #SID

[0053] python3 Enhancement / test_from_dataset.py--opt Options / RetinexFormer_SID.yml--weightspretrained_weights / SID.pth--dataset SID

[0054] #SMID

[0055] python3 Enhancement / test_from_dataset.py--opt Options / RetinexFormer_SMID.yml--weightspretrained_weights / SMID.pth--dataset SMID

[0056] #SDSD-indoor

[0057] python3 Enhancement / test_from_dataset.py--opt Options / RetinexFormer_SDSD_indoor.yml--weights pretrained_weights / SDSD_indoor.pth--dataset SDSD_indoor

[0058] #SDSD-outdoor

[0059] python3 Enhancement / test_from_dataset.py--opt Options / RetinexFormer_SDSD_outdoor.yml--weights pretrained_weights / SDSD_outdoor.pth--dataset SDSD_outdoor

[0060] #FiveK

[0061] python3 Enhancement / test_from_dataset.py--opt Options / RetinexFormer_FiveK.yml--weightspretrained_weights / FiveK.pth--dataset FiveK

[0062] #NTIRE

[0063] python3 Enhancement / test_from_dataset.py--opt Options / RetinexFormer_NTIRE.yml--weightspretrained_weights / NTIRE.pth--datasetNTIRE--self_ensemble

[0064] #MST_Plus_Plus trained with 4GPUs on NTIRE

[0065] python3 Enhancement / test_from_dataset.py --opt Options / MST_Plus_Plus_NTIRE_4x1800.yml --weights pretrained_weights / MST_Plus_Plus_4x1800.pth --datasetNTIRE --self_ensemble

[0066] #MST_Plus_Plus trained with 8GPUs on NTIRE

[0067] python3 Enhancement / test_from_dataset.py --opt Options / MST_Plus_Plus_NTIRE_8x1150.yml --weights pretrained_weights / MST_Plus_Plus_8x1150.pth --datasetNTIRE --self_ensemble

[0068] 2.4 Evaluation of Image Enhancement Effect

[0069] Use the Params function to evaluate the parameters and computational complexity of the model.

[0070] from utils import my_summarymy_summary(RetinexFormer(),256,256,3,1)

Claims

1. This model is a night image enhancement model based on the Retinex framework (ORF).

2. This model modifies the original Retinex model by introducing a perturbation term in the reflectivity to simulate corrosion.

3. The model proposes an illumination-guided transformer (IGT) to simulate long-range dependencies.

4. The model inserts ORF into IGT as a damage restorer.