DEPMD-Net-based low-illumination and rainy-day image enhancement method

Abstract

The invention provides a low-illumination rainy day image enhancement method based on DEPMD-Net. Through heterogeneous double teachers and multi-stage distillation, polarized rain-light attention and implicit representation are introduced, separation and fusion of raindrop high-frequency information and illumination low-frequency information are realized, and an image with balanced brightness and without rain interference is generated. The method comprises the following steps: 1) synthesizing low-illumination rainy day data based on Rain100L and LOL-v1; 2) pre-training a brightness expert and a rain removal expert, and obtaining prior through bidirectional KL mutual learning; 3) utilizing brightness-rain mask statistics to construct joint priori, and adopting non-zero mean anchoring to simulate night rain distribution in diffusion noise scheduling; 4) proposing PiDNet-X to recover features through cooperation of a double-branch structure and rain-light attention, 5) performing staged distillation through a soft label and feature prompt, obtaining lightweight Student-C / F through a Born-Again strategy, and realizing night rain scene self-correction by double students in a reasoning stage through moving average and mutual learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of image enhancement, digital image processing technology, polarized color space modeling, implicit neural representation and heterogeneous distillation network, and specifically relates to a method for low-illumination superimposed rainy day image enhancement based on a dual-expert progressive mutual-distillation network (DEPMD-Net). BACKGROUND

[0002] With the rapid development of intelligent transportation, night security monitoring, unmanned driving and consumer mobile terminals, higher requirements are put forward for the robustness of visual perception systems in harsh environments. The images in the night rain scene are often simultaneously affected by the following multiple sources of degradation: global brightness deficiency, which leads to low overall contrast, color distortion, and difficulty in distinguishing target details; long and thin stripes and scattered raindrops that destroy texture and edge structure; and nonlinear highlights generated by light sources on the surface of raindrops, which further exacerbate image degradation.

[0003] Traditional image enhancement methods such as Retinex variants, histogram equalization or single-task CNNs often focus on a single source of degradation: only brightening methods amplify rain marks simultaneously, producing overexposure artifacts; only rain removal methods ignore the limited dynamic range caused by low illumination, resulting in dark recovery results. Although deep learning end-to-end networks can consider both types of degradation to some extent, they still have the following shortcomings: a single backbone cannot simultaneously correct low-frequency illumination and suppress high-frequency rain; the statistical prior of low illumination and rain is not effectively encoded, and the noise sampling of the diffusion model still assumes zero mean, making it difficult to accurately align the real night rain noise distribution; existing distillation focuses on classification or single-task denoising, and lacks a dual-teacher collaborative strategy for complex degradation scenarios.

[0004] To break through the above bottlenecks, an integrated solution is needed that can decouple and collaboratively process low-frequency illumination and high-frequency rain information, fully embed priori, and realistically simulate night rain noise during the diffusion process; achieve high performance and lightweight through multi-stage distillation; maintain an end-to-end closed loop for easy deployment of practical systems.

[0005] The DEPMD-Net proposed in the present application adopts a dual-path heterogeneous structure with a brightness teacher and a rain removal teacher, and transfers to a unified student network through bidirectional mutual learning and progressive distillation; jointly uses brightness-rain mask priori to introduce non-zero mean anchored noise in the conditional diffusion model; and cooperates with polarized rain-light attention, implicit neural representation rain suppression modules and online mutual learning mechanisms to achieve the organic unity of low-illumination rain scene image brightness balance, rain mark removal and detail preservation, providing a new technical approach for image enhancement in complex night rain environments. SUMMARY

[0006] The purpose of the present application is to provide an end-to-end enhancement method capable of simultaneously coping with low-illumination and rainy-day interference, improving image details, and jointly brightening and removing rain from low-illumination superimposed rainy-day images. To this end, the present application proposes DEPMD-Net, which realizes the target through the following technical solutions:

[0007] (1) Overall architecture design: the DEPMD-Net proposed by the present application is as shown in Figure 1 , which faces the image enhancement problem of low-illumination superimposed rainy-day scenes, follows the four core principles of "low-frequency brightness compensation + high-frequency rain mark suppression + prior dynamic gating + step-by-step distillation compression", and is composed of six functional domains from top to bottom, including a statistical prior channel, a rain removal high-frequency branch, a brightness low-frequency branch, a cross-branch polarization fusion layer, a gradual decoding and double student online correction layer;

[0008] (1.1) Global statistical prior channel: the global brightness mean mu_I, rain mask density mu_r and binary rain mask M_r of the input image are extracted at one time, and are spliced into a prior vector phi in the channel dimension; the vector is gated by MSLA-Polar as shown in Figure 3 , to generate a channel vector w_c containing 4x4, 8x8 and 16x16 multi-scale window weights, which is used to dynamically modulate the attention strength of the subsequent double branch, and the prior information runs through the whole network. Using RLA-Duo as shown in Figure 5 , the brightness residual and the rain mask construct two Query / Key, which apply higher attention to "dark and rain-intensive" pixels, and reduce intervention in bright areas and sparse rain areas; the generated weighted features are injected into the high-frequency rain removal branch and the low-frequency brightness branch, respectively, to provide more accurate saliency guidance for subsequent decoupling;

[0009] (1.2) Rain removal high-frequency branch: first, perform slice embedding in the polar DCT frequency domain to encode the high-frequency rain mark directionality into Token; then introduce CA-WinxDilatedSwin as shown in Figure 2 to obtain a variable window through content-aware clustering, and then superimpose a four-level dilated convolution Query / Key bias to efficiently capture 0°, 45°, 90° and other diagonal rain filaments; through the rain removal decoder (with built-in BFPU-RainSwitch as shown in Figure 8 ) to suppress residual rain strips and restore texture details, output high-frequency rain removal features F_R;

[0010] (1.3) Brightness low-frequency branch: first generate a frequency-conditioned convolution kernel gamma(f), which respectively adjusts the light compensation and high-frequency color noise suppression in large and small rain areas according to the frequency radius difference; the low-frequency path uses Retinex3x3 deep separable stack for global brightening, and the high-frequency path uses Tiny-DDPM 4-step denoising network to eliminate color noise and fine raindrops, finally outputting low-frequency brightness features F_L;

[0011] (1.4) Cross-branch polarization fusion layer: In the PRLA polarization rain-light attention module, such as Figure 7 As shown, F_R, F_L and polarization coordinates (r, θ) are input together; spatial-channel asymmetric weights ensure that the deraining features of the "dark and rain-dense" regions dominate, while the "bright" regions retain more illumination details; then the two features are fused for the first time at the residual Add node;

[0012] (1.5) Progressive Decoding and Multi-Level Correction: The fused features are sequentially processed through the RainSwitch texture compensation unit, such as... Figure 8 As shown, the HiLo-DFA frequency divider diffuser is as follows: Figure 4 As shown, mesoscale texture restoration and frequency division-based progressive denoising are completed; at the end, Student-C / Student-F dual student branches are connected in parallel and exponential moving average (EMA) mutual learning is performed during inference to generate the final enhanced image.

[0013] (1.6) Dual-student progressive distillation-mutual learning, the overall process is as follows: Figure 9 As shown: The training phase employs a three-stage strategy of "dual-teacher pre-training → hint-distillation → born-again compression," transferring the complementary knowledge of the luminance teacher and the de-raining teacher to the PiDNet-X student network, such as... Figure 6 During the inference phase, Student-C / Student-F continue to maintain lightweight mutual learning and online temperature scaling correction to ensure real-time adaptability to extreme night rain samples;

[0014] (2) Improved dataset: Based on the public Rain100L dataset, the overall brightness was adjusted according to the random γ transformation and brightness decay curve to generate low-light rain images, with a total of about 5,000 images; rain streaks and raindrops based on physical models or real sampling were injected into the existing low-light image dataset LOL-v1 to generate low-light rain images, with a total of about 6,500 images; 80% of them were selected as the training set and 20% were selected as the test set.

[0015] (3) Training a heterogeneous dual-teacher network, including the following sub-steps;

[0016] (3.1) Luminance Teacher Network TL: The HVI-CIDNet backbone is used, and the input is a six-channel tensor (RGB+) after HVI color gamut separation. + +I), the brightness teacher provides a low-frequency global illumination prior and outputs a brightness-corrected image. ;

[0017] (3.2) Rain-removing teacher network T-R: using NeRD-Rain backbone, embedding coordinate-feature hybrid implicit neural representation (INR) into Transformer-UNet network, rain-removing teacher provides high-frequency rain streak suppression capability, and outputs a rain-removed image ;

[0018] (3.3) Bidirectional mutual learning: using symmetric KL divergence and InfoNCE loss to complementarily align the features of and , mutual learning enables the two networks to share advantageous features, laying a complementary knowledge foundation for subsequent distillation;

[0019] (4) Generate joint brightness-rain mask prior and anchor noise, including the following sub-steps:

[0020] (4.1) Calculate the global brightness mean μ I and rain mask density μᵣ;

[0021] (4.2) Encode μ l , μᵣ and rain mask Mᵣ, generate a 128-dimensional prior vector φ through a prior mapping network, and minimize L prior for correction;

[0022] (4.3) Use non-zero mean noise in conditional diffusion forward noise scheduling to simulate night rain noise distribution, where non-zero mean anchoring makes the diffusion process truly reflect the statistical characteristics of night rain noise, improving the modeling ability of the diffusion model for complex degradation;

[0023] (5) Construct a unified student network PiDNet-X structure as shown in Figure 6 , and perform distillation training, including the following sub-steps:

[0024] (5.1) Design a double-branch backbone: the brightness branch uses a lightweight Deform-Conv-UNet to restore low-frequency illumination; the rain-removing branch uses a MobileViT-S+SIREN-INR to suppress high-frequency rain streaks;

[0025] (5.2) Insert a polarized rain-light attention PRLA module as shown in Figure 7 between the two branches Bottleneck to dynamically fuse rain mask and brightness information for spatial domain rain streak suppression;

[0026] (5.3) Introduce HVI-Gate gates as shown in Figure 7 at the input end of each layer feature to modulate the feature F and suppress color noise;

[0027] (5.4) Add a BFPU-RainSwitch module as shown in Figure 8 at the Decoder node of the rain-removing branch.As shown, the rain streak mis-transmission is blocked, and the detail compensation is enhanced;

[0028] (5.5) The double-teacher output soft label and the intermediate feature are taken as a hint, distilled to the PiDNet-X through a gated fusion, to obtain a coarse student Student-C;

[0029] (5.6) The Student-C is compressed through Born-Again compression, and the number of channels is reduced by 1 / 2, to obtain a fine student Student-F;

[0030] (6) Online mutual learning and adaptive correction;

[0031] (6.1) The Student-C and the Student-F are simultaneously executed in the inference stage, and the outputs are updated through an exponential moving average (EMA) weight η≤0.1;

[0032] (6.2) For difficult example images, the probability distribution is readjusted using a mutual learning temperature scaling τ=2.0, to improve robustness;

[0033] (6.3) An enhanced result Î with balanced brightness, removed rain marks and rich details is output.

[0034] The application faces the image enhancement demand of the night rainfall scene, and proposes a joint brightening-rain removal method based on DEPMD-Net. First, the input image is subjected to size unification and intensity normalization, and is converted to the HVI color domain to obtain a clean brightness channel and a color channel. Then, a heterogeneous double backbone network is constructed: the high-frequency branch inherits the NeRD-Rain structure of the rain removal teacher, focusing on raindrop rain mark and edge texture recovery; the low-frequency branch inherits the HVI-CIDNet backbone of the brightness teacher, responsible for global illumination and structure reconstruction. The two branches introduce polarized rain-light attention (PRLA) in the feature fusion stage as shown in Figure 7 , and the HVI-Gate gate as shown in Figure 7 , to dynamically coordinate rain streak suppression and illumination compensation. The brightness histogram mean and the coarse rain mask density are extracted through a prior encoder to generate a joint prior vector, and the noise scheduling parameter is used to map it to a non-zero mean anchor noise, so that the noise injected in the forward diffusion stage accurately conforms to the night rain statistical characteristics. In the reverse diffusion process, the double backbone predicts high and low frequency noise residuals respectively, and updates the latent variable with an adaptive weight fusion; finally, the denoising latent variable is mapped back to the sRGB space through the BFPU-RainSwitch closed-loop decoder as shown in Figure 8 , to output an enhanced image with balanced brightness, rain mark elimination and rich texture. The method fully utilizes the high and low frequency decoupling advantage, the prior anchor diffusion mechanism and the heterogeneous distillation network technology, and can simultaneously recover details and global structure in complex night rain environment, to generate high-quality and visually natural results. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is the overall network structure diagram of DEPMD-Net in the embodiment of the application;

[0036] Figure 2 is the cross-polarization dual-domain fusion transformer module structure diagram of CPD-Fusion in the embodiment of the application;

[0037] Figure 3 is the multi-scale polarized gate attention module structure diagram of MSLA-PolarGate in the embodiment of the application;

[0038] Figure 4 is the HiLo-DFA frequency division diffusion attention module structure diagram in the embodiment of the application;

[0039] Figure 5 is the RLA-Duo rain-light asymmetric dual attention module structure diagram in the embodiment of the application;

[0040] Figure 6 is the PiDNet-X double-branch student network structure diagram in the embodiment of the application;

[0041] Figure 7 is the PRLA and HVI-Gate gate module structure diagram in the embodiment of the application;

[0042] Figure 8 is the BFPU-RainSwitch closed-loop decoder module structure diagram in the embodiment of the application;

[0043] Figure 9 is the double-student progressive distillation-interlearning structure diagram in the embodiment of the application; DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present application clearer and more obvious, the content of the present application will be further described below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

[0045] Reference Figure 1 A low-illumination superimposed rainy day image enhancement method based on DEPMD-Net, comprising the following steps:

[0046] (1) Data preprocessing: First, clean / rain paired samples from the Rain100L dataset were selected. Brightness attenuation of γ∈[2.0,2.4] was applied to the clean image and the rain image was attenuated synchronously to maintain the consistency of rain texture details, thus obtaining about 5000 "low light + rain" composite images (accounting for 70% of the total). Then, 1500 original night scene low light images from the LOL-v1 dataset (accounting for 30%) were introduced and divided into training and validation at an 8:2 ratio. In addition to the original RGB three channels, H, V, and I polarization color features were calculated simultaneously. First, the image size was adjusted to 512*512 pixels and normalized to [-1,1]. Finally, after histogram equalization, it was concatenated with RGB to form a six-channel input to the subsequent joint brightness-rain mask prior mapping network.

[0047] (1.1) The image is standardized to 512×512 pixels using a proportional scaling and reflection filling method, where the original image size is H×W, s=max(H,W), and the scaling factor k=512 / s, as shown in the following formula:

[0048]

[0049] (1.2) If one side is less than 512px after scaling, use reflection fill to make it full. This method maintains the aspect ratio and reduces interpolation artifacts. The formula is as follows:

[0050]

[0051] (1.3) To improve training stability, pixel values ​​are linearly mapped to the interval [−1, 1]:

[0052]

[0053] (1.4) Perform random γ transformation and scaling on the rainy image to simulate insufficient night scene illumination. Select a γ value of 2 and an α value of 0.5. The formula is as follows:

[0054]

[0055] Rain patterns / drops are rendered using particle ray tracing, generating a rain layer R with an alpha channel and its binary rain mask M. r Rain streak angle θ∈(-30°, 30°), raindrop radius r∈(3.5px, 1). The rain layer is overlaid onto the low-light image I in ExDark. d Simultaneously record the rainfall density statistic μ r The formula is as follows:

[0056]

[0057] The value of β used in the application is 0.6, and the low-light image I with rain effect can be synthesized by the above formula.

[0058] (2) Training the heterogeneous dual-teacher network: for the composite degraded samples generated in step (1), first, the input image is uniformly converted to the HVI color domain and spliced with the original RGB channel to obtain a 6-channel tensor; the tensor is sent to the brightness teacher network T-L, and the backbone HVI-CIDNet thereof focuses on global brightness and color structure recovery through low-pass gating and context extraction module. The optimization objective of T-L is composed of brightness L1 loss, MS-SSIM structure preservation term and CIELab ΔE color difference constraint, and the formula is as follows:

[0059]

[0060] Ensure that the color is true and the details are kept while highlighting. The same degraded image is also sent to the rain removal teacher network T-R together with its rain mask M_r. T-R embeds a coordinate-feature hybrid implicit neural representation (INR) at the encoding end, which can analyze high-frequency rain marks and model raindrop highlights; its loss function is stacked with pixel L 1、 perception distance and Focal rain removal term based on rain mask, and the formula is as follows:

[0061]

[0062] The loss function promotes the network to obtain a larger gradient in the rain pixel area and accurately suppresses the stripes and scattered raindrops. In order to further finely suppress the oblique high-frequency rain marks, a CPD-Fusion transformer is introduced at the bottleneck of the rain removal teacher network T-R, as shown in Figure 2The input features of the rain removal branch Bottleneck are first mapped to the polarization frequency domain: a Polar-DCT is performed on the CxHxW tensor to divide the image energy into multiple directional blocks according to the polarization angle φ; after the PatchToken is generated in the frequency domain, the traditional double Swin-Tiny branch is upgraded to a CA-WinxDilation Swin structure to more accurately capture diagonal and coarse rain streaks in the form of adaptive window clustering combined with directional dilated convolution. The specific process is as follows: first, content-aware clustering is performed on the Token sequence after Polar-DCT blocking, and adjacent Tokens are dynamically combined according to the cosine similarity, outputting a variable window set {W_c}; then, with each clustering center as a seed, a 3x3 convolution with dilation rates d=1, 2, 3, 4 is applied in k-space to generate Query / Key guided bias, so that the window size automatically matches the rain streak span. The Query / Key / Value in the clustering window enter the modified SwinBlock to complete multi-head attention calculation, and the left and right branches use different initial clustering thresholds for 90° vertical rain and 45° diagonal rain respectively, outputting high-frequency rain suppression features and medium-frequency texture features. After the two-branch results are rescaled by the angle weights generated by PolarGate according to μ_I and μ_r, they are converted back to the time domain through 1x1 frequency domain convolution and Inverse-DCT, and added to the Bottleneck residual, which can reduce the coarse rain streak residual rate by 25% while maintaining the global frequency domain advantage of CPD-Fusion.

[0063] When both teachers can output stable results, enter bidirectional mutual learning: compress the features of each layer of T-L and T-R to the same size 64-dimensional vector through 1x1 convolution, and calculate the symmetric KL divergence with temperature T=2, whose formula is as follows:

[0064]

[0065] and supplemented by InfoNCE contrast loss to narrow the positive samples and push away the negative samples, encouraging the complementary of low-frequency features of brightness and high-frequency features of rain streaks without excessive assimilation. After mutual learning, T-L can generate a brightness reference with balanced illumination and true color on the same night rain image, and T-R can output a rain-free and complete texture reference; the feature entropy of both is significantly improved, forming a complementary knowledge base covering global illumination and local rain streaks, laying a precise and diverse teacher signal foundation for the next stage of knowledge transfer to the student network.

[0066] (3) Training the joint brightness-rain mask prior mapping network: first, the normalized global brightness mean μ_I, the rain mask density μ_r and the binary rain mask M_r are concatenated in the channel dimension to form a prior tensor ψ_in of 3×512×512; this tensor is first input into a depth separable convolution with a kernel size of 5×5 and a step size of 2, and is equipped with SiLU activation and GroupNorm to complete the first layer of down-sampling, outputting features with 96 channels and a resolution of 256×256; then two levels of “1×1 compression-3×3 empty depth convolution (dilation=2)-1×1 expansion” residual bottleneck are connected in series, where the first level maintains the resolution and outputs 128 channels, and the second level sets the step size to 2 at the empty convolution, further reduces the size to 128×128 and expands to 192 channels; then a depth separable convolution with a kernel of 3×3 and a step size of 2 is used to complete the third down-sampling, obtaining a feature map with 256 channels and 64×64, and the SE-Attention (global average pooling→FC(256→64)→ReLU→FC(64→256)→Sigmoid) is used to adaptively re-scale each channel to enhance the perception of rain streak distribution. After completing the spatial feature extraction, AdaptiveAvgPool2d and AdaptiveMaxPool2d are used to aggregate the feature map into two 256-dimensional vectors g and h, which are concatenated into a 512-dimensional descriptor, and then passed through the fully connected mapping head FC1(512→256)-Mish-Dropout(0.4)-FC2(256→128) to obtain a compact prior vector φ, which is limited to the interval [-1, 1] by HardTanh. The vector φ∈ℝ¹² 8 In the forward noise scheduling stage, it is converted into non-zero mean noise μ_t by the time mapping function g_t(·). The final φ can accurately encode the brightness and rain mask joint statistics, so that the model can significantly reduce the mean fitting error in the night rain scene and improve the subsequent rain removal and brightening effect.

[0067] (4) Construct a unified student network PiDNet-X as shown in Figure 6 , and perform distillation training: after establishing the dual-teacher complementary knowledge base, the six-channel tensor (RGB + HVI) of the same low-light rain scene image and the prior vector φ are simultaneously input into PiDNet-X. PiDNet-X adopts a “one network with two branches” backbone: the left brightness branch is a lightweight Deform-Conv-UNet, which is specifically designed to restore low-frequency illumination and structure; the right rain removal branch is a MobileViT-S stacked with a SIREN-INR sub-module, which can analyze high-frequency rain marks and raindrop highlights.

[0068] Before the brightness branch F_L and the rain removal branch F_R enter the PRLA interaction, a layer of multi-scale polarized gate attention module MSLA-PolarGate is connected in series as shown in Figure 3are shown. The specific process is as follows: first, F_L and F_R are divided into 4x4, 8x8, and 16x16 windows to perform local self-attention, and the Query / Key / Value are shared within different windows, but the windows do not interfere with each other; then, the three features are dynamically weighted and fused by pixel using the Soft-Gate gating mechanism, and the weight is controlled by the 3-channel gating vector output by the global brightness mean μ_I and the rain mask density μ_r through two layers of MLP + Sigmoid, so as to automatically enlarge the large window receptive field in dark and rainy areas, and the small window is used in the bright area / less rainy area to retain details. The fusion result is added to the original polarization angle coordinate coding to form a polarization position-sensitive multi-scale feature, and then sent to the PRLA such as Figure 7 The module explicitly introduces multi-scale directional information without increasing the depth, which can further improve the perception ability of oblique long rain filaments, and adaptively allocates the calculation amount using the rain-brightness prior.

[0069] After the MSLA-PolarGate coarse screening, the rain-light asymmetric dual attention module RLA-Duo such as Figure 5 is used to further screen the brightness and rain removal features. The module first calculates the brightness residual and rain mask density μ_r, and drives the spatial and channel attention respectively: the spatial branch uses 3x3 deep convolution to estimate the degree of insufficient illumination of each pixel position, and gets the spatial weight w_s through Softmax; the channel branch uses 1x1 convolution and Sigmoid to output the channel weight w_c related to the rain density. Then, the feature is screened in the channel by w_c, and then screened in the position by w_s, and the double-weighted feature focuses on the "dark and rainy" area; the feature then interacts with the rain removal branch through the PRLA such as Figure 7 is shown, while the bright area or rain trace sparse area is more along the Skip straight through to avoid excessive smoothing.

[0070] The two branches pass through the polarization rain-light attention PRLA such as Figure 7To perform cross-branch dynamic interaction, the processing flow of the PRLA polarized rain-light attention module is as follows: first, the luminance branch feature F_L and the rain removal branch feature F_R are respectively sent into a convolution layer with a kernel size of 1*1 and a step of 1 after being aligned by channel, and light mapping is realized by depth separable, and the output is Query and Key with half of the original channel number (C->C / 2); the convolution result is rearranged into a matrix with a size of HW*C / 2 after SiLU activation and BatchNorm normalization. At the same time, the original feature of the rain removal branch generates a Value tensor through another 1*1 convolution with shared weights. Subsequently, the global luminance mean μ_I and the rain mask density μ_r obtained in the data preprocessing stage are spliced, and then mapped through two fully connected layers and activated by Sigmoid to obtain a channel gating vector β with a length of C / 2. The vector is broadcasted to Query and Key by channel to realize weight amplification at positions with high rain density and low luminance. Then, the weighted Query and Key are multiplied to obtain spatial attention coefficients through Softmax, and then multiplied with Value to return to the original resolution to obtain attention enhanced features F_att. Finally, β is broadcasted to C-dimensional gating coefficients γ by channel again, and then weighted and fused with the luminance branch feature F_L by channel: more F_att output is retained in low luminance and rain trace dense areas, and F_L original information is retained in bright areas; the fusion result is sent into the decoder after being restored to C channels through 1*1 convolution, GroupNorm and SiLU activation. Subsequently, the channel is recalibrated by HVI-Gate, and the color noise is modulated by rain mask saliency and luminance gradient;

[0071] The decoder end inserts BFPU-RainSwitch as Figure 8As shown, the workflow of the BFPU-RainSwitch closed-loop decoder in the decoding stage can be summarized as the following sequential operations: first, the bottleneck features from the encoding end are sent into an UpscaleBlock; the block first performs channel compression through a 1x1 convolution with a step size of 1, followed by a SiLU activation and a GroupNorm, and then uses a PixelShufflex2 to double the feature map size, followed by a layer of 3x3, step size 1, padding 1 convolution (with BatchNorm and ReLU) to remove the up-sampling artifacts and restore the local texture, to obtain the initial decoding features F0 with doubled resolution and halved channel number. Next, it enters the step-by-step recovery module composed of 3 levels of UpscaleBlock: each level first concatenates F0 with the corresponding scale of the encoder Skip features in the channel dimension, and then uses a 3x3 convolution (s=1, p=1) to fuse and BN+ReLU; then it is up-sampled again using PixelShufflex2, and followed by a layer of 3x3 convolution + BatchNorm + ReLU to fine-tune the texture, so that the feature continuously enlarges to 1 / 2, 1 / 1 of the original image size. After reaching the target resolution, the feature is sent to the BasicFeatureProcessingUnit (BFPU) to refine the rain streaks: the BFPU contains a double-branch residual unit inside, with the left branch being a normal 3x3 convolution (s=1, p=1) to capture low-frequency structures, and the right branch being a 3x3 dilated convolution (s=1, p=2) with an expansion rate of 2 to extract long strip rain marks; after BatchNorm+ReLU of the two branches respectively, they are added together, and then compressed back to the original channel number through a 1x1 convolution to form the fused feature F_bfpu, which is stacked twice to further strengthen the texture. Then it enters the RainSwitchGate: first, F_bfpu is compressed to C / 8 channels through a 1x1 convolution, and then mapped into a single-channel rain streak confidence map through a 1x1 convolution after BatchNorm+ReLU, and the Sigmoid activation obtains the mask a(x, y); the mask performs element-wise multiplication on F_bfpu, realizing the suppression of high-rain-area features and the preservation of low-rain-area features. In parallel with this, the Skip feature from the previous layer of UpscaleBlock directly reaches the RainSwitch output, and is element-wise added to the gated feature to form F_out. Finally, F_out is mapped back to the sRGB space through a 1x1 convolution + Tanh function to output the enhanced image, and is sent back to the input of the first UpscaleBlock through a closed-loop feedback line, so that the decoder can adaptively refine the residual rain marks in the next frame or iteration.

[0072] After the output resolution of the rain removal decoder is restored to be consistent with the input, the HiLo-DFA is as follows Figure 4As shown, an overall FFT is performed on the feature map, and the frequency spectrum is divided into low-frequency and high-frequency parts with an energy ratio of less than 25%. The low-frequency branch first passes through a set of learnable Retinex kernels (3x3 deep separable convolution stack) for global light stretching, which explicitly compensates for the dynamic range compression caused by low illumination; the high-frequency branch calls the 4-step Tiny-DDPM inverse diffusion process, which uses the previously encoded non-zero mean noise anchor to re-denoise high-frequency noise points and fine raindrops. After the two branches return to the time domain, they are spliced by 1x1 convolution, and the channel response is recalibrated using SE-Attention; finally, the residual addition is performed with the main Decoder output. The frequency-diffusion joint strategy can simultaneously brighten the dark area and restore the high-frequency texture, significantly reducing the fog-like noise compared with the convolution Decoder.

[0073] The two-branch Decoder output is aligned in the color domain, and the enhanced result Î is generated by fusion according to the learnable weights α and β.

[0074] The distillation process is completed in three stages:

[0075] Stage one soft label distillation: taking the brightness teacher output Ŷ l and the rain removal teacher output Îᵣ as soft targets, temperature smoothing cross entropy is used, and its formula is as follows:

[0076]

[0077] where T=2; α+β=1, prompting the student network to first obtain the global trend of image illumination and rain removal.

[0078] Stage two feature prompt distillation: extract the teacher corresponding layer features F l , Fᵣ, which are spliced into Hint vectors H after being compressed to 64 dimensions by 1x1 convolution, and the fine-grained texture and structure are explicitly injected into the student middle layer by Hint-Loss. Its formula is as follows:

[0079]

[0080] Stage three gated fusion distillation: open the learnable parameters of HVI-Gate, dynamically balance the illumination and rain removal features, and jointly minimize the formula as follows:

[0081]

[0082] where Ĩ is the reference result obtained by fusing the teacher double output by PRLA-same.

[0083] The total loss of the three stages is as follows:

[0084]

[0085] Top-down progressive convergence. The distilled coarse student is denoted as Student-C; each layer is pruned by half of the original number of channels using Born-Again compression, resulting in a lightweight Student-F, while preserving PRLA and HVI-Gate to maintain the rain removal and highlight enhancement effects.

[0086] The reasoning phase executes Student-C and Student-F in parallel, and the outputs of the two paths are updated using an exponential moving average (EMA) coefficient η. Subsequently, the Student-F is fine-tuned for difficult samples through a lightweight mutual learning (temperature τ = 2), so that the enhanced results can maintain brightness balance, complete rain removal, and complete texture details under extreme night rain.

[0087] (5) Online mutual learning and adaptive correction: the system simultaneously loads the coarse student Student-C and the fine student Student-F. For any input night rain image, the two are inferred in parallel to obtain the enhanced result I C and I F First, the fixed exponential moving average (EMA) weight η = 0.07 is used for fusion to obtain the basic output, and the formula is as follows:

[0088]

[0089] Among them, Student-F focuses on rain removal high-frequency details, Student-C retains stronger light stability, and EMA can smooth the differences between the two within a single frame and eliminate flicker.

[0090] At the final output position of the Student-F decoder, the main branch first generates a convolution result Î_conv through 3x3 convolution, BatchNorm, and SiLU activation. To further weaken dark fine noise and small raindrops, a simple inverse diffusion branch is connected in parallel at the same position: first, use a 1x1 convolution with a step size of 1 to compress the decoding features to C / 2 channels, and then normalize them through GroupNorm, and then map them to the diffusion latent space. Then, use DDPM-Tiny with only 4 time steps to add noise forward using the prior noise mean μ_t, and then predict the noise residual ε_t backward, and then reconstruct the diffusion image Î_diff through single-step inversion. Î_diff is restored to C channels through a 1x1 convolution, and then fused with Î_conv at the pixel level using an exponential moving average with a weight coefficient of 0.7:0.3 (the main branch has a higher weight than the diffusion correction branch), and the final enhanced image Î_out is output. In the training stage, a self-consistency loss is additionally introduced, where λ = 0.1, to constrain the outputs of the two branches to be consistent in content and structure, and to promote the convolution main branch to learn the detail repair ability of the diffusion branch. The formula is as follows:

[0091]

[0092] To detect the difficult case scenarios, the system computes the mean of the input luminance in real time and the rain mask density , and the Mahalanobis distance with the training set statistics mean, whose formula is as follows:

[0093]

[0094] When d M > 3, it is determined as an extreme night-rain map outside the distribution. At this time, trigger a lightweight online mutual learning: take the sliding average result as the temporary soft label, take the output of Student-C as the student and the output of Student-F as the teacher (the roles of the two are exchanged in the next frame), and use the KL distillation loss with temperature τ = 2, whose formula is as follows:

[0095]

[0096] Fine-tune Student-C for 1-2 steps (where the learning rate is 1x10 −5 , and the weights are only updated for the last two layers of the decoder). Then update the Student-F in reverse distillation, and also adjust only the penultimate layer MobileViT-SBlock. Recalculate after fine-tuning, whose formula is as follows:

[0097]

[0098] , where is the final enhanced result output.

Claims

1. A low-light overlay rainy day image enhancement method based on heterogeneous multi-stage teacher-student distillation, DEPMD-Net is shown in Figure 1. Its features include the following method steps: (1) Overall architecture design: The DEPMD-Net proposed in this invention is shown in Figure 1. It addresses the image enhancement problem of low-light superimposed rain scene and follows the four core principles of "low frequency brightness compensation + high frequency rain streak suppression + prior dynamic gating + stepwise distillation compression". The overall network is composed of six functional domains connected in series from top to bottom: statistical prior channel, rain removal high frequency branch, brightness low frequency branch, cross-branch polarization fusion layer, progressive decoding and dual student online correction layer. (1.1) Global Statistical Prior Channel: The global brightness mean μ_I, rain mask density μ_r, and binary rain mask M_r are extracted from the input image in one go and concatenated in the channel dimension to form a prior vector φ. This vector is gated by MSLA-Polar as shown in Figure 3 to generate a channel vector w_c containing 4×4, 8×8, and 16×16 multi-scale window weights, which is used to dynamically modulate the attention intensity of the subsequent dual branches. Prior information runs through the entire network. Using RLA-Duo as shown in Figure 5, the brightness residual and rain mask are used to construct two Query / Key paths, which apply higher attention to "dark and rain-dense" pixels and reduce intervention in bright areas and sparse rain areas. The generated weighted features are injected into the high-frequency rain removal branch and the low-frequency brightness branch, respectively, to provide more accurate saliency guidance for subsequent decoupling. (1.2) High-frequency rain removal branch: First, slice embedding is performed in the polarization DCT frequency domain to encode the significant directionality of high-frequency rain streaks as tokens; then, CA-Win×DilatedSwin is introduced as shown in Figure 2, and variable windows are obtained by content-aware clustering, and then four-level dilated convolution Query / Key bias is superimposed to efficiently capture oblique rain streaks such as 0°, 45°, and 90°; after passing through the rain removal decoder (built-in BFPU-RainSwitch as shown in Figure 8), residual rain streaks are suppressed and texture details are restored, and the high-frequency rain removal feature F_R is output; (1.3) Low-frequency branch of brightness: First, generate frequency-conditional convolution kernel γ(f), and amplify the light filling of the small rain area and suppress high-frequency color noise according to the difference in frequency radius; the low-frequency path uses Retinex 3×3 depth separable stack for global brightening, and the high-frequency path uses Tiny-DDPM 4-step denoising network to eliminate color noise and raindrops, and finally outputs low-frequency brightness feature F_L. (1.4) Cross-branch polarization fusion layer: As shown in Figure 7, in the PRLA polarization rain-light attention module, F_R, F_L and polarization coordinate (r,θ) position encoding are input together; spatial-channel asymmetric weights ensure that the rain removal features of the "dark and rain-dense" region dominate, while the "bright" region retains more lighting details; then the two features are fused for the first time at the residual Add node; (1.5) Progressive decoding and multi-level correction: The fused features are sequentially passed through the RainSwitch texture compensation unit as shown in Figure 8, and the HiLo-DFA frequency division diffuser as shown in Figure 4, to complete the mesoscale texture restoration and frequency division progressive denoising; the end is connected in parallel with Student-C / Student-F dual student branches and performs exponential moving average (EMA) mutual learning during the inference period to generate the final enhanced image Î; (1.6) Dual-student progressive distillation-mutual learning, the overall process is shown in Figure 9: During the training phase, a three-stage strategy of "dual-teacher pre-training → Hint-distillation → Born-Again compression" is adopted to transfer the complementary knowledge of the brightness teacher and the rain removal teacher to the PiDNet-X student network as shown in Figure 6. During the inference phase, Student-C / Student-F continue to maintain lightweight mutual learning and online temperature scaling correction to ensure the real-time adaptability of extreme night rain samples. (2) Improved dataset: Based on the public Rain100L dataset, the overall brightness was adjusted according to the random γ transformation and brightness decay curve to generate low-light rain images, with a total of about 5,000 images; rain streaks and raindrops based on physical models or real sampling were injected into the existing low-light image dataset LOL-v1 to generate low-light rain images, with a total of about 6,500 images; 80% of them were selected as the training set and 20% were selected as the test set. (3) Training a heterogeneous dual-teacher network, including the following sub-steps; (3.1) Luminance Teacher Network TL: The HVI-CIDNet backbone is used, and the input is a six-channel tensor (RGB+) after HVI color gamut separation. + +I), the brightness teacher provides a low-frequency global illumination prior and outputs a brightness-corrected image. ; (3.2) Rain De-Rain Teacher Network (TR): Using the NeRD-Rain backbone, the coordinate-feature hybrid implicit neural representation (INR) is embedded into the Transformer-UNet network. The rain de-rain teacher network provides high-frequency rain ripple suppression capability and outputs a rain-de-rain image. ; (3.3) Bidirectional Mutual Learning: Using Symmetric KL Divergence and InfoNCE Loss Pairs and The features are aligned complementaryly, and mutual learning enables the two networks to share advantageous features, laying a complementary knowledge foundation for subsequent distillation; (4) Generate joint brightness-rain mask prior and anchor noise, including the following sub-steps; (4.1) Calculate the global mean brightness μ I With rain mask density μᵣ; (4.2) Encoding μ l μᵣ and the rain mask Mᵣ are used to generate a 128-dimensional prior vector φ through a prior mapping network, and L is minimized. prior Make corrections; (4.3) Non-zero mean noise is used in the conditional diffusion positive noise scheduling to simulate the distribution of night rain noise. Non-zero mean anchoring makes the diffusion process truly reflect the statistical characteristics of night rain noise and improves the diffusion model's ability to model complex degradation. (5) Construct the unified student network PiDNet-X structure as shown in Figure 6, and perform distillation training, including the following sub-steps; (5.1) Design of dual-branch backbone: The brightness branch adopts lightweight Deform-Conv-UNet to restore low-frequency illumination; the rain removal branch adopts MobileViT-S+SIREN-INR to suppress high-frequency rain fringes; (5.2) A polarized rain-light attention PRLA module is inserted between the two branches of Bottleneck as shown in Figure 7 to dynamically fuse rain mask and brightness information for spatial domain rain ripple suppression. (5.3) HVI-Gate gates are introduced at the feature input terminals of each layer as shown in Figure 7 to modulate feature F and suppress color noise; (5.4) Add the BFPU-RainSwitch module to the Decoder node of the rain removal branch as shown in Figure 8 to block the mistransmission of rain patterns and enhance detail compensation; (5.5) The soft labels output by the two teachers and the intermediate features are used as hints, and then gated and distilled into PiDNet-X to obtain the coarse student Student-C; (5.6) Perform Born-Again compression on Student-C, reducing the number of channels by 1 / 2, to obtain the refined student Student-F; (6) Online mutual learning and adaptive correction; (6.1) During the inference phase, Student-C and Student-F are executed simultaneously, and their outputs are updated with an exponential moving average (EMA) weight η≤0.1; (6.2) For difficult example images, the probability distribution is readjusted using mutual learning temperature scaling τ=2.0 to improve robustness; (6.3) Output enhanced results with balanced brightness, rain streaks removed, and rich details.

Images